Animal behavior recognition system and method based on acoustic-visual fusion

Abstract

The application discloses an animal behavior recognition system and method based on acoustic-visual fusion, comprising: collecting animal video data and synchronous sound data to generate acoustic-visual animal behavior data set; extracting continuous video frames and synchronous sound segments and converting time-frequency spectrum to generate visual feature map sequence and sound time-frequency feature map sequence; inputting the visual feature map sequence and the sound time-frequency feature map sequence into an improved VMamba model to generate acoustic-visual fusion behavior feature sequence; identifying sound response leading position, acoustic-visual synchronous keeping position and sound response continuation position to generate acoustic-visual behavior evolution trajectory; performing same-posture different-sound branch splitting based on the acoustic-visual behavior evolution trajectory to generate behavior branch discrimination result; checking each behavior branch state to generate animal behavior recognition result. The application realizes continuous and accurate recognition of animal behavior in a complex scene through acoustic-visual multi-modal time lag calibration, occlusion compensation and same-posture different-sound branch discrimination.

Description

Technical Field

[0001] This invention relates to the field of intelligent animal behavior recognition technology, and in particular to an animal behavior recognition system and method based on acoustic-visual fusion. Background Technology

[0002] With the development of smart farming, animal health monitoring, and animal behavior research, the automatic identification of animal activity using video surveillance equipment has become an important technical means for intelligent animal management. Existing animal behavior recognition methods typically extract features from the animal's body outline, key limb points, posture changes, and movement trajectory by continuously acquiring animal video footage. Based on these continuous visual features, they determine whether the animal is eating, drinking, walking, lying down, or engaging in other behaviors. Some methods further incorporate sound data, combining animal calls, chewing sounds, drinking sounds, and activity sounds with video features to increase the information sources for behavior recognition.

[0003] Existing methods for identifying animal behavior based on sound and vision typically map sound segments to video frames according to the time of acquisition and then splice or jointly classify sound and visual features. However, in actual animal behavior, sound responses and visual actions are not always synchronized; sound may precede visual actions or continue after the visual actions have ended. Furthermore, in group settings, animals may occlude each other, have partially invisible limbs, or exhibit similar postures, making it easy for visual actions to be interrupted or for different actual behaviors to display similar visual features.

[0004] Therefore, existing technologies are unable to accurately characterize the temporal relationship between sound response and visual action. When animals exhibit sound-visual temporal misalignment, visual occlusion, or different sound responses corresponding to the same posture, sound information is easily associated with incorrect visual actions. Furthermore, it is difficult to use sound changes before and after a behavior to distinguish different real behaviors under similar postures, resulting in the incorrect segmentation of continuous behaviors, confusion of adjacent behaviors, and misjudgment of animal behavior categories, which affects the accuracy of animal behavior recognition in complex breeding environments.

[0005] Therefore, how to provide an animal behavior recognition system and method based on acoustic-visual fusion is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose an animal behavior recognition system and method based on acoustic-visual fusion. This invention fully utilizes acoustic-visual multimodal fusion technology and an improved VMamba model. Through cross-modal time delay calibration, occlusion-triggered selective scanning, acoustic-visual difference state correction, behavioral evolution trajectory construction, and branch splitting of the same posture but different sounds, it achieves correlation analysis between sound response and visual action and animal behavior category recognition. It has the advantages of accurate acoustic-visual temporal correlation, strong adaptability to visual occlusion, strong ability to distinguish similar animal behaviors, and high behavior recognition accuracy.

[0007] An animal behavior recognition method based on acoustic-visual fusion according to an embodiment of the present invention includes: Collect animal video data and synchronized audio data, perform preprocessing on the animal video data and synchronized audio data, and generate an audiovisual animal behavior dataset; Based on the acoustic-visual animal behavior dataset, continuous video frames and synchronized sound segments are extracted, and the synchronized sound segments are converted into time-frequency spectra to generate visual feature map sequences and sound time-frequency feature map sequences. The visual feature map sequence and the sound time-frequency feature map sequence are input into the improved VMamba model, which includes a bimodal state coding layer, a cross-modal time-delay state bridge, an occlusion-triggered selective scanning layer, and an acoustic-visual difference state correction layer to generate an acoustic-visual fusion behavioral feature sequence. Based on the sound and visual fusion behavior feature sequence identification, the sound response precursor position, the sound and visual synchronization maintenance position, and the sound response continuation position are identified. Behavior precursor region, behavior manifestation region, and behavior residue region are constructed respectively. The regions are connected according to the time sequence to generate the sound and visual behavior evolution trajectory. Based on the acoustic-visual behavior evolution trajectory, the same posture different sound branch is split. The sound response state is tracked in the behavior manifestation area. The sound response conversion position is determined as the behavior branch boundary. The branch connection relationship is established by combining the corresponding behavior precursor area and behavior residual area to generate behavior branch discrimination results. Based on the behavioral branch discrimination results, the duration interval, branch connection relationship and sound response state of each behavioral branch are verified to determine the animal behavior category and generate animal behavior recognition results.

[0008] Optionally, the animal video data includes continuous animal image frames, animal target location, animal body outline, key points of animal limbs, animal posture, animal movement direction, and video acquisition timestamp. The synchronized sound data includes animal calls, chewing sounds, drinking sounds, breathing sounds, activity sounds, and sound acquisition timestamp.

[0009] Optionally, the preprocessing of animal video data and synchronized audio data includes establishing a unified timeline based on the video acquisition timestamp and the audio acquisition timestamp, performing frame rate unification, image size normalization, and image noise suppression on the animal video data, performing sampling rate unification, background noise suppression, and amplitude normalization on the synchronized audio data, and performing initial time pairing of continuous image frames and synchronized audio segments according to the unified timeline to generate an audiovisual animal behavior dataset.

[0010] Optionally, the generation of the visual feature map sequence and the sound time-frequency feature map sequence includes: Read continuous image frames and synchronous sound segments from the acoustic-visual animal behavior dataset according to a unified timeline, divide the acoustic-visual time window according to the acquisition time period corresponding to the continuous image frame, and generate an acoustic-visual time window sequence. Based on the audiovisual time window sequence, continuous image frames are extracted window by window. Animal image regions are determined according to the animal target location and animal body outline. The animal image regions are then arranged in chronological order to generate an animal visual frame sequence. Based on animal visual frame sequences, animal body contours, limb key points, posture and movement direction are extracted. Visual information corresponding to the same acquisition time period is combined according to spatial location to generate a visual feature map sequence. Based on the audiovisual time window sequence, synchronous sound segments corresponding to each animal visual frame sequence are extracted from the synchronous sound data. Framing and windowing processing are performed on each synchronous sound segment, and they are arranged in chronological order to generate an animal sound frame sequence. Perform time-frequency transformation on the animal sound frame sequence to extract the frequency distribution, energy distribution and time change information of each sound frame, establish the correspondence according to the sound-visual time window sequence, and generate a sound time-frequency feature map sequence.

[0011] Optionally, the generation of the acoustic-visual fusion behavioral feature sequence includes: An improved VMamba model is constructed based on the visual state space structure of the VMamba model. The input embedding, two-dimensional selective scanning, state space transfer and output mapping structures are retained. A bimodal state coding layer is set in parallel in the input embedding structure. A cross-modal time-delay state bridge is set in front of the two-dimensional selective scanning structure. An occlusion-triggered selective scanning layer is set inside the two-dimensional selective scanning structure. An acoustic-visual difference state correction layer is set between the state space transfer structure and the output mapping structure. The visual feature map sequence and the sound time-frequency feature map sequence are input into the bimodal state coding layer, and feature embedding and state coding are performed respectively. The coding results are arranged according to the sound-visual time window sequence to generate a visual action state sequence and a sound response state sequence. Based on the visual action state sequence and the sound response state sequence, the positions of continuous state changes are extracted, and the time offset of the sound response state relative to the visual action state is determined according to a unified time axis to generate an acoustic-visual time delay relationship sequence. Input the acoustic-visual time delay relationship sequence into the cross-modal time delay state bridge, adjust the time position corresponding to the sound response state according to the time offset, establish a cross-time connection relationship with the corresponding visual action state, and generate a time delay calibration state sequence. The location of visual action state continuity interruption is determined based on the time-delay calibration state sequence. The corresponding sound response state is read by selecting the scanning layer through occlusion trigger, and the corresponding sound response state is written into the continuity interruption location and transmitted along the two-dimensional selected scanning direction to generate an occlusion compensation state sequence. Based on the occlusion compensation state sequence, a continuous segment is determined where the visual action state is maintained and the sound response state changes. The corresponding visual action state is corrected by the sound-visual difference state correction layer according to the sound response state change, and a difference correction state sequence is generated. The time-delay calibration state sequence, occlusion compensation state sequence, and difference correction state sequence are fused along a unified time axis, and the fusion results are arranged in the order of state succession to generate an audiovisual fusion behavior feature sequence.

[0012] Optionally, the generation of the acoustic-visual behavioral evolution trajectory includes: Read the visual action state and sound response state in the audio-visual fusion behavior feature sequence according to a unified time axis, extract the state change direction and state persistence relationship of adjacent time positions, and generate audio-visual state change record. Based on the recording of changes in acoustic and visual states, the starting position where the acoustic response state changes while the visual action state remains continuous is determined. This starting position is defined as the acoustic response precursor position, and the system continuously tracks the position where the visual action state changes to construct a behavioral precursor zone. Continue tracking the visual action state and sound response state from the end of the behavioral precursor zone, and determine the continuous positions where the two simultaneously maintain the corresponding state as the sound-visual synchronization maintenance positions, and form the behavioral manifestation zone by the continuous sound-visual synchronization maintenance positions. Continue tracking the visual action state and sound response state from the end of the behavior manifestation area, determine the starting position where the visual action state ends and the sound response state continues, determine the starting position as the sound response continuation position, and construct the behavior residue area along the direction of the sound response state continuation. Based on the start and end positions of the behavioral precursor zone, behavioral manifestation zone, and behavioral residue zone on a unified timeline, establish the temporal continuity relationship from the behavioral precursor zone to the behavioral manifestation zone and from the behavioral manifestation zone to the behavioral residue zone, and generate behavioral region continuity records. The corresponding behavioral precursor area, behavioral manifestation area, and behavioral residue area are sequentially connected according to the behavioral region records. The connection results are arranged according to the visual action state and sound response state corresponding to each region to generate the audiovisual behavioral evolution trajectory.

[0013] Optionally, the generated behavior branch discrimination result includes: According to the evolution trajectory of sound and visual behavior, read the visual action state and sound response state corresponding to each behavior manifestation area, filter the behavior manifestation areas where the visual action state is continuously maintained, and generate the same posture maintenance segment. The sound response state is continuously tracked along the time direction of the same posture maintenance segment. The frequency distribution, energy distribution and duration of adjacent sound response states are read, and the time position where the sound response state changes is determined as the sound response change position. Based on each sound response transition position, the corresponding attitude-maintaining segment is divided. The continuous segments between adjacent sound response transition positions and between the boundary of the attitude-maintaining segment and adjacent sound response transition positions are determined as behavior branches, and each sound response transition position is determined as the boundary of the behavior branch. Based on the start and end positions of each behavior branch on a unified timeline, the corresponding behavior precursor area is traced before the start position of the behavior branch, and the corresponding behavior residue area is traced after the end position of the behavior branch, generating regional association records for each behavior branch. Based on the regional association records, the corresponding behavioral precursor areas, behavioral branches and behavioral residue areas are connected in chronological order to establish the branch-support connection relationship of each behavioral branch, and the corresponding visual action state, sound response state and duration are recorded. Based on the branch-support connection relationship of each behavior branch, different sound response states corresponding to the same visual action state are distinguished, and each behavior branch is arranged according to the boundary of the behavior branch to generate behavior branch discrimination results.

[0014] Optionally, the generation of animal behavior recognition results includes: Read the visual action state, sound response state, behavior branch boundary and duration interval corresponding to each behavior branch in the behavior branch discrimination result, and establish a behavior branch state record according to a unified time axis. Based on the behavior branch state record, the duration interval of each behavior branch is verified, and the duration interval is matched with the start and end positions of the corresponding behavior precursor area, behavior manifestation area and behavior residue area to generate the duration interval verification result. Based on the continuous interval verification results, read the branch support connection relationship between adjacent behavior branches, verify the time continuity state between the behavior residual area of ​​the previous behavior branch and the behavior precursor area of ​​the next behavior branch, and generate the branch support connection verification results. By combining the results of the branch connection verification with the sound response state corresponding to each behavioral branch, the behavioral branches corresponding to different sound response states under the same visual action state are distinguished, and the animal behavior category is determined based on the correspondence between the visual action state and the sound response state. Animal behavior categories are associated with the start and end positions, duration, visual action state, and sound response state of their corresponding behavior branches, and arranged according to a unified timeline to generate animal behavior recognition results.

[0015] An animal behavior recognition system based on acoustic-visual fusion according to an embodiment of the present invention includes: The data acquisition module is used to collect and preprocess animal video data and synchronized audio data to generate an audiovisual animal behavior dataset. The feature extraction module is used to extract video and sound features based on the acoustic and visual animal behavior dataset, and generate visual feature map sequences and sound time-frequency feature map sequences. The audio-visual fusion module is used to input the visual feature map sequence and the audio time-frequency feature map sequence into the improved VMamba model to generate an audio-visual fusion behavior feature sequence. The behavior evolution module is used to construct the behavior precursor region, behavior manifestation region and behavior residue region based on the acoustic-visual fusion behavior feature sequence, and generate the acoustic-visual behavior evolution trajectory. The branch discrimination module is used to perform branch splitting based on the acoustic-visual behavior evolution trajectory, determine the behavior branch boundary and the branch connection relationship, and generate behavior branch discrimination results. The behavior recognition module is used to verify the status of each behavior branch based on the behavior branch discrimination results and determine the animal behavior category, and generate animal behavior recognition results.

[0016] The beneficial effects of this invention are: This invention combines animal video data with synchronized audio data, inputting visual feature map sequences and audio time-frequency feature map sequences into an improved VMamba model. It utilizes a bimodal state coding layer, a cross-modal time-delay state bridge, an occlusion-triggered selective scanning layer, and an audiovisual difference state correction layer to establish a correlation between audio response states and visual action states. This ensures accurate temporal alignment between audio information and corresponding visual actions, reducing audiovisual mismatches caused by premature or delayed audio responses and improving the effectiveness of audiovisual fusion behavioral features.

[0017] This invention identifies the preceding position of sound response, the position of simultaneous sound and visual recognition, and the position of continuation of sound response based on the sequence of sound and visual fusion behavioral features. By constructing a behavioral precursor zone, a behavioral manifestation zone, and a behavioral residue zone, it forms a sound and visual behavioral evolution trajectory, transforming animal behavior from a state judgment based on a single time and location into a continuous representation of the complete behavioral process. At the same time, it uses the sound response state to supplement the missing action information caused by visual occlusion, improving the stability of behavior recognition under conditions of mutual occlusion, partial limb obscuration, and interruption of continuous visual state.

[0018] This invention performs branch splitting of similar postures but different sounds within the behavior manifestation area, determines the boundary of the behavior branch by the sound response transition position, and establishes a branch connection relationship by combining the behavior precursor area and the behavior residue area. This enables the effective differentiation of animal behaviors with similar visual postures but different sound responses, reduces confusion between feeding, drinking, sniffing and other similar posture behaviors, and determines the animal behavior category through joint verification of continuous intervals, branch connection relationships and sound response states, thereby improving the accuracy and continuity of animal behavior recognition in complex environments. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of an animal behavior recognition method based on acoustic-visual fusion proposed in this invention; Figure 2 This is a diagram showing the acoustic-visual compensation result under occlusion conditions of an animal behavior recognition method based on acoustic-visual fusion proposed in this invention. Figure 3 This is an evolution diagram of the behavioral precursor zone, manifest zone, and residual zone of an animal behavior recognition method based on acoustic-visual fusion proposed in this invention. Figure 4 This is a schematic diagram of the structure of an animal behavior recognition system based on acoustic-visual fusion proposed in this invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0021] refer to Figures 1-3 An animal behavior recognition method based on acoustic-visual fusion includes: Collect animal video data and synchronized audio data, perform preprocessing on the animal video data and synchronized audio data, and generate an audiovisual animal behavior dataset; Based on the acoustic-visual animal behavior dataset, continuous video frames and synchronized sound segments are extracted, and the synchronized sound segments are converted into time-frequency spectra to generate visual feature map sequences and sound time-frequency feature map sequences. The visual feature map sequence and the sound time-frequency feature map sequence are input into the improved VMamba model, which includes a bimodal state coding layer, a cross-modal time-delay state bridge, an occlusion-triggered selective scanning layer, and an acoustic-visual difference state correction layer to generate an acoustic-visual fusion behavioral feature sequence. Based on the sound and visual fusion behavior feature sequence identification, the sound response precursor position, the sound and visual synchronization maintenance position, and the sound response continuation position are identified. Behavior precursor region, behavior manifestation region, and behavior residue region are constructed respectively. The regions are connected according to the time sequence to generate the sound and visual behavior evolution trajectory. Based on the acoustic-visual behavior evolution trajectory, the same posture different sound branch is split. The sound response state is tracked in the behavior manifestation area. The sound response conversion position is determined as the behavior branch boundary. The branch connection relationship is established by combining the corresponding behavior precursor area and behavior residual area to generate behavior branch discrimination results. Based on the behavioral branch discrimination results, the duration interval, branch connection relationship and sound response state of each behavioral branch are verified to determine the animal behavior category and generate animal behavior recognition results.

[0022] In this embodiment, the animal video data includes continuous animal image frames, animal target location, animal body outline, key points of animal limbs, animal posture, animal movement direction, and video acquisition timestamp. The synchronized sound data includes animal calls, chewing sounds, drinking sounds, breathing sounds, activity sounds, and sound acquisition timestamp.

[0023] In this embodiment, the preprocessing of animal video data and synchronized audio data includes establishing a unified timeline based on the video acquisition timestamp and the audio acquisition timestamp, performing frame rate unification, image size normalization and image noise suppression on the animal video data, performing sampling rate unification, background noise suppression and amplitude normalization on the synchronized audio data, and performing initial time pairing of continuous image frames and synchronized audio segments according to the unified timeline to generate an audiovisual animal behavior dataset.

[0024] In this embodiment, the generation of the visual feature map sequence and the sound time-frequency feature map sequence includes: Read continuous image frames and synchronous sound segments from the acoustic-visual animal behavior dataset according to a unified timeline, divide the acoustic-visual time window according to the acquisition time period corresponding to the continuous image frame, and generate an acoustic-visual time window sequence. Based on the audiovisual time window sequence, continuous image frames are extracted window by window. Animal image regions are determined according to the animal target location and animal body outline. The animal image regions are then arranged in chronological order to generate an animal visual frame sequence. Based on animal visual frame sequences, animal body contours, limb key points, posture and movement direction are extracted. Visual information corresponding to the same acquisition time period is combined according to spatial location to generate a visual feature map sequence. Based on the audiovisual time window sequence, synchronous sound segments corresponding to each animal visual frame sequence are extracted from the synchronous sound data. Framing and windowing processing are performed on each synchronous sound segment, and they are arranged in chronological order to generate an animal sound frame sequence. Perform time-frequency transformation on the animal sound frame sequence to extract the frequency distribution, energy distribution and time change information of each sound frame, establish the correspondence according to the sound-visual time window sequence, and generate a sound time-frequency feature map sequence.

[0025] In this embodiment, generating the acoustic-visual fusion behavioral feature sequence includes: An improved VMamba model is constructed based on the visual state space structure of the VMamba model. The input embedding, two-dimensional selective scanning, state space transfer and output mapping structures are retained. A bimodal state coding layer is set in parallel in the input embedding structure. A cross-modal time-delay state bridge is set in front of the two-dimensional selective scanning structure. An occlusion-triggered selective scanning layer is set inside the two-dimensional selective scanning structure. An acoustic-visual difference state correction layer is set between the state space transfer structure and the output mapping structure. The visual feature map sequence and the sound time-frequency feature map sequence are input into a bimodal state coding layer, where feature embedding and state coding are performed respectively. The coding results are arranged according to the sound-visual time window sequence to generate a visual action state sequence and a sound response state sequence. The specific steps of feature embedding and state coding are as follows: The visual feature map corresponding to each sound-visual time window is divided into continuous visual feature blocks. Each continuous visual feature block is flattened, and the flattened visual features are mapped to a unified feature dimension according to their spatial position to generate visual embedding features. The scanning order is determined according to the spatial position of each visual embedding feature in the visual feature map. The visual embedding features corresponding to adjacent spatial positions are sequentially transferred to a state, and the state of the current spatial position is updated by combining the state encoding result of the previous spatial position, thereby generating the visual action state corresponding to each acoustic-visual time window. The sound time-frequency feature map corresponding to each sound-visual time window is divided into continuous sound time-frequency feature blocks along the time and frequency dimensions. Each continuous sound time-frequency feature block is flattened, and the flattened sound time-frequency features are mapped to the same feature dimension as the visual embedding features to generate sound embedding features. The scanning order is determined according to the time and frequency positions corresponding to each sound embedding feature. The sound embedding features of adjacent time positions are sequentially transferred to a state, and the state of the current time position is updated by combining the state encoding result of the previous time position, thereby generating the sound response state corresponding to each sound-visual time window. Arrange the visual action state and sound response state corresponding to each audiovisual time window according to the audiovisual time window sequence to generate the visual action state sequence and the sound response state sequence. Based on the visual action state sequence and the sound response state sequence, the positions of continuous state changes are extracted, and the time offset of the sound response state relative to the visual action state is determined according to a unified time axis to generate an acoustic-visual time delay relationship sequence. The acoustic-visual time delay relationship sequence is input into the cross-modal time delay state bridge. Based on the time offset, the time position corresponding to the acoustic response state is adjusted, establishing a cross-time continuity relationship with the corresponding visual action state, generating a time delay calibration state sequence, where: The cross-modal time-delay state bridge includes a time offset reading structure, a state position rearrangement structure, and a cross-time connection structure. The time offset reading structure reads the time offset and offset direction of each sound response state relative to the corresponding visual action state in the acoustic-visual time delay relationship sequence, and determines the target time position corresponding to each sound response state; The state position rearrangement structure moves the sound response state forward, maintains it, or moves it backward according to the target time position, and rearranges the adjusted sound response state according to the sound-visual time window sequence. The cross-time transition structure reads the adjusted sound response state and the visual action state at the corresponding target time position, establishes a state transition according to the time sequence, and records the source position of the sound response state, the target time position and the corresponding visual action state to generate a time delay calibration state sequence. The location of visual action state continuity interruption is determined based on the time-delay calibration state sequence. The corresponding sound response state is read from the selected scanning layer by occlusion triggering, and written to the continuity interruption location. This is then transmitted along the two-dimensional selected scanning direction to generate an occlusion compensation state sequence. Specifically, the generation of the occlusion compensation state sequence is as follows: Read adjacent visual action states in the time-delay calibration state sequence, and determine the location where the continuity of the visual action state is interrupted based on the changes in the visual action state in continuous time position and adjacent spatial position. Using the target time position corresponding to the interruption position of each visual action state as an index, the sound response state that establishes a cross-time connection with the target time position is read from the time delay calibration state sequence to generate the interruption position sound response record; Record the sound response at the interruption location and write it to the corresponding visual action state continuity interruption location. Also retain the visual action state at the previous spatial position and the previous time position before the continuity interruption location to generate the interruption compensation state. By triggering occlusion, the selected scanning layer sequentially transmits the interruption compensation state along the visual feature map in the horizontal forward order, horizontal reverse order, vertical forward order, and vertical reverse order. The state transmission results corresponding to each scanning direction are written back to the continuous interruption position and its adjacent positions to generate multi-directional compensation states. Arrange the multi-directional compensation states corresponding to each continuity interruption position according to the audiovisual time window sequence, and combine them with the visual action states that have not experienced continuity interruption to generate an occlusion compensation state sequence. Based on the occlusion compensation state sequence, continuous segments are determined where the visual action state is maintained and the sound response state transitions. A difference-corrected state sequence is generated by correcting the corresponding visual action state according to the sound response state transition through an acoustic-visual difference state correction layer. Specifically, the correction of the corresponding visual action state based on the sound response state transition through the acoustic-visual difference state correction layer is as follows: Read the visual action state and sound response state at adjacent time positions within each continuous segment, and determine the time position where the visual action state remains continuous and the sound response state changes as the sound response change position. Using each sound response transition position as a boundary, the sound response state before the transition, the sound response state after the transition, and the corresponding visual action state are extracted to generate a sound-visual difference state record. Based on the audiovisual difference state records, compare the frequency distribution, energy distribution, and duration of the sound response state before and after the conversion, determine the direction of change of the corresponding sound response state, and establish a corrective correspondence between the change of sound response state and the visual action state. According to the correction correspondence, the converted sound response state is written into the state representation of the corresponding visual action state, and the spatial and temporal positions of the original visual action state are retained to generate the corrected visual action state. Arrange each corrected visual action state according to the acoustic-visual time window sequence, and combine it with the visual action state that has not undergone sound response conversion to generate a difference correction state sequence. The time-delay calibration state sequence, occlusion compensation state sequence, and difference correction state sequence are fused along a unified time axis, and the fusion results are arranged in the order of state succession to generate an audiovisual fusion behavior feature sequence.

[0026] This invention structurally improves the traditional VMamba model, proposing an improved VMamba model for animal sound-visual fusion behavior recognition. In terms of bimodal feature processing, a bimodal state encoding layer is introduced, performing feature embedding and state encoding on the visual feature map sequence and the sound time-frequency feature map sequence respectively. Under a unified sound-visual time window, visual action state sequences and sound response state sequences are formed, converting visual action information and sound response information from different data forms into associative state representations, enhancing the model's ability to jointly represent changes in animal action and sound.

[0027] To address the issues of temporal misalignment between animal auditory responses and visual actions, as well as state interruptions caused by visual occlusion, a cross-modal time-delay state bridge and an occlusion-triggered selective scanning layer are introduced. The cross-modal time-delay state bridge reads the time offset between the auditory response state and the visual action state, performs temporal rearrangement of the auditory response state, and establishes a cross-temporal continuity relationship. When the continuity of the visual action state is interrupted, the occlusion-triggered selective scanning layer calls the corresponding auditory response state to write it to the interruption location and transmits the compensation state along the two-dimensional selective scanning direction. This allows the auditory response to participate in state transmission at the location where visual information is missing, thereby reducing the impact of auditory-visual temporal misalignment and animal occlusion on continuous behavioral representation.

[0028] To address the issue of similar visual postures but different auditory responses in animal behaviors such as feeding, drinking, and sniffing, an acoustic-visual difference state correction layer is introduced. Under the condition of continuous visual action state, the layer tracks the transition of auditory response state, extracts the difference in auditory response before and after the transition, and corrects the corresponding visual action state. This transforms animal behaviors that originally had similar visual representations into state representations with auditory differences. The improved VMamba model, while retaining the two-dimensional selective scanning and state space transfer capabilities, achieves acoustic-visual time delay calibration, occlusion state compensation, and enhancement of differences in different sounds within the same posture. This improves the accuracy and robustness of recognizing continuous and similar animal behaviors in complex breeding environments.

[0029] In this embodiment, the generation of the acoustic-visual behavioral evolution trajectory includes: Read the visual action state and sound response state in the audio-visual fusion behavior feature sequence according to a unified time axis, extract the state change direction and state persistence relationship of adjacent time positions, and generate audio-visual state change record. Based on the recording of changes in acoustic and visual states, the starting position where the acoustic response state transitions while the visual action state remains continuous is determined. This starting position is defined as the acoustic response precursor position, and continuous tracking is performed towards the position where the visual action state transitions to construct a behavioral precursor region. Specifically, constructing the behavioral precursor region involves: Read the sound response state and visual action state corresponding to adjacent time positions in the sound and visual state change record, and continuously compare the sound response state transition position and the visual action state transition position according to a unified time axis. When the sound response state changes and the corresponding visual action state remains continuous, the time position of the sound response state change is determined as the sound response precedence position, and the sound response state corresponding to the sound response precedence position is recorded. Starting from the sound response precursor position, the visual action state is continuously tracked along the time direction. The sound response state is recorded synchronously during the continuous visual action state. The time position where the visual action state first changes is determined as the precursor termination position. Extract the continuous visual action states and sound response states between the sound response precursor position and the precursor termination position, and establish the connection relationship between each state position in chronological order to generate a precursor state sequence. The preceding position of the sound response is used as the starting boundary of the region, the ending position of the precursor is used as the ending boundary of the region, and the precursor state sequence is assigned to the corresponding region to construct the behavior precursor region; Continue tracking the visual action state and sound response state from the end of the behavioral precursor zone, and determine the continuous positions where the two simultaneously maintain the corresponding state as the sound-visual synchronization maintenance positions, and form the behavioral manifestation zone by the continuous sound-visual synchronization maintenance positions. Continuing to track the visual action state and sound response state from the end of the behavior manifestation area, the starting position where the visual action state ends and the sound response state continues is determined. This starting position is defined as the sound response continuation position, and a behavior residue area is constructed along the direction of sound response state continuation. Specifically, constructing the behavior residue area along the direction of sound response state continuation involves: Starting from the position where the sound response continues, read the sound response state and visual action state sequentially along a unified time axis after the position where the sound response continues, and record the continuous maintenance of the sound response state. After the visual action state ends, the frequency distribution, energy distribution and duration of the sound response state are continuously tracked, and the time positions in which the sound response state is maintained continuously are successively determined as the residual state positions. When the sound response state changes or terminates, the corresponding time position is determined as the residual termination position, and the continuous tracking of the current sound response state is stopped. Extract the residual state positions between the continuation position of the sound response and the termination position of the residual state, establish the state succession relationship between adjacent residual state positions in chronological order, and generate a residual state sequence. The position where the sound response continues is taken as the starting boundary of the region, the position where the residue ends is taken as the ending boundary of the region, and the residual state sequence is assigned to the corresponding region to construct the behavior residue region. Based on the start and end positions of the behavioral precursor zone, behavioral manifestation zone, and behavioral residue zone on a unified timeline, a temporal continuity relationship is established between the behavioral precursor zone and the behavioral manifestation zone, and between the behavioral manifestation zone and the behavioral residue zone, generating behavioral region continuity records. Specifically, establishing the temporal continuity relationship between the behavioral precursor zone and the behavioral manifestation zone, and between the behavioral manifestation zone and the behavioral residue zone, is as follows: Read the starting and ending boundaries of the regions corresponding to the precursor region, manifestation region, and residual region of each behavior, and determine the order of each region according to a unified timeline. The regional termination boundary of the behavioral precursor zone is matched with the regional starting boundary of the adjacent behavioral manifestation zone in time. When the two are connected continuously on the same time axis, the first time connection relationship between the behavioral precursor zone and the behavioral manifestation zone is established. The termination boundary of the behavior manifestation area is matched with the starting boundary of the adjacent behavior residue area in time. When the two are connected continuously on the same time axis, a second time connection relationship is established from the behavior manifestation area to the behavior residue area. The corresponding visual action state and sound response state are continuously tracked along the first and second time continuity relationships, and the state continuity direction and state transition position between each region are recorded. The boundaries, first-time continuity, second-time continuity, and corresponding state transition positions of the behavioral precursor zone, behavioral manifestation zone, and behavioral residue zone are associated to generate a behavioral zone continuity record. The corresponding behavioral precursor zone, behavioral manifestation zone, and behavioral residue zone are sequentially connected according to the behavioral region records. The connection results are then arranged based on the visual action state and sound response state corresponding to each region to generate an acoustic-visual behavioral evolution trajectory. Specifically, the generation of the acoustic-visual behavioral evolution trajectory is as follows: Read the behavior precursor area, behavior manifestation area, behavior residue area, first time connection relationship and second time connection relationship in the behavior area connection record, and determine the connection order of each area according to the unified timeline; Connect the behavioral precursor zone and the behavioral manifestation zone along the first time sequence, and establish a continuous state connection between the sound response state at the end of the behavioral precursor zone and the visual action state and sound response state at the beginning of the behavioral manifestation zone. Connect the behavior manifestation area and the behavior residue area along the second time sequence, and establish a continuous state connection between the visual action state and sound response state at the end of the behavior manifestation area and the sound response state at the beginning of the behavior residue area. The visual action state and sound response state in each area are arranged sequentially according to the time position of the behavioral precursor area, behavioral manifestation area and behavioral residue area, and the state transition position corresponding to the boundary of each area is retained to generate a continuous behavioral state chain. The continuous behavioral state chain is associated with the corresponding region start boundary, region end boundary, state transition position and time sequence relationship, and the continuous behavioral state chain is arranged according to a unified time axis to generate the audiovisual behavioral evolution trajectory.

[0030] In this embodiment, the generated behavior branch discrimination result includes: According to the evolution trajectory of sound and visual behavior, read the visual action state and sound response state corresponding to each behavior manifestation area, filter the behavior manifestation areas where the visual action state is continuously maintained, and generate the same posture maintenance segment. The sound response state is continuously tracked along the time direction of the same posture maintenance segment. The frequency distribution, energy distribution and duration of adjacent sound response states are read, and the time position where the sound response state changes is determined as the sound response change position. Based on the sound response transition positions, the corresponding attitude-maintaining segments are segmented. Continuous segments between adjacent sound response transition positions and between the boundary of the attitude-maintaining segment and adjacent sound response transition positions are defined as behavior branches, and each sound response transition position is defined as the boundary of the behavior branch. Specifically, the segmentation of the corresponding attitude-maintaining segments based on each sound response transition position is as follows: Read the region start boundary, region end boundary and sound response transition position within the same attitude maintenance section corresponding to each same attitude maintenance section, and sort the sound response transition positions according to a unified time axis; The continuous segment between the starting boundary of the same posture-maintaining segment and the first sound response transition position is defined as the first segment segment. The continuous segments between adjacent sound response transition positions are sequentially defined as intermediate segment segments, and each intermediate segment segment is recorded in the time order of the sound response transition positions. The continuous segment between the last sound response transition position and the region termination boundary of the same posture maintenance segment is defined as the end segment; Arrange the segmented sections according to the time order of the first segmented section, the middle segmented section, and the last segmented section, and mark the position of each sound response conversion as the segmentation boundary between adjacent segmented sections to complete the segmentation of the corresponding same posture-maintaining section; Based on the start and end positions of each behavior branch on a unified timeline, the corresponding behavior precursor area is traced before the start position of the behavior branch, and the corresponding behavior residue area is traced after the end position of the behavior branch, generating regional association records for each behavior branch. Based on the regional association records, the corresponding behavioral precursor areas, behavioral branches and behavioral residue areas are connected in chronological order to establish the branch-support connection relationship of each behavioral branch, and the corresponding visual action state, sound response state and duration are recorded. Based on the branch-support relationship of each behavior branch, different sound response states corresponding to the same visual action state are distinguished, and each behavior branch is arranged according to the behavior branch boundary to generate behavior branch discrimination results. Specifically, the generation of behavior branch discrimination results is as follows: Read the visual action state, sound response state, behavioral precursor area, behavioral residue area and branch connection relationship corresponding to each behavioral branch, and establish a behavioral branch state record; Filter out behavioral branches with the same visual action state, and compare the frequency distribution, energy distribution, duration, and sound response transition position of the corresponding sound response state to determine the sound response differences between each behavioral branch. Based on the differences in sound response and the state continuity of the corresponding premonition zone and residual zone of behavior, behavioral branches with different sound response states under the same visual action state are distinguished, and behavioral branch distinction records are generated. Arrange the behavioral branches according to their order on a unified timeline, and write the corresponding visual action state, sound response state, behavioral precursor zone, behavioral residue zone and branch connection relationship into the corresponding arrangement position. The behavior branch distinction records are associated with the sorted behavior branches to generate behavior branch discrimination results.

[0031] In this embodiment, generating animal behavior recognition results includes: Read the visual action state, sound response state, behavior branch boundary and duration interval corresponding to each behavior branch in the behavior branch discrimination result, and establish a behavior branch state record according to a unified time axis; Based on the behavior branch state record, the duration interval of each behavior branch is verified, and the duration interval is matched with the start and end positions of the corresponding behavior precursor area, behavior manifestation area and behavior residue area to generate the duration interval verification result. Based on the continuous interval verification results, read the branch support connection relationship between adjacent behavior branches, verify the time continuity state between the behavior residual area of ​​the previous behavior branch and the behavior precursor area of ​​the next behavior branch, and generate the branch support connection verification results. By combining the branch connection verification results with the corresponding sound response states of each behavioral branch, the behavioral branches corresponding to different sound response states under the same visual action state are distinguished. The animal behavior category is then determined based on the correspondence between visual action states and sound response states. Specifically, the determination of the animal behavior category based on the correspondence between visual action states and sound response states is as follows: Read the visual action state and sound response state corresponding to each behavior branch that has passed the branch connection verification, and establish a corresponding record of audio-visual state according to the duration interval of the behavior branch. Based on the body contour, limb key points, posture and direction of movement in the visual-motor state, determine the visual-motor type corresponding to each behavior branch, and establish a correlation between the visual-motor type and the corresponding sound response state. Read the frequency distribution, energy distribution and duration of the sound response state, determine the sound response type of each behavior branch, and match the corresponding visual action type with the corresponding sound and visual state records. Behavioral branches with the same visual action type but different sound response type are established as sound-visual behavior combinations, and the behavior state of each sound-visual behavior combination is determined by combining the state change sequence of the corresponding behavior precursor area, behavior manifestation area and behavior residue area. Based on the visual action type, sound response type and behavioral state corresponding to each combination of sound and visual behavior, behavior categories are matched, and the matching results are determined as the animal behavior category of the corresponding behavior branch. Animal behavior categories are associated with the start and end positions, duration, visual action state, and sound response state of their corresponding behavior branches, and arranged according to a unified timeline to generate animal behavior recognition results.

[0032] refer to Figure 4 An animal behavior recognition system based on audio-visual fusion includes: The data acquisition module is used to collect and preprocess animal video data and synchronized sound data to generate an audiovisual animal behavior dataset. The feature extraction module is used to extract video and sound features based on the acoustic and visual animal behavior dataset, and generate visual feature map sequences and sound time-frequency feature map sequences. The audio-visual fusion module is used to input the visual feature map sequence and the audio time-frequency feature map sequence into the improved VMamba model to generate an audio-visual fusion behavior feature sequence. The behavior evolution module is used to construct the behavior precursor region, behavior manifestation region and behavior residue region based on the acoustic-visual fusion behavior feature sequence, and generate the acoustic-visual behavior evolution trajectory. The branch discrimination module is used to perform branch splitting based on the acoustic-visual behavior evolution trajectory, determine the behavior branch boundary and the branch connection relationship, and generate behavior branch discrimination results. The behavior recognition module is used to verify the status of each behavior branch based on the behavior branch discrimination results and determine the animal behavior category, and generate animal behavior recognition results.

[0033] Example 1: In a closed-system group animal behavior monitoring scenario, the identification device received a batch of raw audio-visual data from video and audio acquisition devices. The video data contained continuous footage of 18 animals behaving in their feeding, drinking, and activity areas, with a resolution of 1920×1080 pixels and a frame rate of 25 frames per second. The synchronous audio data sampling rate was 16000Hz, acquired via a single channel. The initial acquisition time was 6 hours, resulting in 540,000 video frames and 21,600 seconds of audio data. After manual annotation, 7200 behavioral samples were generated, including 1680 feeding behaviors, 1160 drinking behaviors, 980 sniffing behaviors, 1320 walking behaviors, 1100 lying-down behaviors, and 960 contact and fighting behaviors. Of the samples, 1496 segments contained animal occlusion, accounting for 20.8%; 1820 segments were in a similar head-down posture, accounting for 25.3%. Among them, feeding, drinking, and sniffing were visually represented by head drooping, neck forward extension, and body stillness, which could easily lead to misjudgment using traditional visual methods.

[0034] After the data enters the processing flow, the recognition device normalizes the size of the video frames, uniformly scaling the images to 512×512 pixels, and suppresses brightness fluctuations and motion blur. Before processing, the average grayscale contrast ratio of the low-light image was 0.26, which was improved to 0.41 after processing; the availability rate of key point detection increased from 83.7% to 91.4%. Synchronous audio data underwent background noise suppression and amplitude normalization, reducing the average background noise from -31.6dB to -42.8dB, and making the short-time energy boundaries of chewing sounds, drinking sounds, and nasal sniffing sounds clearer. The recognition device establishes a unified timeline based on the video and audio acquisition timestamps, initially pairing video frames with audio segments to generate an audiovisual animal behavior dataset.

[0035] After generating the acoustic-visual animal behavior dataset, the recognition device divided the dataset into 3-second acoustic-visual time windows. Each acoustic-visual time window contained 75 consecutive images and a 3-second synchronized sound segment, resulting in a total of 7200 acoustic-visual time windows. The training set used 5000 windows, the validation set used 1000 windows, and the test set used 1200 windows. Sample T-0264 from the training set shows that the animal's visual behavior is characterized by lowering its head close to the feeding trough, with the head angle changing from 42° to 58° and a body displacement of only 0.18m. A continuous chewing sound is observed in the 1.7-4.2kHz frequency band of the sound clip, with an average energy of 0.64, corresponding to feeding behavior. In sample T-0418, the animal also lowers its head, with a head angle of 55° and a body displacement of 0.12m, but a periodic drinking sound is observed in the 0.35-1.10kHz frequency band of the sound clip, with an average energy of 0.58, corresponding to drinking behavior. In sample T-0732, the visual posture remains head down with the nose close to the ground. A short, sharp airflow sound lasts for 0.4 seconds in the sound clip, with an average energy of 0.23, corresponding to snoring behavior. The visual action states of the three groups of samples are similar, but the sound response states differ significantly, which is used to verify the necessity of branching the different sounds in the same posture.

[0036] In the feature generation stage, the recognition device extracts the animal's body contour, limb key points, posture, and direction of movement from each acoustic-visual time window. Each image frame retains one animal target box, 18 limb key points, one set of posture angles, and one set of direction of movement vectors, which are then spatially mapped to form a visual feature map of size 128×128×64. The sound segment is divided into 96 sound frames, each 40ms long with a 20ms frame shift. After time-frequency transformation, a 96×80 sound time-frequency matrix is ​​obtained, which is then mapped to a 96×80×32 sound time-frequency feature map. In training sample T-0264, the peak time-frequency energy corresponding to the chewing sound is 0.82, and the average visual head-down stable response is 0.77. In training sample T-0418, the peak interval of the drinking sound cycle is 0.31 seconds, and the average visual head-down stable response is 0.75, indicating that visual response alone is insufficient to accurately distinguish between the two types of behavior.

[0037] After entering the improved VMamba model, the bimodal state encoding layer encodes the visual feature map sequence and the sound time-frequency feature map sequence into states respectively. The single-window state dimension of the visual action state sequence is 128, and the single-window state dimension of the sound response state sequence is also 128. The cross-modal time-delay state bridge reads the change position of the two types of states. In training sample T-0264, the sound response state transitions at 0.42 seconds, and the visual action state transitions at 0.91 seconds, calculating that the sound response precedes the visual action by 0.49 seconds. The recognition device shifts the sound response state forward to align with the corresponding visual action state, thus preserving the precursor to the feeding behavior. In sample T-0418, the drinking sound persists for 0.73 seconds after the visual head leaves the drinking device. The cross-modal time-delay state bridge records this sound response state as a continuation state, preventing the drinking behavior from ending prematurely.

[0038] In the occlusion-triggered selective scanning layer processing, there are 248 occlusion windows in the test set. Taking test sample V-0336 as an example, the animal's head region is occluded by adjacent animals from frame 38 to frame 51, and the number of visible keypoints decreases from 18 to 9. Traditional visual methods would segment the feeding behavior into two behavioral segments in this interval. After detecting the location of the visual action state continuity interruption, the improved VMamba model reads the corresponding sound response state. The chewing sound energy remains between 0.61 and 0.68 in the occlusion interval, with the peak frequency stabilizing around 2.6 kHz. The occlusion-triggered selective scanning layer writes the sound response state to the interruption location and propagates the state along the horizontal ascending order, horizontal descending order, vertical ascending order, and vertical descending order. The continuous response of the behavior in the occlusion interval increases from 0.44 to 0.79, and no erroneous segmentation occurs.

[0039] In the audiovisual difference state correction layer processing, the recognition device corrects for intervals where the visual action state is continuously maintained but the sound response state changes. In test sample V-0579, the animal continuously lowered its head for 34.2 seconds, with a stable visual action state response average of 0.81. Traditional visual methods output a single feeding behavior. This method detected that the sound response state changed from chewing sound to drinking sound at 12.6 seconds, and from drinking sound to nasal sniffing sound at 24.8 seconds. Therefore, this continuous head-lowering segment was divided into 3 behavioral branches. The first behavioral branch lasted 12.6 seconds, with a chewing sound average energy of 0.66, outputting feeding; the second behavioral branch lasted 12.2 seconds, with a drinking sound period peak interval of 0.29 seconds, outputting drinking; the third behavioral branch lasted 9.4 seconds, with a short airflow sound average energy of 0.25, outputting sniffing. Manual verification showed that all three branch categories were correct, with an average deviation of 0.28 seconds at the branch boundaries.

[0040] During the behavioral evolution trajectory construction stage, the recognition device identifies the preceding position of sound response, the position of simultaneous sound and visual action maintenance, and the position of continuation of sound response based on the sound-visual fusion behavioral feature sequence. In test sample V-0112, the preceding position of sound response for feeding behavior was at 0.36 seconds, the position of visual action transition was at 0.88 seconds, and the behavioral precursor zone lasted for 0.52 seconds; the position of simultaneous sound and visual action maintenance lasted from 0.88 seconds to 18.70 seconds, and the behavioral manifestation zone lasted for 17.82 seconds; after the visual action ended, the chewing sound continued until 19.56 seconds, and the behavioral residue zone lasted for 0.86 seconds. The feeding start error recorded by the traditional visual method was 0.94 seconds, and the end error was 1.21 seconds; the starting error of this method was 0.18 seconds, and the end error was 0.24 seconds.

[0041] In the comparative experiment, the traditional method uses visual behavior recognition based solely on video frames, while this method employs an audio-visual fusion behavior recognition method. Both methods used the same 5000 training samples, 1000 validation samples, and 1200 test samples. On the same batch of test data, the overall recognition accuracy of the traditional method was 82.6%, while that of this method was 94.1%; the traditional method's accuracy for recognizing feeding behavior was 80.4%, while that of this method was 95.2%; the traditional method's accuracy for recognizing drinking behavior was 77.5%, while that of this method was 92.9%; the traditional method's accuracy for recognizing sniffing behavior was 76.8%, while that of this method was 91.8%; the traditional method's accuracy for recognizing occluded windows was 72.9%, while that of this method was 88.7%; the traditional method's accuracy for recognizing head-down similarity... The pose window recognition accuracy was 78.1%, while our method achieved 93.0%. The average error of the starting position of the behavior was 1.08 seconds in the traditional method, while it was 0.23 seconds in our method. The average error of the ending position of the behavior was 1.35 seconds in the traditional method, while it was 0.32 seconds in our method. The traditional method merged 72 adjacent behavior errors, which was reduced to 15 in our method. The traditional method split 64 single behavior errors, which was reduced to 19 in our method. The traditional method involved 105 mutual judgments between feeding, drinking, and sniffing, which was reduced to 27 in our method.

[0042] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An animal behavior recognition method based on acoustic-visual fusion, characterized in that, include: Collect animal video data and synchronized audio data, perform preprocessing on the animal video data and synchronized audio data, and generate an audiovisual animal behavior dataset; Based on the acoustic-visual animal behavior dataset, continuous video frames and synchronized sound segments are extracted, and the synchronized sound segments are converted into time-frequency spectra to generate visual feature map sequences and sound time-frequency feature map sequences. The visual feature map sequence and the sound time-frequency feature map sequence are input into the improved VMamba model, which includes a bimodal state coding layer, a cross-modal time-delay state bridge, an occlusion-triggered selective scanning layer, and an acoustic-visual difference state correction layer to generate an acoustic-visual fusion behavioral feature sequence. Based on the sound and visual fusion behavior feature sequence identification, the sound response precursor position, the sound and visual synchronization maintenance position, and the sound response continuation position are identified. Behavior precursor region, behavior manifestation region, and behavior residue region are constructed respectively. The regions are connected according to the time sequence to generate the sound and visual behavior evolution trajectory. Based on the acoustic-visual behavior evolution trajectory, the same posture different sound branch is split. The sound response state is tracked in the behavior manifestation area. The sound response conversion position is determined as the behavior branch boundary. The branch connection relationship is established by combining the corresponding behavior precursor area and behavior residual area to generate behavior branch discrimination results. Based on the behavioral branch discrimination results, the duration interval, branch connection relationship and sound response state of each behavioral branch are verified to determine the animal behavior category and generate animal behavior recognition results.

2. The animal behavior recognition method based on acoustic-visual fusion according to claim 1, characterized in that, The animal video data includes continuous animal image frames, animal target location, animal body outline, key points of animal limbs, animal posture, animal movement direction, and video acquisition timestamp. The synchronized audio data includes animal calls, chewing sounds, drinking sounds, breathing sounds, activity sounds, and audio acquisition timestamp.

3. The animal behavior recognition method based on acoustic-visual fusion according to claim 1, characterized in that, The preprocessing of animal video data and synchronized audio data includes establishing a unified timeline based on the video acquisition timestamp and the audio acquisition timestamp, performing frame rate unification, image size normalization and image noise suppression on the animal video data, performing sampling rate unification, background noise suppression and amplitude normalization on the synchronized audio data, and performing initial time pairing of continuous image frames and synchronized audio segments according to the unified timeline to generate an audiovisual animal behavior dataset.

4. The animal behavior recognition method based on acoustic-visual fusion according to claim 1, characterized in that, The generated visual feature map sequence and sound time-frequency feature map sequence include: Read continuous image frames and synchronous sound segments from the acoustic-visual animal behavior dataset according to a unified timeline, divide the acoustic-visual time window according to the acquisition time period corresponding to the continuous image frame, and generate an acoustic-visual time window sequence. Based on the audiovisual time window sequence, continuous image frames are extracted window by window. Animal image regions are determined according to the animal target location and animal body outline. The animal image regions are then arranged in chronological order to generate an animal visual frame sequence. Based on animal visual frame sequences, animal body contours, limb key points, posture and movement direction are extracted. Visual information corresponding to the same acquisition time period is combined according to spatial location to generate a visual feature map sequence. Based on the audiovisual time window sequence, synchronous sound segments corresponding to each animal visual frame sequence are extracted from the synchronous sound data. Framing and windowing processing are performed on each synchronous sound segment, and they are arranged in chronological order to generate an animal sound frame sequence. Perform time-frequency transformation on the animal sound frame sequence to extract the frequency distribution, energy distribution and time change information of each sound frame, establish the correspondence according to the sound-visual time window sequence, and generate a sound time-frequency feature map sequence.

5. The animal behavior recognition method based on acoustic-visual fusion according to claim 1, characterized in that, The generated audio-visual fusion behavior feature sequence includes: An improved VMamba model is constructed based on the visual state space structure of the VMamba model. The input embedding, two-dimensional selective scanning, state space transfer and output mapping structures are retained. A bimodal state coding layer is set in parallel in the input embedding structure. A cross-modal time-delay state bridge is set in front of the two-dimensional selective scanning structure. An occlusion-triggered selective scanning layer is set inside the two-dimensional selective scanning structure. An acoustic-visual difference state correction layer is set between the state space transfer structure and the output mapping structure. The visual feature map sequence and the sound time-frequency feature map sequence are input into the bimodal state coding layer, and feature embedding and state coding are performed respectively. The coding results are arranged according to the sound-visual time window sequence to generate a visual action state sequence and a sound response state sequence. Based on the visual action state sequence and the sound response state sequence, the positions of continuous state changes are extracted, and the time offset of the sound response state relative to the visual action state is determined according to a unified time axis to generate an acoustic-visual time delay relationship sequence. Input the acoustic-visual time delay relationship sequence into the cross-modal time delay state bridge, adjust the time position corresponding to the sound response state according to the time offset, establish a cross-time connection relationship with the corresponding visual action state, and generate a time delay calibration state sequence. The location of visual action state continuity interruption is determined based on the time-delay calibration state sequence. The corresponding sound response state is read by selecting the scanning layer through occlusion trigger, and the corresponding sound response state is written into the continuity interruption location and transmitted along the two-dimensional selected scanning direction to generate an occlusion compensation state sequence. Based on the occlusion compensation state sequence, a continuous segment is determined where the visual action state is maintained and the sound response state changes. The corresponding visual action state is corrected by the sound-visual difference state correction layer according to the sound response state change, and a difference correction state sequence is generated. The time-delay calibration state sequence, occlusion compensation state sequence, and difference correction state sequence are fused along a unified time axis, and the fusion results are arranged in the order of state succession to generate an audiovisual fusion behavior feature sequence.

6. The animal behavior recognition method based on acoustic-visual fusion according to claim 1, characterized in that, The generated acoustic-visual behavioral evolution trajectory includes: Read the visual action state and sound response state in the audio-visual fusion behavior feature sequence according to a unified time axis, extract the state change direction and state persistence relationship of adjacent time positions, and generate audio-visual state change record. Based on the recording of changes in acoustic and visual states, the starting position where the acoustic response state changes while the visual action state remains continuous is determined. This starting position is defined as the acoustic response precursor position, and the system continuously tracks the position where the visual action state changes to construct a behavioral precursor zone. Continue tracking the visual action state and sound response state from the end of the behavioral precursor zone, and determine the continuous positions where the two simultaneously maintain the corresponding state as the sound-visual synchronization maintenance positions, and form the behavioral manifestation zone by the continuous sound-visual synchronization maintenance positions. Continue tracking the visual action state and sound response state from the end of the behavior manifestation area, determine the starting position where the visual action state ends and the sound response state continues, determine the starting position as the sound response continuation position, and construct the behavior residue area along the direction of the sound response state continuation. Based on the start and end positions of the behavioral precursor zone, behavioral manifestation zone, and behavioral residue zone on a unified timeline, establish the temporal continuity relationship from the behavioral precursor zone to the behavioral manifestation zone and from the behavioral manifestation zone to the behavioral residue zone, and generate behavioral region continuity records. The corresponding behavioral precursor area, behavioral manifestation area, and behavioral residue area are sequentially connected according to the behavioral region records. The connection results are arranged according to the visual action state and sound response state corresponding to each region to generate the audiovisual behavioral evolution trajectory.

7. The animal behavior recognition method based on acoustic-visual fusion according to claim 1, characterized in that, The generation behavior branch discrimination result includes: According to the evolution trajectory of sound and visual behavior, read the visual action state and sound response state corresponding to each behavior manifestation area, filter the behavior manifestation areas where the visual action state is continuously maintained, and generate the same posture maintenance segment. The sound response state is continuously tracked along the time direction of the same posture maintenance segment. The frequency distribution, energy distribution and duration of adjacent sound response states are read, and the time position where the sound response state changes is determined as the sound response change position. Based on each sound response transition position, the corresponding attitude-maintaining segment is divided. The continuous segments between adjacent sound response transition positions and between the boundary of the attitude-maintaining segment and adjacent sound response transition positions are determined as behavior branches, and each sound response transition position is determined as the boundary of the behavior branch. Based on the start and end positions of each behavior branch on a unified timeline, the corresponding behavior precursor area is traced before the start position of the behavior branch, and the corresponding behavior residue area is traced after the end position of the behavior branch, generating regional association records for each behavior branch. Based on the regional association records, the corresponding behavioral precursor areas, behavioral branches and behavioral residue areas are connected in chronological order to establish the branch-support connection relationship of each behavioral branch, and the corresponding visual action state, sound response state and duration are recorded. Based on the branch-support connection relationship of each behavior branch, different sound response states corresponding to the same visual action state are distinguished, and each behavior branch is arranged according to the boundary of the behavior branch to generate behavior branch discrimination results.

8. The animal behavior recognition method based on acoustic-visual fusion according to claim 1, characterized in that, The generated animal behavior recognition results include: Read the visual action state, sound response state, behavior branch boundary and duration interval corresponding to each behavior branch in the behavior branch discrimination result, and establish a behavior branch state record according to a unified time axis; Based on the behavior branch state record, the duration interval of each behavior branch is verified, and the duration interval is matched with the start and end positions of the corresponding behavior precursor area, behavior manifestation area and behavior residue area to generate the duration interval verification result. Based on the continuous interval verification results, read the branch support connection relationship between adjacent behavior branches, verify the time continuity state between the behavior residual area of ​​the previous behavior branch and the behavior precursor area of ​​the next behavior branch, and generate the branch support connection verification results. By combining the results of the branch connection verification with the sound response state corresponding to each behavioral branch, the behavioral branches corresponding to different sound response states under the same visual action state are distinguished, and the animal behavior category is determined based on the correspondence between the visual action state and the sound response state. Animal behavior categories are associated with the start and end positions, duration, visual action state, and sound response state of their corresponding behavior branches, and arranged according to a unified timeline to generate animal behavior recognition results.

9. An animal behavior recognition system based on acoustic-visual fusion, comprising the animal behavior recognition method based on acoustic-visual fusion as described in any one of claims 1 to 8, characterized in that, include: The data acquisition module is used to collect and preprocess animal video data and synchronized sound data to generate an audiovisual animal behavior dataset. The feature extraction module is used to extract video and sound features based on the acoustic and visual animal behavior dataset, and generate visual feature map sequences and sound time-frequency feature map sequences. The audio-visual fusion module is used to input the visual feature map sequence and the audio time-frequency feature map sequence into the improved VMamba model to generate an audio-visual fusion behavior feature sequence. The behavior evolution module is used to construct the behavior precursor region, behavior manifestation region and behavior residue region based on the acoustic-visual fusion behavior feature sequence, and generate the acoustic-visual behavior evolution trajectory. The branch discrimination module is used to perform branch splitting based on the acoustic-visual behavior evolution trajectory, determine the behavior branch boundary and the branch connection relationship, and generate behavior branch discrimination results. The behavior recognition module is used to verify the status of each behavior branch based on the behavior branch discrimination results and determine the animal behavior category, and generate animal behavior recognition results.

Images