A dangerous animal detection and early warning method and device based on sound guidance

Abstract

The embodiment provides a dangerous animal detection and early warning method and device based on sound guidance, introduces an audio-visual contrast learning method to supervise network learning. Through a learnable linear mapping layer, audio and visual two modal features are mapped into a multimodal joint space to realize alignment constraint, so that the network can accurately understand that the same target may have multiple forms of sound features, thereby giving the network the ability to consistently recognize two modalities at the feature level. Further, in order to deeply interact audio and video features, so that the visual features can be related to the specific dangerous animal object by means of the audio features, at the same time, the audio features can be related to the specific frequency sound signal by means of the visual features, the application proposes an audio-visual attention mechanism. Through realizing deep fusion of two modal features, a video segmentation network can obtain a good prior feature, effectively suppresses an undesirable inductive bias, and improves recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and device for detecting and warning of dangerous animals based on sound guidance, a computer device, and a storage medium. Background Technology

[0002] Encounters with wildlife during outdoor adventures and sports can pose potential safety risks. Especially in the wild, people may unexpectedly encounter dangerous animals such as venomous snakes and carnivorous mammals, potentially resulting in attacks and injuries. Therefore, developing an intelligent system capable of timely identifying and issuing warnings about dangerous animals is crucial for the safety of outdoor enthusiasts.

[0003] Currently, many methods for identifying dangerous animals still rely on visual or olfactory perception, but these methods have certain limitations. Visual perception can be affected by factors such as ambient light, angle, and obstruction, while olfactory perception requires sophisticated equipment and is subject to conditions such as wind direction. Therefore, a more reliable and practical method is needed to identify dangerous animals and ensure the safety of people outdoors.

[0004] Dangerous animal detection networks based on sound-guided video object segmentation networks, combining sound and visual information, promise to overcome the limitations of traditional methods. On one hand, using Transformer-based video object segmentation networks can accurately segment animal objects in images or videos, thus achieving more accurate animal identification. On the other hand, since dangerous animals often emit specific sounds, fusing information from these sound modalities can further improve the detection rate of dangerous animals.

[0005] Unfortunately, firstly, existing methods struggle to achieve effective alignment between visual modalities and audio features due to the difficulty in modeling representations of different modalities. Secondly, audio contains noise elements of varying frequencies, providing inaccurate object reference information and reducing the accuracy of network recognition. Furthermore, in complex natural scenes, the color and texture of some dangerous animals are similar to the surrounding background, leading to visual similarity. When using networks constrained by existing mask loss mechanisms, insufficient pixel-level mask constraints result in low pixel confidence, easily leading to confusion with the background and reducing detection accuracy and reliability. These problems significantly hinder the development of dangerous animal recognition technology based on sound-guided video object segmentation networks. They impede network optimization training, resulting in insufficiently precise object localization and severely impacting the practical performance of this method in outdoor sports applications. Summary of the Invention

[0006] In view of the above problems, this embodiment is proposed to provide a sound-guided dangerous animal detection and early warning method, a sound-guided dangerous animal detection and early warning device, a computer device, and a storage medium to overcome or at least partially solve the above problems;

[0007] This paper proposes a sound-guided Transformer video segmentation network based on an audio-visual contrastive learning method and a bidirectional attention mechanism. Utilizing both visual and auditory information, it detects dangerous objects outdoors and transmits warning messages back to the backend via BeiDou terminal SMS, serving backend detection personnel and other terminal users. First, this invention introduces an audio-visual contrastive learning method to supervise network learning. Through a learnable linear mapping layer, audio and visual modal features are mapped to a multimodal joint space to achieve alignment constraints, enabling the network to accurately understand that the same target may have multiple forms of sound features, thus giving the network the ability to consistently recognize both modalities at the feature level. Furthermore, to achieve deep interaction between audio and video features, allowing visual features to focus on specific dangerous animals with the help of audio features, and simultaneously, audio features to focus on specific frequency sound signals with the help of visual features, this invention proposes an audio-visual attention mechanism. By achieving deep fusion of the two modal features, the video segmentation network can obtain a good prior feature set, effectively suppressing unwanted inductive bias and improving recognition accuracy. Finally, to alleviate the problem that existing loss methods are insufficient in constraining pixel-level masks and have low pixel confidence, leading to confusion between dangerous animals and the background, this invention proposes an improved cross-entropy loss method. This method theoretically delves into the loss gradient value to ensure dynamic adjustment of the mask's perceived boundary and global internal structure, thereby achieving more accurate and reliable identification and early warning of dangerous animals.

[0008] To address the aforementioned problems, this embodiment discloses a sound-guided method for detecting and issuing early warnings of dangerous animals, including:

[0009] Step 1: Constructing a Dangerous Animal Feature Dataset: Collect image or video data related to dangerous animals, and simultaneously collect their corresponding sound source data to construct a comprehensive animal feature dataset. To ensure effective training, validation, and evaluation of the network, the collected data is grouped into training, testing, and validation sets. Furthermore, based on the audio information of the corresponding animal species, accurate ground truth masking is performed on dangerous animal targets in the image or video data.

[0010] Step 2, Multimodal Feature Encoding: Input the video to be detected and the corresponding audio information into the multimodal coding network to extract the corresponding visual and audio features;

[0011] The multimodal coding network includes two branches: a video image extraction branch and an audio branch. Deep features are extracted from the original video images and original audio. Specifically, visual information is processed by a pre-trained visual encoder to extract multi-scale visual features, which are then concatenated to form global visual features after unifying the channel dimensions through a designed learnable linear layer. Audio information is processed by a pre-trained audio encoder and then unified with the visual features through a designed learnable linear layer to form an audio representation in the feature space.

[0012] Step 3: Multimodal Feature Fusion and Interaction: The extracted visual and audio features are input into a designed multimodal feature fusion and interaction module. This involves a bidirectional, multi-head cross-modal attention mechanism for the audio-visual modality, enabling full interaction between the two modal features, enhancing intermodal correlation and matching of key features, and obtaining audio-aware visual features and visual-aware audio features.

[0013] Step 4, Dangerous Animal Detection and Segmentation: The fused and interactive visual features with audio awareness and visual features with audio awareness are fed into a Transformer-based video object segmentation network. The visual features are processed frame by frame, and then a re-identification post-processing module is used to locate the identified object to obtain the identification and detection results of dangerous animals in continuous video frames.

[0014] Step 5, Network Training and Optimization: In a training batch, for example object features obtained from the Transformer segmentation network, positive and negative samples are further constructed with audio features extracted by the multimodal coding network in the same batch. This constructs audio-visual contrastive learning to promote alignment between different modalities. Furthermore, for the detection results output from the Transformer segmentation network, this invention uses an improved cross-entropy loss to constrain the detection mask.

[0015] Step Six: Transmission and Early Warning of Detection Results: The detection results obtained from the network will be transmitted back to the backend using Beidou terminals, serving the backend detection personnel and other terminal users.

[0016] As a preferred improvement of the present invention, in step one, the training set, test set and validation set are divided in a ratio of 7:1.5:1.5.

[0017] As a preferred improvement of the present invention, in step one, the videos in the database possess environmental diversity and lighting complexity to ensure that the trained network has strong generalization ability. Within a video, there is one or more dangerous animals, and for each target species, there are one or more sound source data points, which are paired with the video data.

[0018] As a preferred improvement of the present invention, in step two, the visual encoder may include, but is not limited to, a visual extraction backbone pre-trained on a large-scale dataset such as the ResNet series and the ViT series; the audio feature encoder may include, but is not limited to, an audio extraction backbone pre-trained on a large-scale dataset such as the Vgg series and the ResNet series.

[0019] As a preferred improvement of the present invention, in step two, a set of video and audio inputs are given. video sequence Where T is the number of video frames. i represents the i-th frame in the video; the audio source. S represents the number of samples when the audio sampling frequency is 16000. The visual encoder extracts multi-scale visual features from each frame of the input image, and then unifies the dimension C through a linear layer to obtain the visual features. The audio encoder extracts feature vectors from the input audio source, then passes them through two linear layers and one Dropout layer to obtain the audio features. Where C is the number of feature channels, L v and L a Let be the sequence length of each of the two modal features.

[0020] As a preferred improvement of the present invention, in step three, the extracted visual features and audio features The input is fed into a visual-audio modal feature fusion interaction module of the design. Specifically, firstly, visual features... and audio features The visual query value is obtained by normalizing through a LayerNorm layer and then passing it through a set of learnable linear projection matrices. and audio key values The similarity matrix A is calculated using visual features as the query. v2a :

[0021]

[0022] By transposing the matrix, we obtain another similarity matrix A with audio features as query values. a2v :

[0023]

[0024] Then, the normalized visual features are respectively processed using a set of learnable linear projection matrices. and audio features Mapping yields numerical values and Then, perform a cross product with the similarity matrix obtained by the Softmax function:

[0025]

[0026]

[0027] Here, Softmax(*) represents the Softmax function. Finally, the obtained visual features... and audio features and normalized visual features and audio features Perform residual operations to obtain audio-conscious visual features. and audio features with visual awareness

[0028]

[0029]

[0030] As a preferred improvement of the present invention, in step four, visual features with audio awareness are... and audio features with visual awareness The input is fed into a Transformer-based video object segmentation network, where the Transformer encoding network operates on audio-conscious visual features. After further deep encoding, it is fed into the Transformer decoding network and queried through N learnable instances. Decode visual features to capture target objects.

[0031] As a preferred improvement of the present invention, in step four, in order to calculate whether the predicted target is a dangerous animal, N learnable instances are queried. Will be associated with visually conscious audio features The matching score S is calculated. ref Specifically, firstly, through average pooling operations, visually conscious audio features are obtained. Extract audio feature vectors with a sequence length of 1, and compare them with instance queries. The features are cross-producted as follows:

[0032]

[0033] Here, Avg(*) represents the average pooling operation.

[0034] As a preferred improvement of the present invention, in step five, the instance query in the Transformer segmentation network is processed through a multilayer perceptron consisting of two linear layers and one ReLU activation function layer. Capture the query features of the target and audio features The target query embedding ε is obtained by projecting it onto the multimodal joint space. ins and audio query embedding ε a The details are as follows:

[0035]

[0036]

[0037] Where Avg(*) represents the average pooling operation, and MLP(*) represents a multilayer perceptron consisting of two linear layers and one ReLU activation function, used to map embedded features from a low-dimensional representation space to a high-dimensional multimodal joint embedding space.

[0038] As a preferred improvement of the present invention, in step five, in order to achieve alignment between different modalities and improve the accuracy of the network in matching sound sources and target objects, the present invention, based on the proposed method for constructing positive and negative samples of audio-visual pairs, implements comparative learning between sound source visions. Specifically, in a training batch, for N in a video frame I... O Different instance objects From each instance object N corresponding to the type A Randomly select one audio source from the sound sources As positive samples. In addition to selecting the audio sources corresponding to other types of instance objects in the same frame as negative samples, other audio sources of different types with similar sound source features are randomly selected from other video frames as negative samples to increase the training difficulty of the network's contrastive learning and encourage the network to capture more robust latent feature connections between audio modalities and visual modalities.

[0039] As a preferred improvement of the present invention, in step five, after constructing audio-visual positive and negative sample pairs and obtaining them by projecting them into a multimodal joint space, the following contrastive loss function l is used. embed Supervised network training:

[0040]

[0041] Where, N batch denoted by the size of the training batch, i and j represent the indices of the embedded features, S(*), * represents the cosine similarity between two embedded features, τ represents the adjustable temperature coefficient, log(*) represents the logarithm, and exp(*) represents the natural constant e raised to the power of *.

[0042] As a preferred improvement of this invention, in step five, for the mask of the Transformer segmentation network output, this invention uses an improved cross-entropy loss for constraint optimization. By introducing two optimization schemes to improve the existing cross-entropy loss, the direction of network training is dynamically changed, alleviating the problem of excessively low confidence due to overall gradient suppression. Specifically, given the predicted P of the segmentation network output... n and the true label G t The existing Focal loss can be expressed as:

[0043] l focal =-(1-P t ) γ log(P t );

[0044] Among them, the prediction probability γ is the modulation factor, γ∈[0,5]. focal When encountering difficult pixels, When encountering easy pixels, However, in the later stages of network training, there is a problem that the color and texture of some dangerous animals are similar to the surrounding background. Existing Focal loss still inappropriately focuses on difficult and simple pixels, ignoring the overall optimization of the prediction results, making it difficult for the network to converge further. First, this invention introduces a global adjustment coefficient β to dynamically reflect the overall situation of pixel prediction. Assuming that all pixels are predicted correctly, the prediction probability P... t =1, denoted as P r This is divided by the current predicted probability. The predicted probability for each pixel is expanded to the power of γ, which can be expressed as:

[0045]

[0046] Here, H and W represent the length and width of the current video frame I. Secondly, this invention further introduces a local modulation factor α. First, the gradient distribution of the Focal loss is observed by performing a Taylor expansion:

[0047]

[0048] Differentiating the above expression, we get:

[0049]

[0050] Therefore, except for the first term, the gradient values ​​of the other terms decrease exponentially. Thus, the first term plays a dominant role in the backpropagation of network optimization. The number of difficult pixels is often less than the number of simple pixels, and the gradient changes in the later stages of training mainly come from simple pixels, which slows down the convergence speed in the later stages of training. Therefore, this invention introduces a modulation factor α∈(0,2) to reduce the weight value of the first term, which can be expressed as:

[0051]

[0052] Integrating the global adjustment coefficient β, we can obtain the final improved cross-entropy loss:

[0053]

[0054] As a preferred improvement of the present invention, in step six, the mask result of network detection is stored in the entity class of the program using polygon representation or RLE compression representation, and after setting attributes such as detection category, the warning message is sent back to the background, including but not limited to inter-process communication such as TCP, UDP, overlay network, RPC remote call, message queue and other indirect communication methods.

[0055] This embodiment discloses a sound-guided dangerous animal detection and early warning device, including:

[0056] The dangerous animal feature dataset construction module collects image or video data related to dangerous animals, and simultaneously gathers their corresponding sound source data to construct a comprehensive animal feature dataset. To ensure effective training, validation, and evaluation of the network, the collected data is grouped into training, testing, and validation sets. Furthermore, based on the audio information of the corresponding animal species, accurate ground truth masking is performed on dangerous animal targets in the image or video data.

[0057] Multimodal feature encoding module: Input the video to be detected and the corresponding audio information into the multimodal encoding network to extract the corresponding visual and audio features.

[0058] The multimodal coding network includes two branches: a video image extraction branch and an audio branch. Deep features are extracted from the original video images and original audio. Specifically, visual information is processed by a pre-trained visual encoder to extract multi-scale visual features, which are then concatenated to form global visual features after unifying the channel dimensions through a designed learnable linear layer. Audio information is processed by a pre-trained audio encoder, which unifies the dimensions with the visual features through a designed learnable linear layer, forming an audio representation in the feature space.

[0059] Multimodal Feature Fusion Interaction Module: The extracted visual and audio features are input into a designed multimodal feature fusion interaction module. Specifically, a bidirectional multi-head cross-modal attention mechanism for the audio-visual modality is used to achieve full interaction between the two modal features, enhancing the correlation and matching of key features between modalities, thereby obtaining audio-aware visual features and visual-aware audio features.

[0060] Dangerous animal detection and segmentation module: The fused interactive visual features with audio awareness and visual features with audio awareness are fed into a Transformer-based video object segmentation network. The visual features are processed frame by frame, and then a re-identification post-processing module is used to locate the identified object to obtain the identification and detection results of dangerous animals in continuous video frames.

[0061] Network Training and Optimization Module: In a training batch, for example object features obtained from the Transformer segmentation network, positive and negative samples are further constructed with audio features extracted by the multimodal coding network in the same batch. This constructs audio-visual contrastive learning, promoting alignment between different modalities. Furthermore, for the detection results output from the Transformer segmentation network, this invention uses an improved cross-entropy loss to constrain the detection mask.

[0062] Detection result transmission and early warning module: For the detection results obtained from the network, the early warning message will be sent back to the backend with the help of the Beidou terminal, serving the backend detection personnel and other terminal users.

[0063] This embodiment also discloses a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above-described steps of sound-guided dangerous animal detection and early warning.

[0064] This embodiment also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described steps of sound-guided dangerous animal detection and early warning.

[0065] This embodiment has the following advantages:

[0066] In this embodiment of the invention, the dangerous animal detection network based on a sound-guided video object segmentation network, building upon the Transformer-based video object segmentation network, can fuse and process sound information from animal sound sources, thereby achieving more accurate identification of dangerous animals. Deep interaction between audio and video features allows visual features to target specific dangerous animal objects with the help of audio features, and conversely, audio features can target specific frequency sound signals with the help of visual features. Based on the reasonable construction of positive and negative samples, a learnable linear mapping layer maps the audio and visual modal features into a multimodal joint space, achieving alignment constraints and effectively supervising the network to accurately understand that the same target may have multiple forms of sound features, thus giving the network the ability to consistently recognize the two modalities at the feature level. This alleviates the problem of insufficient pixel-level mask constraints in existing loss mechanisms, leading to low pixel confidence in the mask, optimizes the learning and convergence in the later stages of network training, improves the network's adaptability in complex lighting scenarios, and reduces the detection network's missed and false detections of dangerous animals. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in this embodiment, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart illustrating the steps of an embodiment of a sound-guided method for detecting and warning of dangerous animals.

[0069] Figure 2 This is a schematic diagram of the detection network structure of the dangerous animal detection and early warning method based on sound-guided video object segmentation network of the present invention;

[0070] Figure 3 This is a schematic diagram of the visual-audio modality feature fusion interaction module of the dangerous animal detection and early warning method based on sound-guided video object segmentation network of the present invention;

[0071] Figure 4 This is a schematic diagram illustrating the improved cross-entropy loss effect of the dangerous animal detection and early warning method based on sound-guided video object segmentation network of the present invention.

[0072] Figure 5 This is a schematic diagram showing the test results of the dangerous animal detection and early warning method based on sound-guided video object segmentation network of the present invention;

[0073] Figure 6 This is a structural block diagram of an embodiment of a sound-guided dangerous animal detection and early warning device.

[0074] Figure 7 This is an internal structural diagram of a computer device according to one embodiment. Detailed Implementation

[0075] To make the technical problems, technical solutions, and beneficial effects solved by this embodiment clearer, the following detailed description of this embodiment is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0076] The sound-guided dangerous animal detection and early warning method provided in this embodiment can be applied to application environments that include terminals and servers. The terminal communicates with the server via a network. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be a standalone server or a server cluster consisting of multiple servers.

[0077] Reference Figure 1 The diagram illustrates a step-by-step flowchart of an embodiment of a sound-guided method for detecting and warning of dangerous animals, which may include the following steps:

[0078] Step 1: Constructing a Dangerous Animal Feature Dataset: Collect image or video data related to dangerous animals, and simultaneously collect their corresponding sound source data. Group the collected data into training, testing, and validation sets, and store them in COCO format. In a video, there must be one or more dangerous animals, and for each target species, there must be one or more sound source data points, paired with the video data.

[0079] Step 2, Multimodal Feature Encoding: Input the video to be detected and the corresponding audio information into the multimodal coding network to extract the corresponding visual and audio features.

[0080] The multimodal feature encoding is referred to Figure 2 As shown, the multimodal coding network comprises two branches: a visual coding network and an audio coding network, which extract deep features from the original video images and the original audio, respectively. The optional visual encoder includes, but is not limited to, visual extraction backbones pre-trained on large-scale datasets such as the ResNet series and ViT series; the optional audio feature encoder includes, but is not limited to, audio extraction backbones pre-trained on large-scale datasets such as the Vgg series and ResNet series.

[0081] Given a set of video and audio inputs video sequence Where T is the number of video frames. i represents the i-th frame in the video; the audio source. S represents the number of samples when the audio sampling frequency is 16000. The visual encoder extracts multi-scale visual features from each frame of the input image, and then unifies the dimension C through a linear layer to obtain the visual features. The audio encoder extracts feature vectors from the input audio source, then passes them through two linear layers and one Dropout layer to obtain the audio features. Where C is the number of feature channels, L v and L a Let be the sequence length of each of the two modal features.

[0082] Step 3: Multimodal feature fusion interaction: Input the extracted visual and audio features into a designed visual-audio modal feature fusion interaction module.

[0083] The visual-audio modality feature fusion interaction module is described in [reference needed]. Figure 3 As shown. Specifically, the visual features extracted from multimodal feature encoding. and audio features The visual query value is obtained by normalizing through a LayerNorm layer and then passing it through a set of learnable linear projection matrices. and audio key values The similarity matrix A is calculated using visual features as the query. v2a :

[0084]

[0085] By transposing the matrix, we obtain another similarity matrix A with audio features as query values. a2v :

[0086]

[0087] Then, the normalized visual features are respectively processed using a set of learnable linear projection matrices. and audio features Mapping yields numerical values and Then, perform a cross product with the similarity matrix obtained by the Softmax function:

[0088]

[0089]

[0090] Here, Softmax(*) represents the Softmax function. Finally, the obtained visual features... and audio features and normalized visual features and audio features Perform residual operations to obtain audio-conscious visual features. and audio features with visual awareness

[0091]

[0092]

[0093] Step 4, Dangerous Animal Detection and Segmentation: The fused and interactive audio-conscious visual features and audio-conscious visual features are fed into a Transformer-based video object segmentation network. The visual features are processed frame by frame, and then a re-identification post-processing module is used to locate the identified object, so as to obtain the identification and detection results of dangerous animals in continuous video frames.

[0094] The process for detecting and segregating dangerous animals is described in [reference]. Figure 1 As shown. Visual features with audio awareness. and audio features with visual awareness The input is fed into a Transformer-based video object segmentation network, where the Transformer encoding network operates on audio-conscious visual features. After further deep encoding, it is fed into the Transformer decoding network and queried through N learnable instances. Decode visual features to capture the target object. To determine whether the predicted target is a dangerous animal, query N learnable instances. Will be associated with visually conscious audio features The matching score S is calculated. ref Specifically, firstly, through average pooling operations, visually conscious audio features are obtained. Extract audio feature vectors with a sequence length of 1, and compare them with instance queries. The features are cross-producted as follows:

[0095]

[0096] Here, Avg(*) represents the average pooling operation.

[0097] Step 5, Network Training and Optimization: In a training batch, for the example object features obtained from the Transformer segmentation network, positive and negative samples will be constructed with the audio features extracted by the multimodal coding network in the same batch to construct audio-visual contrast learning and promote alignment between different modalities.

[0098] On the one hand, for instance querying in Transformer segmentation networks Capture the query features of the target and audio features The target query embedding ε is obtained by projecting a multi-layer perceptron consisting of two linear layers and one ReLU activation function layer onto a multimodal joint space. ins and audio query embedding ε a The details are as follows:

[0099]

[0100]

[0101] Here, Avg(*) represents the average pooling operation, and MLP(*) represents a multilayer perceptron consisting of two linear layers and one ReLU activation function, used to map embedded features from a low-dimensional representation space to a high-dimensional multimodal joint embedding space. To promote alignment between different modalities and improve the accuracy of network matching sound sources and target objects, this invention, based on the proposed method of constructing positive and negative samples for audio-visual pairs, implements comparative learning between sound source and visual representations. Specifically, in a training batch, for a video frame I with N... O Different instance objects From each instance object N corresponding to the type A Randomly select one audio source from the sound sources As positive samples, in addition to selecting audio sources corresponding to other types of instance objects in the same frame as negative samples, audio sources of different types with similar sound source features are randomly selected from other video frames as negative samples. This increases the training difficulty of the network's contrastive learning and encourages the network to capture more robust latent feature connections between audio and visual modalities. After constructing audio-visual positive and negative sample pairs and projecting them into the multimodal joint space, the following contrastive loss function is used. embed Supervised network training:

[0102]

[0103] Where, N batch denoted by the size of the training batch, i and j represent the indices of the embedded features, S(*), * represents the cosine similarity between two embedded features, τ represents the adjustable temperature coefficient, log(*) represents the logarithm, and exp(*) represents the natural constant e raised to the power of *.

[0104] On the other hand, the effect of the improved cross-entropy loss can be seen in [reference needed]. Figure 5As shown. For the mask of the Transformer segmentation network output, this invention uses an improved cross-entropy loss for constraint optimization. By introducing two optimization schemes to improve the existing cross-entropy loss, the direction of network training is dynamically changed, alleviating the problem of excessively low confidence caused by the overall suppression of gradients. Specifically, given the predicted P of the segmentation network output... n and the true label G t The existing Focal loss can be expressed as:

[0105] l focal =-(1-P t ) γ log(P t );

[0106] Among them, the prediction probability γ is the modulation factor, γ∈[0,5]. focal When encountering difficult pixels, When encountering easy pixels, However, in the later stages of network training, there is a problem that the color and texture of some dangerous animals are similar to the surrounding background. Existing Focal loss still inappropriately focuses on difficult and simple pixels, ignoring the overall optimization of the prediction results, making it difficult for the network to converge further. First, this invention introduces a global adjustment coefficient β to dynamically reflect the overall situation of pixel prediction. Assuming that all pixels are predicted correctly, the prediction probability P... t =1, denoted as P r This is divided by the current predicted probability. The predicted probability for each pixel is expanded to the power of γ, which can be expressed as:

[0107]

[0108] Here, H and W represent the length and width of the current video frame I. Secondly, this invention further introduces a local modulation factor α. First, the gradient distribution of the Focal loss is observed by performing a Taylor expansion:

[0109]

[0110] Differentiating the above expression, we get:

[0111]

[0112] Therefore, except for the first term, the gradient values ​​of the other terms decrease exponentially. Thus, the first term plays a dominant role in the backpropagation of network optimization. The number of difficult pixels is often less than the number of simple pixels, and the gradient changes in the later stages of training mainly come from simple pixels, which slows down the convergence speed in the later stages of training. Therefore, this invention introduces a modulation factor α∈(0,2) to reduce the weight value of the first term, which can be expressed as:

[0113]

[0114] Integrating the global adjustment coefficient β, we can obtain the final improved cross-entropy loss:

[0115]

[0116] Step Six: Transmission and Early Warning of Detection Results: The detection results obtained from the network will be transmitted back to the backend using Beidou terminals, serving the backend detection personnel and other terminal users.

[0117] This invention implements its method using an RTX 6000 GPU with 48GB of video memory. Specifically, the Transformer video segmentation network is configured with 6 encoder and decoder layers, though this can be reduced due to limitations in terminal computing power; the feature dimension C = 256, and the number of instance queries N = 900. The entire network is trained over 10 training epochs with a batch size of 2. The initial learning rate is set to 0.0010, decreasing to 0.0005 after 6 training epochs. In addition to using common class loss, bounding box loss, and inter-frame contrast loss, the optimized training of the network employs the improved cross-entropy loss and visual-audio contrast loss of this invention, where α = 1.5, achieving optimal performance on the validation set.

[0118] Table 1 shows the regional similarity between existing methods and the method proposed in this invention. Contour accuracy and its average value As an evaluation metric for the network, it can be seen that the method proposed in this application achieves the best performance compared with other state-of-the-art methods. Compared with the best single-modal detection algorithm, the average value is... The improvement was 17.4%; for multimodal detection methods, the proposed method, compared with the best method, showed an average improvement of 17.4%. It increased by 11.3%.

[0119] Compared to basic detectors, the proposed method significantly reduces false negatives and false positives. This method greatly improves recognition performance by introducing a visual-audio modality feature fusion interaction module, audio-visual contrastive learning, and improved cross-entropy loss. Furthermore, this method achieves deep interaction between the information from the two modalities, resulting in higher information fusion and better detection performance.

[0120] Table 1: Regional similarity between existing methods and the method proposed in this invention Contour accuracy and its average value As an evaluation indicator for the network

[0121]

[0122]

[0123] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that this embodiment is not limited to the described order of actions, because according to this embodiment, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this embodiment.

[0124] Reference Figure 6 The diagram shows a structural block diagram of an embodiment of a sound-guided dangerous animal detection and early warning device, which may specifically include the following modules:

[0125] Dangerous Animal Feature Dataset Construction Module 301: This module collects image or video data related to dangerous animals, and simultaneously gathers their corresponding sound source data to construct a comprehensive animal feature dataset. To ensure effective training, validation, and evaluation of the network, the collected data is grouped into training, testing, and validation sets. Furthermore, based on the audio information of the corresponding animal species, accurate real-world masking is performed on dangerous animal targets in the image or video data.

[0126] Multimodal feature encoding module 302: Inputs the video to be detected and the corresponding audio information into the multimodal encoding network to extract the corresponding visual and audio features.

[0127] The multimodal coding network includes two branches: a video image extraction branch and an audio branch. Deep features are extracted from the original video images and original audio. Specifically, visual information is processed by a pre-trained visual encoder to extract multi-scale visual features, which are then concatenated to form global visual features after unifying the channel dimensions through a designed learnable linear layer. Audio information is processed by a pre-trained audio encoder, which unifies the dimensions with the visual features through a designed learnable linear layer, forming an audio representation in the feature space.

[0128] Multimodal feature fusion interaction module 303: Inputs the extracted visual and audio features into a designed multimodal feature fusion interaction module. Specifically, it utilizes a bidirectional multi-head cross-modal attention mechanism for the audio-visual modality to achieve full interaction between the two modal features, enhancing the correlation and matching of key features between modalities, thereby obtaining audio-aware visual features and visual-aware audio features.

[0129] Dangerous animal detection and segmentation module 304: It feeds the fused interactive visual features with audio awareness and visual features with audio awareness into a Transformer-based video object segmentation network, processes the visual features frame by frame, and then uses a re-identification post-processing module to locate the identified object in order to obtain the identification and detection results of dangerous animals in continuous video frames.

[0130] Network training and optimization module 305: In a training batch, for example object features obtained from the Transformer segmentation network, positive and negative samples are further constructed with audio features extracted by the multimodal coding network in the same batch, constructing audio-visual contrastive learning to promote alignment between different modalities. Furthermore, for the detection results output from the Transformer segmentation network, this invention uses an improved cross-entropy loss to constrain the detection mask.

[0131] Detection result transmission and early warning module 306: For the detection results obtained from the network, the early warning message will be transmitted back to the background with the help of the Beidou terminal, serving the background detection personnel and other terminal users.

[0132] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0133] Specific limitations regarding sound-guided dangerous animal detection and early warning devices can be found in the above-mentioned limitations on sound-guided dangerous animal detection and early warning methods, and will not be repeated here. Each module in the aforementioned sound-guided dangerous animal detection and early warning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0134] The sound-guided dangerous animal detection and early warning device provided above can be used to execute the sound-guided dangerous animal detection and early warning method provided in any of the above embodiments, and has the corresponding functions and beneficial effects.

[0135] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for operation and maintenance. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0136] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0137] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement, for example, Figures 1-6 The steps described:

[0138] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program implementing, when executed by a processor, as shown in the figure. Figures 1-6The steps described above.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0140] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0141] Those skilled in the art will understand that embodiments of this example can be provided as methods, apparatus, or computer program products. Therefore, this example can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this example can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This embodiment is described with reference to flowchart illustrations and / or block diagrams of the method, terminal device (system), and computer program product according to this embodiment. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] Although preferred embodiments of this embodiment have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this embodiment.

[0146] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0147] The foregoing has provided a detailed description of a sound-guided method and device for detecting and warning dangerous animals, a computer device, and a storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting and issuing early warnings of dangerous animals based on sound guidance, characterized in that, include: Step 1: Construction of Dangerous Animal Feature Dataset: Collect image or video data related to dangerous animals, and simultaneously collect their corresponding sound source data to construct a comprehensive animal feature dataset; to ensure effective training, validation and evaluation of the network, the collected data is grouped into training set, test set and validation set; and, based on the audio information of the corresponding animal species, accurate real-world masking is performed on dangerous animal targets in the image or video data. Step 2, Multimodal Feature Encoding: Input the video to be detected and the corresponding audio information into the multimodal coding network to extract the corresponding visual and audio features; The multimodal coding network includes two branches: a video image extraction branch and an audio branch. Deep features are extracted from the original video images and original audio. Specifically, visual information is processed by a pre-trained visual encoder to extract multi-scale visual features, which are then concatenated to form global visual features after unifying the channel dimensions through a designed learnable linear layer. Audio information is processed by a pre-trained audio encoder and then unified with the visual features through a designed learnable linear layer to form an audio representation in the feature space. Step 3: Multimodal Feature Fusion and Interaction: The extracted visual and audio features are input into a designed multimodal feature fusion and interaction module. A bidirectional multi-head cross-modal attention mechanism for audio-visual modalities is used to achieve full interaction between the two modal features, enhance the correlation and matching of key features between modalities, and obtain audio-conscious visual features and visual-conscious audio features. Step 4, Dangerous Animal Detection and Segmentation: The fused and interactive visual features with audio awareness and audio features with visual awareness are fed into a Transformer-based video object segmentation network. The visual features are processed frame by frame, and then a re-identification post-processing module is used to locate the identified object to obtain the identification and detection results of dangerous animals in continuous video frames. Step 5, Network Training and Optimization: In a training batch, for example object features obtained from the Transformer segmentation network, positive and negative samples will be constructed with audio features extracted by the multimodal coding network in the same batch to construct audio-visual contrast learning and promote alignment between different modalities; and, for the detection results output from the Transformer segmentation network, cross-entropy loss will be used to constrain the detection mask. Step Six: Transmission and Early Warning of Detection Results: The detection results obtained from the network will be transmitted back to the backend using Beidou terminals, serving the backend detection personnel and other terminal users.

2. The method according to claim 1, characterized in that, In step two, the visual encoder includes a visual extraction backbone pre-trained on large-scale datasets of the ResNet and ViT series. The audio feature encoders include audio extraction backbones pre-trained on large-scale datasets from the Vgg and ResNet series.

3. The method according to claim 1, characterized in that, In step two, a set of video and audio inputs are given. Video sequence ,in, For video frame rate, , Indicates the first in the video Frame; audio source , This represents the number of samples when the audio sampling frequency is 16000; the visual encoder extracts multi-scale visual features from each frame of the input image, and unifies the dimensionality through a linear layer. Then, visual features were obtained. The audio encoder extracts feature vectors from the input audio source, then passes them through two linear layers and one Dropout layer to obtain the audio features. ,in, The number of feature channels, and Let be the sequence length of each of the two modal features.

4. The method according to claim 1, characterized in that, In step three, the extracted visual features and audio features The input is to a visual-audio modal feature fusion interaction module designed with visual features. and audio features The visual query value is obtained by normalizing the result through a LayerNorm layer and then passing it through a set of learnable linear projection matrices. and audio key value The similarity matrix is ​​calculated using visual features as the query. : ； By transposing the matrix, we obtain another similarity matrix with audio features as query values. : ； The normalized visual features are represented by a set of linear projection matrices. and audio features Mapping yields numerical values and Then, perform a cross product with the similarity matrix obtained after applying the Softmax function: ； ； in, This represents the Softmax function; finally, the visual features are obtained. and audio features and normalized visual features and audio features Perform residual operations to obtain audio-conscious visual features. and audio features with visual awareness : ； 。 5. The method according to claim 4, characterized in that, In step four, visual features with audio awareness are... and audio features with visual awareness The input is fed into a Transformer-based video object segmentation network, where the Transformer encoding network operates on audio-conscious visual features. After further deep encoding, it is then fed into the Transformer decoding network, through... Learnable example queries Decode visual features to capture target objects.

6. The method according to claim 5, characterized in that, In step four, in order to calculate whether the predicted target is a dangerous animal, Learnable example queries Will be associated with visually conscious audio features Calculate the matching score Visually conscious audio features are obtained through average pooling operations. Extract audio feature vectors of length 1 and compare them with instance queries. The features are cross-producted as follows: ； in, This indicates the average pooling operation.

7. The method according to claim 1, characterized in that, In step five, a multilayer perceptron consisting of two linear layers and one ReLU activation function is used to process instance queries in the Transformer segmentation network. Capture the query features of the target and audio features The target query embedding is obtained by projecting it into the multimodal joint space. and audio query embedding Specifically: ； ； in, This indicates the average pooling operation. This represents a multilayer perceptron consisting of two linear layers and one ReLU activation function, used to map embedded features from a low-dimensional representation space to a high-dimensional multimodal joint embedding space.

8. The method according to claim 1, characterized in that, In step five, after constructing positive and negative audio-visual sample pairs and projecting them into the multimodal joint space, a contrastive loss function is used. Supervised network training: ； in, Indicates the size of the training batch. and Index representing the embedded feature. The cosine similarity between two embedded features is represented. Indicates the temperature coefficient of adjustment. Indicates taking the logarithm. Represents the natural constant of Power; In step five, the mask output by the Transformer segmentation network is used for constraint optimization using cross-entropy loss, given the predicted output of the segmentation network. and real labeling Focal loss is expressed as: ； Among them, the prediction probability , It is the modulation factor. ; When encountering difficult pixels, When encountering easy pixels, If all pixels are predicted correctly, then the prediction probability is... , recorded as This is divided by the current predicted probability, where the predicted probability for each pixel is calculated... Power-law expansion, global adjustment coefficient Represented as: ； in, and Represented as the current video frame The length and width are used to introduce a local modulation factor. We then performed a Taylor expansion of the Focal loss to observe its gradient distribution. ； Differentiating the above expression, we get: ； Introduce a modulation factor To reduce the weight of the first term, it is expressed as: ； Integrating global adjustment coefficients The final cross-entropy loss is obtained as follows: .

9. A sound-guided dangerous animal detection and early warning device, characterized in that, include: Dangerous animal feature dataset construction module: Collects image or video data related to dangerous animals, and simultaneously collects their corresponding sound source data to construct a comprehensive animal feature dataset; To ensure effective training, validation and evaluation of the network, the collected data is grouped into training set, test set and validation set; and, based on the audio information of the corresponding animal species, accurate real-world masking is performed on dangerous animal targets in the image or video data. Multimodal feature encoding module: Input the video to be detected and the corresponding audio information into the multimodal encoding network to extract the corresponding visual and audio features; The multimodal coding network comprises two branches: a video image extraction branch and an audio branch. Deep features are extracted from the original video images and original audio. Visual information is processed by a pre-trained visual encoder to extract multi-scale visual features, which are then concatenated to form global visual features after unifying channel dimensions through a designed learnable linear layer. Audio information is processed by a pre-trained audio encoder, and its dimensions are unified with the visual features through a designed learnable linear layer, forming an audio representation in the feature space. A multimodal feature fusion interaction module is included, inputting the extracted visual and audio features. This module utilizes a bidirectional multi-head cross-modal attention mechanism for audio-visual modalities to achieve full interaction between the two modal features, enhancing intermodal correlation and matching of key features to obtain audio-aware visual features and visual-aware audio features. Dangerous animal detection and segmentation module: The fused and interactive visual features with audio awareness and audio features with visual awareness are fed into a Transformer-based video object segmentation network. The visual features are processed frame by frame, and then a re-identification post-processing module is used to locate the identified object in order to obtain the identification and detection results of dangerous animals in continuous video frames. Network Training and Optimization Module: In a training batch, for example object features obtained from the Transformer segmentation network, positive and negative samples are constructed with audio features extracted by the multimodal coding network in the same batch to construct audio-visual contrast learning and promote alignment between different modalities; and, for the detection results output from the Transformer segmentation network, cross-entropy loss is used to constrain the detection mask. Detection result transmission and early warning module: For the detection results obtained from the network, the early warning message will be sent back to the backend with the help of the Beidou terminal, serving the backend detection personnel and other terminal users.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the sound-guided dangerous animal detection and early warning method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the sound-guided dangerous animal detection and early warning method according to any one of claims 1 to 8.

Images