Off-vehicle biological detection method, device, electronic device and readable storage medium

Abstract

The present application relates to the field of image processing technology, and provides a method, device, electronic device and readable storage medium for off-vehicle biological detection. The method includes: obtaining multiple frames of images to be detected and corresponding inter-frame information; inputting the above data into the first input module of the trained biological detection model to obtain first feature data; inputting the intermediate frame to be detected with front and rear adjacent frames in the multiple frames of images to be detected and the inter-frame information into the second input module of the biological detection model to obtain second feature data; through the dual-channel network module of the biological detection model, dynamic feature data is obtained based on the first feature data, and static feature data is obtained based on the second feature data, thereby obtaining a target feature data group; through the fusion prediction module of the biological detection model, the target detection result of the target to be detected is obtained based on the target feature data group. The off-vehicle biological detection method provided in the present application can improve the accuracy of off-vehicle biological detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an off-vehicle biological detection method, device, electronic device and readable storage medium. Background Art

[0002] In the existing technology, the sentry mode can detect whether there are pedestrians or oncoming vehicles behind the door when the person in the car opens the door, so as to prevent the pedestrians or vehicles passing by without reacting in time due to the door being opened rashly, thus causing a collision. The vehicle's vibration sensor and camera can also be used to jointly identify whether there is any deliberate behavior of smashing the vehicle outside the car.

[0003] However, in scenarios where the external organisms are animals, the vehicle cannot identify the movements of the animals very well, resulting in low accuracy in detecting the external organisms. Summary of the invention

[0004] In view of this, the present application provides a method, device, electronic device and readable storage medium for detecting off-vehicle organisms to solve the problem of low accuracy of off-vehicle organism detection in the prior art.

[0005] In a first aspect of the present application, a method for detecting biological substances outside a vehicle is provided, comprising:

[0006] Acquire multiple frames of images to be detected and inter-frame information of the multiple frames of images to be detected collected by a camera outside the vehicle;

[0007] Inputting a plurality of frames of images to be detected and inter-frame information of the plurality of frames of images to be detected into a first input module of the trained biological detection model to obtain first feature data;

[0008] Inputting the middle frame to be detected image having the front and rear adjacent frames and the inter-frame information among the multiple frames to be detected images into the second input module of the biological detection model to obtain the second feature data;

[0009] By using a dual-channel network module of a biological detection model, dynamic feature data is obtained based on the first feature data, static feature data is obtained based on the second feature data, and a target feature data group is obtained based on the static feature data and the dynamic feature data, wherein the target feature data group includes a plurality of target feature data;

[0010] Through the fusion prediction module of the biological detection model, based on the target feature data group, the target detection results of the target to be detected in multiple frames of images to be detected are output.

[0011] In a second aspect of the present application, there is provided an off-vehicle biological detection device, comprising:

[0012] An acquisition module is configured to acquire multiple frames of images to be detected and inter-frame information of the multiple frames of images to be detected acquired by a camera outside the vehicle;

[0013] A first processing module is configured to input a plurality of frames of images to be detected and inter-frame information of the plurality of frames of images to be detected into a first input module of a trained biological detection model to obtain first feature data;

[0014] A second processing module is configured to input the middle frame to be detected image having front and rear adjacent frames and inter-frame information among the multiple frames to be detected images into a second input module of the biological detection model to obtain second feature data;

[0015] A third processing module is configured to obtain dynamic feature data based on the first feature data, obtain static feature data based on the second feature data, and obtain a target feature data group based on the static feature data and the dynamic feature data through a dual-channel network module of the biological detection model, wherein the target feature data group includes a plurality of target feature data;

[0016] The output module is configured to output the target detection results of the target to be detected in multiple frames of images to be detected based on the target feature data group through the fusion prediction module of the biological detection model.

[0017] According to a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0018] According to a fourth aspect of the present application, a readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] Multiple frames of images to be detected and inter-frame information of multiple frames of images to be detected are input into the first input module of the biological detection model to obtain first feature data, and the intermediate frame of the image to be detected and the inter-frame information are input into the second input module of the biological detection model to obtain second feature data, so that the first feature data combines the information of the intermediate frame and the previous and next frames and the correlation information between the previous and next frames, and the second feature data combines the intermediate frame information and the correlation information between the previous and next frames. Then, the dynamic feature data and the static feature data are obtained based on the first feature data and the second feature data respectively through the dual-channel network module in the biological detection model. Since the first feature data combines the information of the intermediate frame and the previous and next frames and the correlation information between the previous and next frames, and the second feature data combines the intermediate frame information and the correlation information between the previous and next frames, the inter-frame information can enhance the continuous tracking capability of the target to be detected, so that the dynamic feature data can focus on reflecting the movement characteristics of the organisms outside the vehicle, and the static feature data can focus on reflecting the characteristics of the organisms outside the vehicle, so that the dynamic feature data and the static feature data can be combined with each other to obtain more comprehensive target feature data, thereby improving the stability and accuracy of the detection of organisms outside the vehicle in complex and changeable application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a flow chart of an off-vehicle biological detection method provided in an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of a data processing flow of a first input module and a second input module provided in an embodiment of the present application;

[0024] Figure 3 It is a basic structural diagram of the first input module and the second input module provided in the embodiment of the present application;

[0025] Figure 4 It is a basic structure diagram of the dynamic channel network module and the static channel network module provided in the embodiment of the present application;

[0026] Figure 5 It is a schematic diagram of the data processing flow of the dual-channel network module provided in an embodiment of the present application;

[0027] Figure 6 It is a process processing diagram of a feature output module provided in an embodiment of the present application;

[0028] Figure 7 It is a schematic diagram of the data processing flow of the fusion prediction module provided in an embodiment of the present application;

[0029] Figure 8 is a schematic diagram of an off-vehicle biological detection device provided in an embodiment of the present application;

[0030] Fig. 9 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0032] The off-vehicle biological detection method and device according to the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0033] In the existing technology, the traditional sentry mode can detect whether there are pedestrians or oncoming vehicles behind the door when the person in the car opens the door, so as to prevent the pedestrians or vehicles passing by without reacting in time due to the door being opened rashly, thus causing a collision. The vehicle's vibration sensor and camera can also be used to jointly identify whether there are abnormal vibrations outside the car, thereby judging whether there is any deliberate smashing of the vehicle and other behaviors.

[0034] In addition to the traditional Sentry Mode functions, Sentry Mode can also capture images through regular camera capture when the vehicle is parked, thereby identifying destructive behavior of pedestrians, other vehicles, or other creatures against the vehicle through image analysis.

[0035] Through the above-mentioned sentinel mode of visual recognition, creatures can be identified and more detailed clues of creatures' behavior in damaging vehicles can be obtained. For example, the sentinel mode can identify a dog outside the vehicle and record the dog's urination on the vehicle, or it can identify mice and send instructions to prevent mice from getting into the front cabin and chewing the wires when mice are identified. However, in the sentinel mode of visual recognition, there are certain accuracy requirements for the visual recognition algorithm of creatures, and in actual application scenarios, the light conditions are usually general or even dim, and the creatures have certain mobility, so a more accurate recognition method is needed.

[0036] Taking the detection of mice among biological species as an example, there are two main types of visual recognition (i.e., image recognition) mouse detection algorithms at present. One type is based on deep learning algorithms, which is achieved through target detection, segmentation, etc. The other type is through traditional image processing algorithms, which detects the movement of organisms through scene models (Gaussian Mixture Model (GMM) etc.) and performs certain template matching. In traditional image processing algorithms, for each type of biological species, features need to be manually designed so that objects in the image can be detected based on the features. Among them, most of the manually designed features are shallow features that are sensitive to image brightness, noise, etc., and are composed of shallow features alone or a fusion of multiple features, which leads to the poor generalization ability of traditional image processing algorithms for application scenarios.

[0037] In the deep learning algorithm, the first frame image can be divided into several image blocks, and the sub-images can be sent to the network model for detection. After detecting the existence of a certain type of organism, the target position in the four frames after the first frame is judged whether it has moved, and the movement attributes are combined to determine whether it is a creature of that species, that is, the movement features are used as a post-judgment method.

[0038] A single image can also be input into the detection network model, and three different scales of center point heat maps can be obtained through dilated convolution with three different dilation rates. Then, combined with a Gaussian probability model with adaptive standard deviation of the Gaussian distribution, the center point heat maps of the above four different receptive fields are additively fused to construct a multi-scale receptive field Gaussian weighted structure to complete the detection of organisms and biological species, that is, to determine the detection of organisms and biological species based on the characteristics of a single image.

[0039] The feature map can also be refined and candidate positions can be roughly screened out, and refined identification and position regression can be performed based on the rough screening results, that is, feature enhancement and hierarchical processing can be performed based on a single image, and detection results can be obtained by feature refinement and step-by-step refinement.

[0040] The above-mentioned existing technologies for detecting organisms and biological species mainly use static features in a single image. In the present application, an improved dual-channel model is used as the backbone network, and motion features are added to the detection network as input information. This can increase the feature perception in biological detection scenarios and increase the recognition accuracy in low-light conditions such as at night. It makes more full use of the dynamic and static features of the detection object, and has a significant improvement effect on biological detection.

[0041] Figure 1 This is a flow chart of a method for detecting biological organisms outside a vehicle provided by this application. This application can be executed by the vehicle end, or the vehicle end can send the image to be detected to the server and execute it by the server. Figure 1As shown, the off-vehicle biological detection method includes the following steps:

[0042] S101, obtaining a plurality of frames of images to be detected and inter-frame information of the plurality of frames of images to be detected collected by a camera outside the vehicle.

[0043] Specifically, a video captured by a camera outside the vehicle can be obtained, and multiple frames of images to be detected can be extracted from the video sequence. The images to be detected are used for biological detection tasks, and each image to be detected includes one or more targets to be detected.

[0044] Among them, the vehicle's external camera can be a high-definition camera, a surround-view camera, or a video acquisition card and other devices, which can ensure that the image to be detected meets the detection requirements.

[0045] In addition, when acquiring multiple frames of images to be detected, each frame in the video sequence can be processed to acquire multiple frames of initial images to be detected, and the multiple frames of initial images to be detected are optimized to obtain multiple frames of images to be detected, specifically including:

[0046] In the video sequence, 2n+1 frames of images can be extracted at intervals of k frames as the initial multi-frame images to be detected. For example, as an example, setting the value of k to 5 and the value of n to 1, it can represent taking the images of the 1st frame, the 6th frame, and the 12th frame as a group of initial multi-frame images to be detected. The image data of the initial images to be detected can be recorded as [ , , ],in, Indicates the image data of the 5th frame, Indicates the image data of the first frame, represents the image data of the 12th frame, wherein the image data may be in the form of a matrix, and it should be noted that the middle frame, the previous frame of the middle frame, and the next frame of the middle frame are three adjacent frames corresponding to the initial multiple frames of the image to be detected, the middle frame is a frame with previous and next frames among the three adjacent frames, and the previous frame and the next frame are both based on the middle frame as a reference. Continuing with the above example, represents the image data of the intermediate frame, Represents the image data of the frame before the intermediate frame, Indicates the image data of the frame following the intermediate frame.

[0047] Furthermore, each frame of the initial image to be detected can be preprocessed, and the preprocessing may include data enhancement, fixed-size scaling, etc. In particular, during data enhancement, multiple frames of initial images to be detected can be set as a data group, and the data frames in each group can be enhanced by brightness adjustment, gamma value adjustment, random noise, blur degradation, translation transformation, etc. with random intensity within a set range. It should be noted that all frames in the initial images to be detected in the same data group use the same data enhancement parameters.

[0048] Continuing with the above example, after data enhancement is performed on the data group consisting of the initial multiple frames of images to be detected, the initial multiple frames of images to be detected in each data group can be scaled to a fixed size, that is, the short side of each frame of the initial multiple frames of images to be detected is filled with a fixed value, for example, it can be fixedly scaled to 608×608 pixels.

[0049] Furthermore, since the initial images to be detected are mostly low-light scenes, the initial images to be detected can be converted into grayscale images, and the pixel values ​​in each frame of the initial images to be detected are processed according to the following formula so that the pixel values ​​are distributed in the interval [-1, 1]: , where i represents the pixel value. Since the pixel value is generally 0-255, half of 255 is used as the processing parameter.

[0050] At the same time, inter-frame information of multiple frames of images to be detected is obtained. The inter-frame information may include motion vectors between adjacent frames, changing image features or other dynamically changing data, and can capture changes in the target to be detected at different time points.

[0051] By acquiring multiple frames of images to be detected and their inter-frame information, the dynamic changes of the target to be detected over time can be effectively obtained, so that fast-moving targets can be detected or target behavior patterns can be judged, and the stability of target detection can be improved through the inter-frame information of the previous and next adjacent frames.

[0052] S102, inputting a plurality of frames of images to be detected and inter-frame information of the plurality of frames of images to be detected into a first input module of a trained biological detection model to obtain first feature data.

[0053] Specifically, the biological detection model includes a first input module and a second input module, and the input data corresponding to the two input modules are different. Among them, multiple frames of images to be detected and corresponding inter-frame information can be input into the first input module to obtain first feature data, so that the first feature data combines the image information of multiple frames of images to be detected and the correlation information between the previous and next frames. The output of the first input module can be used as the input of the dynamic channel network module in the dual-channel network module of the biological detection model.

[0054] The dynamic channel network module is used to process all dynamic features of the target to be detected in the image to be detected, which includes the motion features of the organism. The dynamic features obtained by the dynamic channel network module are sensitive to resolution. In order to fully capture the motion details, a relatively large resolution can be retained.

[0055] S103, inputting the middle frame image to be detected having the front and rear adjacent frames among the multiple frames of images to be detected and the inter-frame information into the second input module of the biological detection model to obtain second feature data.

[0056] Specifically, the middle image to be detected having the front and rear adjacent frames in the multiple frames of images to be detected and the inter-frame information are input into the second input module of the model. For example, the current group of data frames includes a total of 3 frames of images to be detected, then the image to be detected in the middle frame and the inter-frame information of the 3 frames of images to be detected are input into the second input module to obtain the second feature data, so that the second feature data combines the image information of the middle frame image to be detected and the correlation information between the front and rear frames.

[0057] The output of the second input module can be used as the input of the static channel network module in the dual-channel network module. When extracting static features, high-level feature abstraction is required. In order to ensure the balance between the depth and efficiency of static feature extraction, a convolutional layer with a step size of 2 can be used in the second input module to obtain the static features of the image to be detected, which can reduce the size of the feature layer and increase the network depth, which helps the biological detection model understand the data.

[0058] Among them, the inter-frame information is used as input data to play the role of attention mechanism.

[0059] S104, obtaining dynamic feature data based on the first feature data, obtaining static feature data based on the second feature data, and obtaining a target feature data group based on the static feature data and the dynamic feature data through the dual-channel network module of the biological detection model.

[0060] Wherein the target feature data group includes a plurality of target feature data;

[0061] Specifically, the dual-channel network module of the biological detection model may include a static channel network module and a dynamic channel network module. When obtaining dynamic feature data based on the first feature data and obtaining static feature data based on the second feature data, the dynamic feature data can be obtained based on the first feature data through the dynamic channel network module, and the static feature data can be obtained based on the second feature data through the static channel network module. In this way, since the first feature data includes the motion details of the target to be detected, that is, includes all the action features of the target to be detected, the dynamic feature data obtained based on this can include all the action features of the target to be detected; in addition, since the second feature data includes the stable features of the target extracted from a single image frame to be detected and the correlation information between the previous and next frames, the static feature data obtained based on this can obtain deep feature information.

[0062] In addition, the dual-channel network module of the biological detection model combines and processes the above dynamic feature data and static feature data to obtain a target feature data group, which includes multiple target feature data, each feature data corresponding to a specific attribute or feature of the target to be detected. For example, a target feature data group including three target feature data can be obtained through the dual-channel biological detection model.

[0063] Among them, multiple target feature data are target feature data from different dimensions. For example, static features and dynamic features that have undergone different convolution processing can be combined to obtain target feature data, which can enable features from different sources to complement each other, improve the accuracy of biological detection, and improve the versatility of biological detection models.

[0064] S105, outputting target detection results of the target to be detected in multiple frames of images to be detected based on the target feature data group through the fusion prediction module of the biological detection model.

[0065] Specifically, the biological detection model also includes a fusion prediction module, into which the target feature data set is input, and the target feature data is analyzed by the fusion prediction module to perform target detection and recognition. It should be noted that the target to be detected can be an animal, such as a cat, a dog, a mouse, etc.

[0066] In addition, the target detection results may include relevant information such as the target's location, category, confidence, etc. The target detection results provide a description and positioning of the target to be detected, and realize the biological detection model through deep learning. The dynamic and static features of multiple frames of images to be detected are obtained, and the dynamic and static features are detected and identified from different dimensions to obtain more accurate and versatile target detection results.

[0067] According to the technical solution provided in the embodiment of the present application, a first input module can be used to process multiple frames of images to be detected and their inter-frame information to obtain first feature data, thereby obtaining dynamic features, and a second input module can be used to process intermediate frames of images to be detected and their inter-frame information to obtain second feature data, thereby obtaining static features, thereby realizing the capture of motion trajectory, speed and other features of the target to be detected through the first feature data, and the capture of details of a single frame image, such as texture and shape, through the second feature data; static feature data and dynamic feature data are fused through a dual-channel network module, so that the advantages and disadvantages of features can be complemented to each other, and more comprehensive target feature data can be obtained, thereby improving the stability of detection in complex and changeable application scenarios, and at the same time, the continuous tracking capability of the target to be detected can be enhanced through inter-frame information, thereby enhancing the stability and accuracy of off-vehicle biological detection.

[0068] It should be noted that, as an example, Figure 2 Schematic diagram of the data processing flow of the first input module and the second input module provided in the embodiment of the present application. Figure 2As shown, the first input module may include a first input convolution block, and the second input module may include a second input convolution block and a third input convolution block. The image data of multiple frames of images to be detected and the inter-frame information of multiple frames of images to be detected are input into the first input module, that is, input into the first input convolution block. Continuing with the above example, assuming that the multiple frames of images to be detected include 3 frames of images to be detected, the data input into the first input module includes: , , and ,in, is the image data corresponding to the previous frame of the image to be detected in the middle frame, is the image data of the middle frame to be detected, is the image data of the next frame after the intermediate frame to be detected. is the inter-frame information. It should be noted that in the diagram, <4> It means there are four input data in total, and the step size of the first input convolution block can be 1, that is,<s=1> Indicates that the step size of the convolution process is 1. <m1>The first feature data of the output indicates that the feature value of the m1 layer is obtained. In this way, the input feature is processed by the first input convolution block, which can retain a relatively large resolution, thereby capturing more sufficient motion details, and the first feature data contains all the motion features of the target to be detected.

[0069] In addition, specifically, the input data of the second input module includes and , that is, the second input convolution block in the diagram <2> Indicates that there are two input data, namely and ,<s=1> Indicates that the stride of the second input convolution block is 1, <n1>It means that the output data of the second input convolution block has n1-layer feature values; after the second input convolution block outputs n1-layer feature values, the n1-layer feature values ​​are used as the input data of the third input convolution block, and the second feature data is obtained after being processed by the third input convolution block, where<s=2> Indicates that the stride of the third input convolution block is 2, <n2>The second feature data output has n2 layers of feature values, that is, <n1>conduct<s=2> The convolution process is performed to obtain the n2-layer feature values. In this way, since static features generally require higher-level feature abstraction, the second input module includes two convolution blocks in series, which ensures the balance between depth and efficiency, reduces the size of the feature layer, and increases the network depth, which helps the model understand the data.

[0070] In addition, if Figure 3 As shown, the first input module and the second input module (i.e., the first input convolution block, the second input convolution block, and the third input convolution block) can use the basic CBL structure, which includes a convolution layer (Conv), a normalization layer (BN), and an activation function (Leaky_relu) connected in sequence. The convolution layers in different input modules can use different step sizes, where the step size of the convolution layer in the first input convolution block and the second input convolution block can be 1, and the step size of the convolution layer in the third input convolution block can be 2. After the first input module, the format of the output data is , which is a feature map of 608×608 pixels, and the image has 16 different channels; after the second input module, the output data format is , which is a feature map of 304×304 pixels in size, and the image has 32 different channels.

[0071] In some embodiments, the dual-channel network module includes a static channel network module, a dynamic channel network module and a feature output module, the dynamic channel network module includes a first convolution module, a second convolution module and a third convolution module, and the static channel network module includes a fourth convolution module, a fifth convolution module and a sixth convolution module;

[0072] Through the dual-channel network module of the biological detection model, dynamic feature data is obtained based on the first feature data, static feature data is obtained based on the second feature data, and a target feature data group is obtained based on the static feature data and the dynamic feature data, including:

[0073] Processing the first feature data through a first convolution module to obtain dynamic feature data;

[0074] Processing the second feature data through a fourth convolution module to obtain static feature data;

[0075] Through the second convolution module, convolution processing is performed on the first merged data after the up-sampled static feature data is merged with the dynamic feature data to obtain the first fused feature data;

[0076] Through the fifth convolution module, convolution processing is performed on the second merged data obtained by merging the down-sampled dynamic feature data with the static feature data to obtain second fused feature data;

[0077] Through the three convolution modules, convolution processing is performed on the third combined data obtained by combining the upsampled data of the second fused feature data with the first fused feature data to obtain the third fused feature data;

[0078] Through the sixth convolution module, convolution processing is performed on the fourth combined data obtained by combining the data obtained by downsampling the first fused feature data and the second fused feature data to obtain fourth fused feature data;

[0079] Through the feature output module, a target feature data group is obtained based on the static feature data, the first fused feature data, the second fused feature data, the third fused feature data and the fourth fused feature data.

[0080] Specifically, the dual-channel network module can enhance the performance of the biological detection model by processing static feature information and dynamic feature information in parallel, wherein the dual-channel network module includes a static channel network module, a dynamic channel network module, and a feature output module for fusing static features and dynamic features to obtain target feature data. The static channel network module is used to extract stable features of the target (such as shape, texture, etc.) from a single image frame to be detected, and the dynamic channel network module is used to analyze the changes between multiple frames of images to be detected to capture the motion characteristics of the target (such as speed, trajectory, etc.).

[0081] The dynamic channel network module includes a first convolution module, a second convolution module and a third convolution module connected in sequence, and can gradually extract and deepen dynamic features based on the first feature data. The static channel network module includes a fourth convolution module, a fifth convolution module and a sixth convolution module connected in sequence, and can gradually extract static features based on the second feature data.

[0082] It should be noted that if Figure 4 As shown in the figure, any convolutional module included in the dynamic channel network module and the static channel network module can be constructed based on the channel-wise spatial pyramid network (CSPNet) structure, wherein the CSPNet structure is a deep convolutional neural network for image classification. In the CSPNet structure, the first convolutional layer (Conv), the first normalization layer (BN), the first activation layer (Leaky ReLU) and the second convolutional layer are connected in sequence, and the third convolutional layer and the second convolutional layer are both connected to the connection layer (Concat), and the connection layer is connected to the second normalization layer and the second activation layer in sequence.

[0083] The normalization layer is used to solve the problem of internal covariate shift. By normalizing each mini-batch during model training, it helps the model converge faster and improves training efficiency, alleviates the problem of gradient vanishing, and enables deeper network structures to be effectively trained. Leaky ReLU is an activation function for neural networks that aims to improve the performance of the ReLU function in the negative range to alleviate the problem of neuron death and gradient vanishing, while increasing the expressiveness of the model. Concat is used to connect other modules.

[0084] In addition, any convolutional modules in the dynamic channel network module and the static channel network module may include different numbers or the same number of CSPNet structures.

[0085] In addition, specifically, the first convolution module serves as the starting point of the dynamic channel network module. The first convolution module receives the first feature data obtained from the first input module, extracts the dynamic features through convolution operations and pooling operations, and generates dynamic feature data, which can reveal the change pattern of the target between different frames.

[0086] The fourth convolution module serves as the starting point of the static channel network module. The fourth convolution module receives the second feature data obtained from the second input module, extracts static features through convolution operation, and generates static feature data.

[0087] In addition, in order to further fuse static and dynamic information, the static feature data can be upsampled and merged with the dynamic feature data through an addition operation or a connection layer. The merging method can be matrix addition, and the second convolution module performs convolution processing on the merged feature data to generate first fused feature data. The first fused feature data enhances the richness and stability of the feature representation.

[0088] In addition, in order to interact with dynamic features, dynamic feature data is obtained through the first convolution module and down-sampled to match the static features. The down-sampled data of the dynamic feature data can be merged with the static feature data output by the fourth convolution module through an addition operation or a connection layer, and the fifth convolution module convolves the merged feature data to generate second fused feature data, thereby improving the biological detection model's understanding of the dynamic behavior of the target in a static background.

[0089] In addition, at a deeper level, after obtaining the second fused feature data through the fifth convolution module and upsampling the second fused feature data, the upsampled data can be merged with the first fused feature data output by the second convolution module through an addition operation or a connection layer, and the merged feature data is convolved by the third convolution module to generate the third fused feature data, which can refine and strengthen the combination effect of dynamic features and static features.

[0090] In addition, after obtaining the first fused feature data through the second convolution module and down-sampling the first fused feature data, the down-sampled data can be merged with the second fused feature data output by the fifth convolution module through an addition operation or a connection layer, and the merged feature data is convolved through the sixth convolution module to generate fourth fused feature data, which can enable the comprehensive fusion of static features and dynamic features in the deep network.

[0091] In addition, the feature output module is used to summarize and further process the feature data obtained by the static channel network module and the dynamic channel network module, namely, the static feature data, the first fused feature data, the second fused feature data, the third fused feature data, and the fourth fused feature data. By integrating these multi-level and multi-dimensional feature information, the feature output module can generate a target feature data group, which includes a detailed static description of the target and also integrates the dynamic behavior information of the target.

[0092] As an example, Figure 5 Schematic diagram of a data processing flow of a dual-channel network module provided in an embodiment of the present application. Figure 5 As shown, both the static channel network module and the dynamic channel network module use a three-layer feature extraction structure (i.e., a three-layer convolution module) to obtain feature information of different dimensions, wherein the dynamic channel network module includes a first convolution module, a second convolution module, and a third convolution module, and the static channel network module includes a fourth convolution module, a fifth convolution module, and a sixth convolution module.

[0093] Specifically, for the dynamic channel network module:

[0094] Continuing with the above example, the first convolution module receives the first feature data output by the first input module, and the first feature data is represented as the m1 layer feature value. The first convolution module performs convolution, normalization, activation and other processing on the first feature data to obtain dynamic feature data, which can be represented as the m2 layer feature value. The static feature data output by the fourth convolution module The layer feature values ​​are upsampled to obtain Layer feature values, and dynamic feature data of the previous level Layer feature value merging, static and dynamic feature fusion The layer features are used as the input of the second convolution module to obtain the first fused feature data Layer feature value; the second fused feature data output by the fifth convolution module After upsampling the layer feature values, we get The layer feature value is combined with the first fusion feature data m3 layer feature value of the previous stage to form a static and dynamic feature fusion The layer features are used as the input of the third convolution module to obtain the third fused feature data m4 layer feature values.

[0095] For static network modules:

[0096] The fourth convolution module receives the second feature data output by the second input module, the second feature data is represented as the n2-layer feature value, and the second feature data n2-layer feature value is used as the input of the fourth convolution module. The second feature data is convolved, normalized, activated, and processed by the fourth convolution module to obtain the static feature data n3-layer feature value; the static feature data n3-layer feature value and the dynamic feature data m2-layer feature value output by the fourth convolution module are downsampled by a connection layer or an addition operation (corresponding to the trapezoidal box connected before the fifth convolution module in the figure) Layer feature value merging, static and dynamic feature fusion The layer features are used as the input of the fifth convolution module, and the second fused feature data n4 layer feature values ​​are output; the second fused feature data n4 layer feature values ​​output by the fifth convolution module are downsampled with the first fused feature data m3 layer feature values ​​to obtain Layer feature value merging, static and dynamic feature fusion The layer feature values ​​are used as the input of the sixth convolution module to obtain the feature values ​​of the fourth fused feature data n5 layer.

[0097] It should be noted that in the diagram, the trapezoidal boxes before each convolution module represent addition operations or connection layers to merge the features corresponding to the arrows. In addition, in the convolution module in the diagram, <m1>represents the eigenvalue of layer m1,<s=2> Indicates that the input data is convolved with a step size of 2. <m2>represents the m2 layer eigenvalue, <n2>represents the n2-layer eigenvalue, <n3>represents the eigenvalue of layer n3,<m2+n3'> Represents the m2 layer eigenvalue and The layer feature values ​​are merged.<n3+m2'> Indicates that the n3 layer eigenvalue is merged with the m2' layer eigenvalue. <m3>represents the eigenvalue of the m3 layer, <n4>represents the eigenvalue of layer n4,<m3+n4'> Represents the eigenvalue of the m3 layer and The layer feature values ​​are merged.<n4+m3'> Indicates that the n4 layer eigenvalue is merged with the m3' layer eigenvalue. <m4>represents the eigenvalue of the m4 layer, <n5>Represents the eigenvalue of layer n5.

[0098] According to the technical solution provided in the embodiment of the present application, static features and dynamic features can be extracted through a dual-channel network module, and only part of the dynamic channel feature layer is added to the static channel feature layer through downsampling, and similarly, only part of the static channel feature layer is added to the dynamic channel feature layer through upsampling, so that the static features and dynamic features interact to a certain extent, and the problem of channel equivalence will not occur; in addition, the static channel network module uses a deeper network structure to obtain deep feature information, while the dynamic channel network module uses a shallow network structure to take into account both resolution and computational complexity, which can improve the versatility of the biological detection model and the accuracy of detection.

[0099] In some embodiments, the feature output module includes a seventh convolution module, an eighth convolution module, and a ninth convolution module.

[0100] Through the feature output module, based on the static feature data, the first fused feature data, the second fused feature data, the third fused feature data and the fourth fused feature data, a target feature data group is obtained, including:

[0101] Through the seventh convolution module, convolution processing is performed on the fifth combined data obtained by combining the first fused feature data and the static feature data to obtain the first target feature data;

[0102] Through the eighth convolution module, convolution processing is performed on the sixth combined data obtained by combining the third fused feature data and the second fused feature data to obtain second target feature data;

[0103] Through a ninth convolution module, convolution processing is performed on the seventh combined data obtained by combining the data obtained by downsampling the third fused feature data and the fourth fused feature data to obtain third target feature data;

[0104] The target feature data group includes first target feature data, second target feature data and third target feature data.

[0105] Specifically, the feature output module includes the seventh convolution module, the eighth convolution module and the ninth convolution module, each of which is used for specific data fusion and feature extraction tasks, so that the most representative target feature data group can be extracted from complex and changeable input data.

[0106] The seventh convolution module is used to deeply combine the first fused feature data determined by the second convolution module with the static feature data obtained by the fourth convolution module, that is, the seventh convolution module inputs the data after the first fused feature data and the static feature data are combined into the corresponding convolution kernel, activation function and possible pooling layer, and generates the first target feature data after multi-level nonlinear transformation, which can enhance the richness of the features and improve the model's ability to capture static information. Among them, the static feature information can be the background information in the image to be detected, the inherent attributes of the object, etc.

[0107] The eighth convolution module can broaden the field of view of feature fusion. The eighth convolution module performs convolution processing on the data obtained by merging the third fused feature data with the second fused feature data, wherein the third fused feature data and the second fused feature data respectively represent feature fusion results of different depths or dimensions, which may contain dynamic change information in time series, complex relationships of spatial structures, etc. The eighth convolution module performs convolution processing on the data obtained by merging the third fused feature data with the second fused feature data through, for example, multi-scale convolution, depth-separable convolution, etc., to obtain the second target feature data, thereby improving the adaptability and generalization ability of the biological detection model to complex scenes.

[0108] The ninth convolution module is used to receive the data after downsampling the third fused feature data and the data after merging the fourth fused feature data, and perform convolution processing on the merged data to obtain the third target feature data. The third fused feature data and the fourth fused feature data have undergone multiple rounds of feature extraction and fusion, and contain highly abstract and refined information. The ninth convolution module can fuse them by introducing attention mechanisms, residual connections, etc., effectively reducing information loss and strengthening the representation ability of key features.

[0109] As an example, Figure 6 Schematic diagram of the data processing flow of the feature output module provided in the embodiment of the present application. Figure 6 As shown, specifically, after passing through the dynamic channel network module and the static channel network module, the third fused feature data is downsampled and merged with the fourth fused feature data through an addition operation or a connection layer (the trapezoidal box connected before the ninth convolution module in the figure) as the input of the ninth convolution module, and the merged data is convolved by the ninth convolution module to obtain the third target feature data; the third fused feature data and the second fused feature data are merged as the input of the eighth convolution module through an addition operation or a connection layer, and the merged data is convolved by the eighth convolution module to obtain the second target feature data; the first fused feature data and the static feature data are merged as the input of the seventh convolution module through an addition operation or a connection layer, and the merged data is convolved by the seventh convolution module to obtain the first target feature data, and the first target feature data, the second target feature data and the third target feature data are used as the extraction results of the dual-channel network module in the backbone network.

[0110] Among them, in the figure<m3+n3> Indicates that the input data is the m3 layer eigenvalue and the n3 layer eigenvalue,<s=1> Indicates that the input data is convolved with a step size of 1. <mn1>Indicates that the output data is the first target feature data,<m4+n4> Indicates that the input data is the m4-layer eigenvalue and the n4-layer eigenvalue.<m4'+n5> It means that the input data is the eigenvalue of the m4' layer and the eigenvalue of the n5 layer, and the eigenvalue of the m4' layer is the downsampled value of the eigenvalue of the m4 layer. <mn2>Indicates that the output data is the second target feature data, <mn3>Indicates that the output data is the third target feature data.

[0111] According to the technical solution provided in the embodiment of the present application, the feature output module realizes deep integration and optimization of input feature data from multiple angles and multiple levels through the collaborative work of the seventh convolution module, the eighth convolution module and the ninth convolution module, and realizes the use of static features and dynamic features of the same dimension as output features in the dual-channel network module stage, thereby improving the versatility and accuracy of the biological detection model.

[0112] In some embodiments, the target feature data group includes first target feature data, second target feature data and third target feature data; the fusion prediction module includes a first fusion convolution block, a second fusion convolution block, a third fusion convolution block, a fourth fusion convolution block, a fifth fusion convolution block, a sixth fusion convolution block and a prediction module.

[0113] Through the fusion prediction module of the biological detection model, based on the target feature data group, the target detection results of the target to be detected in multiple frames of images to be detected are output, including:

[0114] Through the first fusion convolution block, convolution processing is performed on the data obtained by upsampling the third target feature data and the eighth combined data obtained by combining the second target feature data to obtain first intermediate data;

[0115] Through the second fusion convolution block, convolution processing is performed on the ninth combined data obtained by combining the data obtained by upsampling the first intermediate data with the first target feature data to obtain second intermediate data;

[0116] Performing convolution processing on the second intermediate data through a third fused convolution block to obtain third intermediate data;

[0117] Performing convolution processing on the tenth combined data obtained by combining the third intermediate data with the first intermediate data through a fourth fusion convolution block to obtain fourth intermediate data;

[0118] Performing convolution processing on the fourth intermediate data through a fifth fused convolution block to obtain fifth intermediate data;

[0119] Performing convolution processing on the eleventh combined data obtained by combining the fifth intermediate data and the third target feature data through a sixth fusion convolution block to obtain sixth intermediate data;

[0120] Through the prediction module, based on the second intermediate data, the fourth intermediate data and the sixth intermediate data, the target detection results of the target to be detected in the multiple frames of the image to be detected are obtained.

[0121] Specifically, the target data group includes first target feature data, second target feature data and third target feature data, wherein the first target feature data comes from a shallow network module, and is characterized by high resolution but less semantic information; the second target feature data comes from a middle network module, and is characterized by moderate resolution and semantic information; the third target feature data comes from a deep network module, and is characterized by low resolution but more semantic information; different target feature data in the target feature data group respectively capture information at different levels of the image to be detected.

[0122] Specifically, as an example, Figure 7 As shown, the fusion prediction module includes a first fusion convolution block, a second fusion convolution block, a third fusion convolution block, a fourth fusion convolution block, a fifth fusion convolution block, a sixth fusion convolution block and a prediction module, and the prediction module includes a first prediction convolution block, a second prediction convolution block and a third prediction convolution block. It should be noted that each module included in the fusion prediction module can be constructed based on the CBL structure, and the convolution kernel can be 1×1 with a step size of 1.

[0123] In addition, after upsampling the third target feature data through an addition operation or a connection layer, it is merged with the second target feature data, and the merged data is convolved through a first fusion convolution block to obtain first intermediate data. The third target feature data can be upsampled by transposed convolution or bilinear interpolation, and the first intermediate data can represent richer semantic information and higher resolution.

[0124] In addition, after upsampling the first intermediate data output by the first fusion convolution block, it is fused with the first target feature data, and the fused data is convolved by the second fusion convolution block to obtain the second intermediate data. The first intermediate data can be upsampled by transposed convolution or bilinear interpolation, so as to enhance the fusion of detail information in the shallow network module and deep speech information.

[0125] In addition, the second intermediate data is convolved by the third fusion convolution block to obtain the third intermediate data, which can further refine the features of the second intermediate data and enhance the expression ability of the internal features.

[0126] In addition, through the fourth fusion convolution block, the third intermediate data and the first intermediate data are combined and convolved to obtain the fourth intermediate data, which can retain more levels of information and enhance the information flow between multi-dimensional features. The merging can be a fusion method such as addition or concatenation.

[0127] In addition, the fourth intermediate data is convolved by the fifth fused convolution block, and the fourth intermediate data can be deeply processed by multiple convolution layers to obtain the fifth intermediate data.

[0128] In addition, the fifth intermediate data and the third target feature data are merged, wherein, considering the differences in resolution, etc., it may be necessary to appropriately upsample or downsample the fifth intermediate data or the third target feature data, and the merged fifth intermediate data and the third target feature data are convolved through the sixth fusion convolution block to obtain the sixth intermediate data, so that the deep semantic information and the shallow high-resolution information can be fused again to enhance the accuracy of the biological detection model.

[0129] In addition, the second intermediate data, the fourth intermediate data and the sixth intermediate data are used as inputs of the prediction module, and are converted into feature maps with the same resolution as the input image through appropriate upsampling or feature integration to obtain target detection results of the targets to be detected in multiple frames of images to be detected. Furthermore, a convolution layer or a fully connected layer can be applied to output the target detection results of each pixel or area in the image to be detected, and the target detection results of each target to be detected can be output based on the fused multi-dimensional feature data. Among them, the target detection results include the location of the target (such as the coordinates of the bounding box) and the confidence level, etc. It should be noted that the sixth intermediate data output by the sixth fused convolution block can be determined as the input of the first prediction convolution block, the fourth intermediate data output by the fourth fused convolution block can be determined as the input of the second prediction convolution block, and the second intermediate data output by the second fused convolution block can be determined as the input of the third prediction convolution block.

[0130] It should also be noted that, in one example, the fusion prediction module can be implemented using the network of the corresponding part of the YoloV5 model. Alternatively, in one example, the part of the fusion prediction module other than the prediction module can be implemented by the partition filter network PFN (‌Partition Filter Encoder) and the pyramid attention PAN network structure.

[0131] According to the technical solution provided in the embodiments of the present application, the biological detection model can more accurately detect the detection targets in multiple frames of images to be detected through multi-dimensional fusion and gradual feature extraction, and show stronger robustness and accuracy when dealing with complex backgrounds and targets of different scales. Among them, the fusion prediction module mainly fuses feature data of different dimensions for further fusion and extraction of the target feature data group, and the prediction module is used to output the target detection result.

[0132] In some embodiments, before inputting multiple frames of images to be detected and inter-frame information of multiple frames of images to be detected into a first input module of a trained biological detection model to obtain first feature data, the method further includes:

[0133] Acquire a training sample set, the training sample set including multiple frames of sample images, inter-frame information of the multiple frames of sample images, and position annotation information of the target to be detected in the sample images;

[0134] The biological detection model is trained by using a training sample set to determine a loss value of the biological detection model, wherein the loss value is a weighted sum of classification loss, confidence loss, and position loss values;

[0135] When it is detected that the loss value is less than the preset loss value, it is determined that the biological detection model training is completed.

[0136] Specifically, in the training phase of the biological detection model, a training sample set is obtained, wherein the training sample set includes multiple sample frames, inter-frame information of the multiple sample frames, and position annotation information of the target to be detected in the sample images, wherein the multiple sample frames may be multiple sample frames under various application scenarios, and may cover different lighting conditions, angles, biological species, and possible occlusion situations. For example, the biological species may be a mouse.

[0137] Among them, the position annotation information can be the use of bounding boxes to annotate the target to be detected. The target to be detected can be a mouse, dog, cat, etc. There should be at least one target to be detected in each group of sample images, and the type of the target to be detected needs to be annotated during annotation. The position annotation information can enable the biological detection model to learn to identify and locate the target organism.

[0138] In addition, the biological detection model is trained through the training sample set, and the loss value of the biological detection model is determined. The loss value is the weighted sum of the classification loss value, the confidence loss value and the position loss value. The classification loss value comes from the classification loss function, the confidence loss value comes from the confidence loss function, and the position loss value comes from the position loss function.

[0139] Among them, the biological detection model is a single-stage model, so the same loss function as other single-stage models can be set, that is, the classification loss function, position loss function and confidence loss function are combined into a loss function. The classification loss is used to measure the accuracy of the biological detection model in classifying the target organism, the confidence loss is used to evaluate the confidence of the biological detection model in the detection results, and the position loss is used to measure the difference between the biological position predicted by the biological detection model and the actual marked position. The performance of the biological detection model in different aspects can be balanced by adjusting the weights of each loss function.

[0140] In addition, the loss value can be determined by the following formula:

[0141] ;

[0142] in, represents the loss value, represents the confidence loss value, The weight value representing the confidence loss value; represents the classification loss value, The weight value representing the classification loss value; represents the position loss value, The weight value representing the position loss value.

[0143] In one embodiment, The value of is 0.5. The value of is 0.25. The value of is 0.25.

[0144] In addition, during the training process, multiple sample images and their inter-frame information in the training sample set are input into the first input module of the biological detection model, and the intermediate sample images and inter-frame information in the training sample set are input into the second input module of the biological detection model. The biological detection model outputs the prediction result according to the current parameters. Then, the loss value between the prediction result and the true annotation is calculated, and the back propagation algorithm can be used to update the parameters of the biological detection model to minimize the loss value. The above steps are repeated multiple times until the performance of the biological detection model (i.e., the loss value) is less than the preset loss value.

[0145] In addition, when the loss value of the biological detection model is less than the preset loss value, it can be considered that the biological detection model has fully learned the features in the training sample set and can be generalized to a certain extent to unseen images to be detected. Then, the training process of the biological detection model can be stopped and the trained biological detection model can be used in actual biological detection tasks.

[0146] Furthermore, by verifying and evaluating the performance of the biological detection model through independent validation sets and test sets, it can be ensured that the biological detection model can maintain stable performance in practical applications. When the validation or test results are not ideal, it indicates that the biological detection model may need to adjust the model structure, loss function or training strategy, etc., to further optimize the model performance of the biological detection model.

[0147] According to the technical solution provided in the embodiment of the present application, the biological detection model is trained by a training sample set, and the training result of the biological detection model is determined by the loss value of each training, so as to ensure the versatility and accuracy of the biological detection model.

[0148] In some embodiments, obtaining inter-frame information of multiple frames of images to be detected includes:

[0149] Acquire image data of an intermediate frame to be detected having front and rear adjacent frames among the multiple frames of images to be detected, and acquire image data of a previous frame and a subsequent frame of the intermediate frame to be detected;

[0150] Determine the average value of the image data of the previous frame image and the image data of the next frame image of the intermediate frame to be detected, and calculate the difference between the average value and the image data of the intermediate frame image to be detected;

[0151] The preset power of the difference is determined as the inter-frame information of the middle frame to be detected image, the previous frame image of the middle frame to be detected image, and the next frame image; wherein, the middle frame to be detected image is separated from the previous frame image and the next frame image by a preset frame.

[0152] Specifically, based on multiple frames of images to be detected, the inter-frame information of the multiple frames of images to be detected can be calculated by the following formula:

[0153] ;

[0154] Represents image data of an intermediate frame to be detected having adjacent frames in front and behind among multiple frames of images to be detected, Represents the image data of the previous frame of the image to be detected in the middle frame, The image data represents the next frame image of the intermediate frame image to be detected, and the intermediate frame image to be detected is separated from the previous frame image and the next frame image by a preset frame.

[0155] Specifically, the inter-frame information can feedback the dynamic scenes or objects between multiple frames of images to be detected, and the continuous visual information that changes over time can be determined by optical flow estimation, feature matching, change detection, and inter-frame difference. The preset interval frames can be a custom interval frame number such as 3 frames, 5 frames, etc. The superscript 3 in the above formula is a power obtained through experiments.

[0156] Since animal detection events are mainly concentrated in the case of insufficient illumination, low signal suppression processing is performed on the inter-frame information. By performing inter-frame difference calculation on three consecutive frames of images, the inter-frame information of all consecutive multiple frames of images to be detected is determined.

[0157] According to the technical solution provided in the embodiments of the present application, inter-frame information can be determined through image data in multiple frames of images to be detected, so that the biological detection model can better understand the motion patterns in dynamic scenes, perform video analysis and anomaly detection, etc.

[0158] In some embodiments, after outputting the target detection results of the target to be detected in multiple frames of images to be detected based on the target feature data set through the fusion prediction module of the biological detection model, it also includes:

[0159] Sending a video clip including multiple frames of images to be detected and prompt information to the user's client through the cloud platform, wherein the prompt information is used to remind the user that there is a creature approaching outside the vehicle; and,

[0160] Control the vehicle's external audio system to emit an expulsion warning sound, which is used to drive away creatures outside the vehicle.

[0161] Specifically, a video clip including multiple frames of images to be detected and prompt information are sent to the user's client through the cloud platform, wherein the multiple frames of images to be detected are images including target detection results. For example, if the target detection result is that a mouse is under the chassis of a vehicle, the mouse will be marked in the form of a marking box in the multiple frames of images to be detected, and a video clip including the mouse will be intercepted and sent to the user's client through the cloud platform. At the same time, a prompt information of "The mouse is under the chassis of the vehicle, please be careful of the mouse" is generated, and the prompt information is also sent to the user's client through the cloud platform, so that the user can obtain information about creatures outside the vehicle more accurately, and the prompt information is sent to the client through the cloud platform so that the user can understand the situation of the creatures outside the vehicle at any time.

[0162] The client can be a mobile phone, tablet computer or vehicle-mounted display screen and other devices, so as to view video clips and prompt information and respond in time.

[0163] In addition, while sending the video clip and prompt information, the vehicle's external audio system is controlled to emit a warning sound, which can be a warning sound with a specific frequency and loudness, so as to attract the attention of creatures outside the vehicle and prompt them to stay away from the vehicle. For example, for mice, simulated natural enemy sounds or high-frequency noises can be played to drive them away, thereby providing protection for the vehicle when the user is not there.

[0164] Furthermore, after the expulsion prompt sound is issued and the video clip and prompt information are sent, the vehicle's surrounding environment can continue to be monitored to confirm the expulsion effect. If the target creature has not left or a new threat situation has emerged, the prompt information can be sent repeatedly or other countermeasures can be taken.

[0165] According to the technical solution provided in the embodiment of the present application, the target detection results of the biological detection model can be applied in the vehicle safety system, and collaborate with the cloud platform, the user's client and the vehicle hardware to improve the safety and usage experience of the user's use of the vehicle.

[0166] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0167] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0168] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0169] Figure 8 Schematic diagram of an off-vehicle biological detection device provided in an embodiment of the present application. Figure 8 As shown, the off-vehicle biological detection device includes: a collection module 801, a first processing module 802, a second processing module 803, a third processing module 804, and an output module 805, wherein:

[0170] The acquisition module 801 is configured to acquire multiple frames of images to be detected and inter-frame information of the multiple frames of images to be detected acquired by a camera outside the vehicle;

[0171] The first processing module 802 is configured to input a plurality of frames of images to be detected and inter-frame information of the plurality of frames of images to be detected into a first input module of the trained biological detection model to obtain first feature data;

[0172] The second processing module 803 is configured to input the middle frame to be detected image having the front and rear adjacent frames and the inter-frame information in the multiple frames to be detected images into the second input module of the biological detection model to obtain the second feature data;

[0173] The third processing module 804 is configured to obtain dynamic feature data based on the first feature data, obtain static feature data based on the second feature data, and obtain a target feature data group based on the static feature data and the dynamic feature data through the dual-channel network module of the biological detection model, wherein the target feature data group includes a plurality of target feature data;

[0174] The output module 805 is configured to output the target detection results of the target to be detected in multiple frames of images to be detected based on the target feature data group through the fusion prediction module of the biological detection model.

[0175] In some embodiments, the dual-channel network module includes a static channel network module, a dynamic channel network module and a feature output module, the dynamic channel network module includes a first convolution module, a second convolution module and a third convolution module, and the static channel network module includes a fourth convolution module, a fifth convolution module and a sixth convolution module;

[0176] The third processing module 804 is configured to:

[0177] Performing convolution processing on the first feature data through a first convolution module to obtain dynamic feature data;

[0178] Performing convolution processing on the second feature data through a fourth convolution module to obtain static feature data;

[0179] Through the second convolution module, convolution processing is performed on the first merged data after the up-sampled static feature data is merged with the dynamic feature data to obtain the first fused feature data;

[0180] Through the fifth convolution module, convolution processing is performed on the second merged data obtained by merging the down-sampled dynamic feature data with the static feature data to obtain second fused feature data;

[0181] Through the three convolution modules, convolution processing is performed on the third combined data obtained by combining the upsampled data of the second fused feature data with the first fused feature data to obtain the third fused feature data;

[0182] Through the sixth convolution module, convolution processing is performed on the fourth combined data obtained by combining the data obtained by downsampling the first fused feature data and the second fused feature data to obtain fourth fused feature data;

[0183] Through the feature output module, a target feature data group is obtained based on the static feature data, the first fused feature data, the second fused feature data, the third fused feature data and the fourth fused feature data.

[0184] In some embodiments, the feature output module includes a seventh convolution module, an eighth convolution module, and a ninth convolution module.

[0185] The third processing module 804 is configured to:

[0186] Through the seventh convolution module, convolution processing is performed on the fifth combined data obtained by combining the first fused feature data and the static feature data to obtain the first target feature data;

[0187] Through the eighth convolution module, convolution processing is performed on the sixth combined data obtained by combining the third fused feature data and the second fused feature data to obtain second target feature data;

[0188] Through a ninth convolution module, convolution processing is performed on the seventh combined data obtained by combining the data obtained by downsampling the third fused feature data and the fourth fused feature data to obtain third target feature data;

[0189] The target feature data group includes first target feature data, second target feature data and third target feature data.

[0190] In some embodiments, the target feature data group includes first target feature data, second target feature data and third target feature data; the fusion prediction module includes a first fusion convolution block, a second fusion convolution block, a third fusion convolution block, a fourth fusion convolution block, a fifth fusion convolution block, a sixth fusion convolution block and a prediction module.

[0191] The output module 805 is configured to:

[0192] Through the first fusion convolution block, convolution processing is performed on the data obtained by upsampling the third target feature data and the eighth combined data obtained by combining the second target feature data to obtain first intermediate data;

[0193] Through the second fusion convolution block, convolution processing is performed on the ninth combined data obtained by combining the data obtained by upsampling the first intermediate data with the first target feature data to obtain second intermediate data;

[0194] Performing convolution processing on the second intermediate data through a third fused convolution block to obtain third intermediate data;

[0195] Performing convolution processing on the tenth combined data obtained by combining the third intermediate data with the first intermediate data through a fourth fusion convolution block to obtain fourth intermediate data;

[0196] Performing convolution processing on the fourth intermediate data through a fifth fused convolution block to obtain fifth intermediate data;

[0197] Performing convolution processing on the eleventh combined data obtained by combining the fifth intermediate data and the third target feature data through a sixth fusion convolution block to obtain sixth intermediate data;

[0198] Through the prediction module, based on the second intermediate data, the fourth intermediate data and the sixth intermediate data, the target detection results of the target to be detected in the multiple frames of the image to be detected are obtained.

[0199] In some embodiments, the first processing module 802 inputs the multiple frames of images to be detected and the inter-frame information of the multiple frames of images to be detected into the first input module of the trained biological detection model, and before obtaining the first feature data, is further configured as follows:

[0200] Acquire a training sample set, the training sample set including multiple frames of sample images, inter-frame information of the multiple frames of sample images, and position annotation information of the target to be detected in the sample images;

[0201] The biological detection model is trained through the training sample set to determine the loss value of the biological detection model, wherein the loss value is the weighted sum of the classification loss, the confidence loss and the position loss value; when the loss value is detected to be less than the preset loss value, it is determined that the training of the biological detection model is completed.

[0202] In some embodiments, the acquisition module 801 is configured to:

[0203] Acquire image data of an intermediate frame to be detected having front and rear adjacent frames among the multiple frames of images to be detected, and acquire image data of a previous frame and a subsequent frame of the intermediate frame to be detected;

[0204] Determine the average value of the image data of the previous frame image and the image data of the next frame image of the intermediate frame to be detected, and calculate the difference between the average value and the image data of the intermediate frame image to be detected;

[0205] Determine the preset power of the difference as the inter-frame information of the middle frame to be detected image, the previous frame image and the next frame image of the middle frame to be detected;

[0206] There are preset frames between the middle frame image to be detected and the previous frame image and the next frame image.

[0207] In some embodiments, after the output module 805 outputs the target detection results of the target to be detected in the multiple frames of the images to be detected through the fusion prediction module of the biological detection model based on the target feature data group, it is further configured to:

[0208] Sending a video clip including multiple frames of images to be detected and prompt information to the user's client through the cloud platform, wherein the prompt information is used to remind the user that there is a creature approaching outside the vehicle; and,

[0209] Control the vehicle's external audio system to emit an expulsion warning sound, which is used to drive away creatures outside the vehicle.

[0210] Fig. 9 Schematic diagram of an electronic device 9 provided in an embodiment of the present application. Fig. 9 As shown, the electronic device 9 of this embodiment includes: a processor 901, a memory 902, and a computer program 903 stored in the memory 902 and executable on the processor 901. When the processor 901 executes the computer program 903, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 901 executes the computer program 903, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0211] The electronic device 9 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 9 may include, but is not limited to, a processor 901 and a memory 902. Those skilled in the art will appreciate that Fig. 9 The electronic device 9 is merely an example and does not limit the electronic device 9 , and may include more or less components than those shown in the figure, or different components.

[0212] The processor 901 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0213] The memory 902 may be an internal storage unit of the electronic device 9, for example, a hard disk or memory of the electronic device 9. The memory 902 may also be an external storage device of the electronic device 9, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 9. The memory 902 may also include both an internal storage unit of the electronic device 9 and an external storage device. The memory 902 is used to store computer programs and other programs and data required by the electronic device.

[0214] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0215] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a readable storage medium). Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0216] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for detecting biological substances outside a vehicle, It is characterized in that include: Acquire multiple frames of images to be detected and inter-frame information of the multiple frames of images to be detected collected by a camera outside the vehicle; Inputting the plurality of frames of images to be detected and the inter-frame information of the plurality of frames of images to be detected into a first input module of the trained biological detection model to obtain first feature data; Inputting the middle frame image to be detected having the front and rear adjacent frames among the multiple frames of images to be detected and the inter-frame information into the second input module of the biological detection model to obtain second feature data; By using the dual-channel network module of the biological detection model, dynamic feature data is obtained based on the first feature data, static feature data is obtained based on the second feature data, and a target feature data group is obtained based on the static feature data and the dynamic feature data, wherein the target feature data group includes a plurality of target feature data from different dimensions; Outputting target detection results of the target to be detected in the multiple frames of images to be detected based on the target feature data group through the fusion prediction module of the biological detection model; The step of obtaining dynamic feature data based on the first feature data and obtaining static feature data based on the second feature data includes: By upsampling, part of the static channel feature layer is added to the dynamic channel feature layer, and by downsampling, part of the dynamic channel feature layer is added to the static channel feature layer; The step of obtaining a target feature data group based on the static feature data and the dynamic feature data includes: The static feature data and the dynamic feature data are integrated to obtain target feature data, wherein the multiple target feature data include first target feature data from a shallow network module, second target feature data from a middle network module, and third target feature data from a deep network module.

2. The method according to claim 1, It is characterized in that The dual-channel network module includes a static channel network module, a dynamic channel network module and a feature output module, the dynamic channel network module includes a first convolution module, a second convolution module and a third convolution module, and the static channel network module includes a fourth convolution module, a fifth convolution module and a sixth convolution module; The dual-channel network module of the biological detection model obtains dynamic feature data based on the first feature data, obtains static feature data based on the second feature data, and obtains a target feature data group based on the static feature data and the dynamic feature data, including: Performing convolution processing on the first feature data through the first convolution module to obtain the dynamic feature data; Performing convolution processing on the second feature data through the fourth convolution module to obtain the static feature data; By means of the second convolution module, convolution processing is performed on the first merged data obtained by merging the upsampled static feature data with the dynamic feature data to obtain first fused feature data; By means of the fifth convolution module, convolution processing is performed on the second merged data obtained by merging the down-sampled data of the dynamic feature data with the static feature data to obtain second fused feature data; Through the three convolution modules, convolution processing is performed on the third combined data obtained by combining the data obtained by upsampling the second fused feature data with the first fused feature data to obtain third fused feature data; By means of the sixth convolution module, convolution processing is performed on the fourth combined data obtained by combining the data obtained by downsampling the first fused feature data and the second fused feature data to obtain fourth fused feature data; The target feature data group is obtained through the feature output module based on the static feature data, the first fused feature data, the second fused feature data, the third fused feature data and the fourth fused feature data.

3. The method according to claim 2, It is characterized in that The feature output module includes a seventh convolution module, an eighth convolution module and a ninth convolution module; The target feature data group is obtained by the feature output module based on the static feature data, the first fused feature data, the second fused feature data, the third fused feature data and the fourth fused feature data, including: By means of the seventh convolution module, convolution processing is performed on fifth combined data obtained by combining the first fused feature data and the static feature data to obtain first target feature data; By means of the eighth convolution module, convolution processing is performed on sixth combined data obtained by combining the third fused feature data and the second fused feature data to obtain second target feature data; By means of the ninth convolution module, convolution processing is performed on the seventh combined data obtained by combining the data obtained by downsampling the third fused feature data with the fourth fused feature data to obtain third target feature data; The target feature data group includes the first target feature data, the second target feature data and the third target feature data.

4. The method according to claim 1, It is characterized in that The target feature data group includes first target feature data, second target feature data and third target feature data; the fusion prediction module includes a first fusion convolution block, a second fusion convolution block, a third fusion convolution block, a fourth fusion convolution block, a fifth fusion convolution block, a sixth fusion convolution block and a prediction module; The fusion prediction module of the biological detection model outputs the target detection results of the target to be detected in the multiple frames of images to be detected based on the target feature data group, including: By using the first fusion convolution block, convolution processing is performed on the data obtained by upsampling the third target feature data and the eighth combined data obtained by combining the second target feature data to obtain first intermediate data; By using the second fusion convolution block, convolution processing is performed on the ninth combined data obtained by combining the data obtained by upsampling the first intermediate data with the first target feature data to obtain second intermediate data; Performing convolution processing on the second intermediate data through the third fused convolution block to obtain third intermediate data; By means of the fourth fusion convolution block, convolution processing is performed on tenth combined data obtained by combining the third intermediate data with the first intermediate data to obtain fourth intermediate data; Performing convolution processing on the fourth intermediate data through the fifth fused convolution block to obtain fifth intermediate data; By means of the sixth fusion convolution block, convolution processing is performed on the eleventh combined data obtained by combining the fifth intermediate data and the third target feature data to obtain sixth intermediate data; Through the prediction module, based on the second intermediate data, the fourth intermediate data and the sixth intermediate data, the target detection results of the target to be detected in the multiple frames of images to be detected are obtained.

5. The method according to claim 1, It is characterized in that Before inputting the plurality of frames of images to be detected and the inter-frame information of the plurality of frames of images to be detected into the first input module of the trained biological detection model to obtain the first feature data, the method further includes: Acquire a training sample set, wherein the training sample set includes multiple frames of sample images, inter-frame information of the multiple frames of sample images, and position annotation information of a target to be detected in the sample images; Training the biological detection model through the training sample set to determine a loss value of the biological detection model, wherein the loss value is a weighted sum of classification loss, confidence loss, and position loss value; When it is detected that the loss value is less than the preset loss value, it is determined that the biological detection model training is completed.

6. The method according to claim 1, It is characterized in that Obtain inter-frame information of multiple frames of images to be detected, including: Acquire image data of an intermediate frame to be detected having front and rear adjacent frames among the multiple frames of images to be detected, and acquire image data of a previous frame and a subsequent frame of the intermediate frame to be detected; Determine the average value of the image data of the previous frame image and the image data of the next frame image of the intermediate frame to be detected, and calculate the difference between the average value and the image data of the intermediate frame image to be detected; Determine the preset power of the difference as the inter-frame information of the intermediate frame to be detected image, the previous frame image and the next frame image of the intermediate frame to be detected image; There are preset frames between the intermediate frame image to be detected and the previous frame image and the next frame image.

7. The method according to claim 1, It is characterized in that After the target detection results of the target to be detected in the multiple frames of images to be detected are outputted by the fusion prediction module of the biological detection model based on the target feature data group, the method further includes: Sending the video clips including the multiple frames of images to be detected and prompt information to the user's client through the cloud platform, wherein the prompt information is used to remind the user that there is a creature approaching outside the vehicle; and The external audio system of the vehicle is controlled to emit an expulsion prompt sound, wherein the expulsion prompt sound is used to expel creatures outside the vehicle.

8. An off-vehicle biological detection device, It is characterized in that include: An acquisition module is configured to acquire multiple frames of images to be detected and inter-frame information of the multiple frames of images to be detected acquired by a camera outside the vehicle; A first processing module is configured to input the multiple frames of images to be detected and the inter-frame information of the multiple frames of images to be detected into a first input module of the trained biological detection model to obtain first feature data; A second processing module is configured to input the middle frame image to be detected having the front and rear adjacent frames among the multiple frames of images to be detected and the inter-frame information into a second input module of the biological detection model to obtain second feature data; a third processing module, configured to obtain dynamic feature data based on the first feature data, obtain static feature data based on the second feature data, and obtain a target feature data group based on the static feature data and the dynamic feature data through the dual-channel network module of the biological detection model, wherein the target feature data group includes a plurality of target feature data from different dimensions; An output module is configured to output target detection results of the target to be detected in the multiple frames of images to be detected based on the target feature data group through the fusion prediction module of the biological detection model; The third processing module is configured to obtain dynamic feature data based on the first feature data, and obtain static feature data based on the second feature data, including: adding part of the static channel feature layer to the dynamic channel feature layer by upsampling, and adding part of the dynamic channel feature layer to the static channel feature layer by downsampling; The third processing module is configured to obtain a target feature data group based on the static feature data and the dynamic feature data, including: fusing the static feature data and the dynamic feature data to obtain the target feature data, wherein the multiple target feature data include a first target feature data from a shallow network module, a second target feature data from a middle network module, and a third target feature data from a deep network module.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images