Animal detection method and device, vehicle-mounted networking control equipment and vehicle
By using sample images under different angles and light training in animal detection models, the problem of low detection accuracy in high-speed driving scenarios is solved, resources are saved, angle and light changes are adapted to, and detection accuracy is improved.
Patent Information
- Application Number
- CN202411042349.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-26
AI Technical Summary
The existing animal detection algorithm has low accuracy in detecting rare animals in high-speed driving scenarios, and requires a large amount of manpower and material resources for text annotation and model training.
By obtaining the scene image in front of the vehicle, training is performed using a pre-constructed animal detection model. The model includes a feature extraction module, a linear layer and a classification module. It uses sample images at different angles and lighting for training, and calculates similarity and cross-entropy loss values. , adjust model parameters to adapt to changes in high-speed driving scenarios, and reduce special training and sample annotation for single animals.
It improves the accuracy of animal detection in high-speed driving scenarios of vehicles, saves manpower and material resources, and adapts to angle and light changes.
Smart Images

Figure CN119027912A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of animal detection technology, and in particular to an animal detection method, device, vehicle-mounted networking control equipment and vehicle. Background Art
[0002] In the case of automatic assisted driving, due to the extremely complex driving environment, the detection algorithm must be able to quickly and accurately capture rare animals that suddenly appear on the road during driving, especially rare animals that appear on high-speed roads. That is, when the detection algorithm detects rare animals appearing on the road at a distance, it will immediately detect relevant information and pass the detection results to the driver after data analysis, which can help the driver make correct decisions, avoid collisions with these rare animals, and ensure driving safety. In actual applications, the vehicle collects animal images through the on-board camera, and sends the animal images to the cloud server or on-board controller through the Internet of Vehicles technology. The animal images are analyzed by the detection algorithm set in the cloud server or on-board controller to obtain the detection results.
[0003] Current detection algorithms usually perform text annotation on sample images of rare animals after collecting them, train detection models based on sample images with text annotation information, and detect rare animal images using the trained detection model. Since the shooting angle and lighting conditions of animal images collected in high-speed vehicle driving scenarios change rapidly, the above detection method has low detection accuracy in high-speed vehicle driving scenarios. In addition, due to the large number of wild protected animals, labeling each rare animal and conducting model training requires a lot of manpower and material resources, as well as a large amount of computing resources.
[0004] Therefore, there is a need to improve current animal detection methods. Summary of the invention
[0005] In view of the above-mentioned shortcomings of the prior art, the present application provides an animal detection method, device, vehicle-mounted networking control equipment and vehicle to solve the above-mentioned technical problems.
[0006] The present application provides an animal detection method, which includes: acquiring a scene image of a target animal in front of a vehicle, wherein the target animal includes an animal on the road; inputting the scene image as an image to be detected into an animal detection model to obtain a detection result; the animal detection model is obtained by training a pre-constructed detection model based on sample images of the target animal, wherein the sample images include images at different angles and under different lighting conditions.
[0007] In one embodiment of the present application, if the pre-constructed detection model includes a feature extraction module, a linear layer and a classification module, the process of training the pre-constructed detection model based on the sample image of the target animal includes: inputting the sample image into the feature extraction module to obtain a multi-layer feature map, and inputting the multi-layer feature map into the linear layer to obtain a plurality of feature vectors; calculating the similarity between the plurality of feature vectors, and calculating a similarity loss value based on the plurality of similarities; inputting the multi-layer feature map, the first preset position vector, the second preset position vector, and the preset classification vector into the classification module to obtain an adjusted classification vector, and calculating a cross entropy loss value based on the adjusted classification vector; the first preset position vector and the second preset position vector have a preset mapping relationship; taking the sum of the similarity loss value and the cross entropy loss value as the loss result, and adjusting the parameters of the feature extraction module, the parameters of the linear layer and the parameters of the classification module according to the loss result; combining the adjusted feature extraction module, the adjusted linear layer and the adjusted classification module to obtain the animal detection model.
[0008] In one embodiment of the present application, if the feature extraction module includes a large-kernel convolution module and a neural network module, the sample image is input into the feature extraction module to obtain a multi-layer feature map, including: inputting the sample image into the large-kernel convolution module to obtain a multi-scale feature map; inputting the multi-scale feature map into the neural network module to obtain multiple layers of the feature map; the neural network module includes a convolution layer and a pooling layer, and the convolution kernel of the large-kernel convolution module is larger than the convolution kernel in the convolution layer.
[0009] In one embodiment of the present application, based on the multiple similarities, the process of calculating the similarity loss value includes: inputting the multiple similarities into a preset contrast loss function to obtain multiple contrast similarities; selecting some contrast similarities from the multiple contrast similarities, and taking the average value of the selected some contrast similarities as the similarity loss value.
[0010] In one embodiment of the present application, if the classification module includes a splicing layer, a first encoding layer, an attention mechanism layer and a second encoding layer, then the multiple layers of the feature map, the first preset position vector, the second preset position vector, and the preset classification vector are input into the classification module to obtain the adjusted classification vector. The process includes: obtaining the key-value pair of the target animal and converting the key-value pair into a category number vector; the key-value pair is used to characterize the mapping relationship between the category of the target animal and the number of the target animal; the multiple layers of the feature map, the first preset position vector, the second preset position vector, the preset classification vector, and the category number vector are input into the splicing layer to obtain a splicing vector; the splicing vector is input into the first encoding layer to obtain an encoded splicing vector; the encoded splicing vector is input into the attention mechanism layer to obtain a weighted vector; the weighted vector is input into the second encoding layer to obtain an encoded weighted vector; the classification vector is extracted from the encoded weighted vector to obtain the adjusted classification vector.
[0011] In one embodiment of the present application, multiple layers of the feature map, the first preset position vector, the second preset position vector, the preset classification vector, and the category number vector are input into the splicing layer, and the process of obtaining the splicing vector includes: embedding the first preset position vector into the category number vector to obtain the category number embedding vector; embedding the second preset position vector into the multiple layers of the feature map to obtain the multiple layers of feature embedding vector; splicing the preset classification vector, the category number embedding vector, and the multiple layers of the feature embedding vector to obtain the splicing vector.
[0012] In one embodiment of the present application, based on the adjusted classification vector, the process of calculating the cross entropy loss value includes: calculating the predicted probability of the sample category to which the sample image belongs through the adjusted classification vector to obtain multiple predicted probabilities; calculating the cross entropy of the multiple predicted probabilities to obtain the cross entropy loss value.
[0013] According to one aspect of an embodiment of the present application, an animal detection device is provided, comprising: an image acquisition module for acquiring a scene image of a target animal in front of a vehicle, wherein the target animal includes an animal on a road surface; an image detection module for inputting the scene image as an image to be detected into an animal detection model to obtain a detection result; the animal detection model is obtained by training a pre-constructed detection model based on sample images of the target animal, wherein the sample images include images at different angles and under different lighting conditions.
[0014] According to one aspect of an embodiment of the present application, a vehicle-mounted networked control device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the vehicle-mounted networked control device implements the animal detection method as described above.
[0015] According to one aspect of an embodiment of the present application, a vehicle is provided, the vehicle comprising the animal detection device as described above or the vehicle-mounted networking control device as described above.
[0016] The beneficial effects of the present application are as follows: the present application obtains a scene image of a target animal in front of a vehicle, inputs the scene image as an image to be detected into an animal detection model, and obtains a detection result. The animal detection model is obtained by training a pre-constructed detection model based on sample images of the target animal, wherein the sample images include images at different angles and under different lighting conditions. In the above process, the pre-constructed detection model is trained by using images of the target animal at different angles and under different lighting conditions, so that the trained animal detection model can adapt to changes in shooting angles and lighting conditions of animal images in scenarios where the vehicle is traveling at high speed, thereby improving the detection accuracy of animal images in scenarios where the vehicle is traveling at high speed. In addition, there is no need to train on images of a single type of animal, nor is there any need to pre-label sample images, thereby saving manpower, material resources and computing resources.
[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0019] Figure 1 is a schematic diagram of an exemplary system architecture shown in an exemplary embodiment of the present application;
[0020] Figure 2 is a flow chart of an animal detection method shown in an exemplary embodiment of the present application;
[0021] Figure 3 is a flow chart of an animal detection method shown in another exemplary embodiment of the present application;
[0022] Figure 4 is a schematic diagram showing a combination of a feature extraction module and a linear layer as shown in an exemplary embodiment of the present application;
[0023] Figure 5 is a schematic diagram showing a similarity matrix composed of multiple similarities according to an exemplary embodiment of the present application;
[0024] Figure 6 is a schematic diagram of a classification module shown in an exemplary embodiment of the present application;
[0025] Figure 7 is a block diagram of an animal detection device shown in an exemplary embodiment of the present application;
[0026] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing an in-vehicle networking control device of an embodiment of the present application is shown. DETAILED DESCRIPTION
[0027] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.
[0028] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.
[0029] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.
[0030] Figure 1 It is a schematic diagram of an exemplary system architecture shown in an exemplary embodiment of the present application.
[0031] Reference Figure 1As shown, the system architecture may include a collection device 101 and a vehicle controller 102. The vehicle controller 102 may be at least one of a desktop graphics processing unit (GPU) computer, a GPU computing cluster, a neural network computer, etc. Relevant technicians may use the vehicle controller 102 to obtain a scene image of the target animal in front of the vehicle, input the scene image as an image to be detected into an animal detection model, and obtain a detection result. The animal detection model is obtained by training a pre-built detection model based on a sample image of the target animal. The collection device 101 is used to collect the scene image of the target animal in front of the vehicle. In this embodiment, the collection device 101 uses a camera, a video camera, etc. to collect the scene image of the target animal in front of the vehicle, and after collecting the scene image of the target animal in front of the vehicle, it is provided to the vehicle controller 102 for processing.
[0032] Illustratively, after acquiring the scene image of the target animal in front of the vehicle in the acquisition device 101, the on-board controller 102 acquires the scene image of the target animal in front of the vehicle, and inputs the scene image as the image to be detected into the animal detection model to obtain a detection result. The animal detection model is obtained by training a pre-constructed detection model based on the sample image of the target animal. In the above process, the pre-constructed detection model is trained through images of the target animal at different angles and under different lighting conditions, so that the trained animal detection model can adapt to the changes in shooting angles and lighting conditions of animal images in vehicle high-speed driving scenarios, thereby improving the detection accuracy of animal images in vehicle high-speed driving scenarios; in addition, there is no need to train on images of a single type of animal, nor is there any need to pre-label sample images, thereby saving manpower, material resources and computing resources.
[0033] It should be noted that the animal detection method provided in the embodiment of the present application is generally executed by the vehicle-mounted controller 102 , and accordingly, the animal detection device is generally disposed in the vehicle-mounted controller 102 .
[0034] The implementation details of the technical solution of the embodiment of the present application are described in detail below:
[0035] Figure 2 is a flowchart of an animal detection method shown in an exemplary embodiment of the present application. The animal detection method can be executed by a computing processing device. The computing processing device can be Figure 1 The vehicle controller 102 shown in FIG. Figure 2 As shown, the animal detection method at least includes steps S210 to S220, which are described in detail as follows:
[0036] In step S210, a scene image of a target animal in front of the vehicle is acquired.
[0037] In one embodiment of the present application, the target animals include animals on the road. The animals on the road include but are not limited to rare animals, which are nationally protected animals.
[0038] In step S220, the scene image is input into the animal detection model as the image to be detected to obtain a detection result.
[0039] In this embodiment, the animal detection model is obtained by training a pre-built detection model based on sample images of the target animal, and the sample images include images at different angles and under different lighting conditions.
[0040] In this embodiment, a pre-constructed detection model is trained through images of target animals at different angles and under different lighting conditions, so that the trained animal detection model can adapt to changes in shooting angles and lighting conditions of animal images in scenarios where vehicles are traveling at high speeds, thereby improving the detection accuracy of animal images in scenarios where vehicles are traveling at high speeds; in addition, there is no need to train on images of a single type of animal, nor is there any need to pre-label sample images, thereby saving manpower, material resources and computing resources.
[0041] In one embodiment of the present application, if the pre-built detection model includes a feature extraction module, a linear layer, and a classification module, the process of training the pre-built detection model based on the sample image of the target animal includes:
[0042] The sample image is input into the feature extraction module to obtain a multi-layer feature map, and the multi-layer feature map is input into the linear layer to obtain multiple feature vectors.
[0043] In this embodiment, before the sample image is input into the feature extraction module for feature extraction, it is necessary to perform a random enhancement operation on the sample image. The random enhancement operation includes but is not limited to enhancement methods such as rotation and color change, and the enhanced sample image is used to expand the road scene image data set containing long-distance rare animals.
[0044] In this embodiment, long-distance rare animals include rare animals whose distance from the vehicle is greater than a preset distance threshold. When the vehicle is driving at high speed, collecting road scene images of long-distance rare animals is beneficial for enhancing and detecting the scene images, and reserving sufficient decision-making time for the driver after obtaining the detection results.
[0045] In this embodiment, when a vehicle makes an emergency stop on a highway, the collected road scene images of rare animals may be rare animals whose distance from the vehicle is greater than a preset distance threshold, or may be rare animals whose distance from the vehicle is less than or equal to a preset distance threshold, without limitation here.
[0046] The similarities between multiple feature vectors are calculated, and based on the multiple similarities, a similarity loss value is calculated.
[0047] In this embodiment, the process of calculating the similarity between multiple feature vectors includes: transforming the length of each feature vector (for example, transforming the length of the feature vector from 1×1024 to 1×128 or 1×256, etc.), obtaining multiple transformed feature vectors, and calculating the similarity between every two transformed feature vectors to obtain multiple similarities.
[0048] In this embodiment, the similarity can be measured by calculating cosine similarity, mutual information, or Pearson correlation coefficient.
[0049] In this embodiment, taking the calculation of cosine similarity as an example, the expression of similarity is as follows:
[0050]
[0051] Among them, S a,b represents the similarity between the feature vector after the ath transformation and the feature vector after the bth transformation, g a represents the ath transformed eigenvector, g b represents the b-th transformed eigenvector, represents the transpose of the a-th transformed eigenvector, μ is an adjustable temperature parameter that affects the distribution of cosine similarity and can extend the similarity to [-1, 1], and ||·|| is the modulus of the eigenvector.
[0052] The multi-layer feature map, the first preset position vector, the second preset position vector, and the preset classification vector are input into the classification module to obtain an adjusted classification vector, and the cross entropy loss value is calculated based on the adjusted classification vector.
[0053] In this embodiment, the first preset position vector and the second preset position vector have a preset mapping relationship, and the preset mapping relationship is used to characterize the correlation relationship between the first preset position vector and the second preset position vector in terms of position, and the preset mapping relationship is characterized by a cosine function.
[0054] The sum of the similarity loss value and the cross entropy loss value is taken as the loss result, and the parameters of the feature extraction module, the parameters of the linear layer, and the parameters of the classification module are adjusted according to the loss result.
[0055] In this embodiment, the process of adjusting the parameters of the feature extraction module, the parameters of the linear layer, and the parameters of the classification module according to the loss result includes: (1) setting a parameter adjustment iteration threshold, and taking the loss result as a first evaluation result, adjusting the parameters of the feature extraction module, the parameters of the linear layer, and the parameters of the classification module once, and recording the number of parameter adjustments as one; (2) after adjusting the parameters of the feature extraction module, the parameters of the linear layer, and the parameters of the classification module once, inputting the sample image into the once-adjusted feature extraction module, the once-adjusted linear layer, and the once-adjusted classification module in sequence, obtaining a plurality of adjusted feature vectors and a second-adjusted classification vector, and calculating the adjusted loss result based on the plurality of adjusted feature vectors and the second-adjusted classification vector, and taking the adjusted loss result as a second evaluation result, and adjusting the once-adjusted feature extraction module parameters, the once-adjusted linear layer parameters, and the once-adjusted classification module parameters again according to the second evaluation result, and updating the number of parameter adjustments until the updated number of parameter adjustments reaches the parameter adjustment iteration threshold, and obtaining the adjusted feature extraction module, the adjusted linear layer, and the adjusted classification module;
[0056] The adjusted feature extraction module, the adjusted linear layer, and the adjusted classification module are combined to obtain an animal detection model.
[0057] In this embodiment, the adjusted feature extraction module, the adjusted linear layer, and the adjusted classification module are combined, and the obtained animal detection model can improve the detection effect of rare animals at a long distance in the scene of high-speed vehicle driving.
[0058] In one embodiment of the present application, if the feature extraction module includes a large kernel convolution module and a neural network module, the process of inputting the sample image into the feature extraction module to obtain a multi-layer feature map includes:
[0059] The sample image is input into the large kernel convolution module to obtain a multi-scale feature map.
[0060] In this embodiment, a large kernel convolution module (LKCM) generally refers to a convolution layer using a larger convolution kernel size, which can capture long-distance dependencies in sample images and thus perform well in visual tasks. The large kernel convolution module has the following characteristics: (1) The large kernel convolution module can capture dependencies between pixels farther away in the sample image, which is very important for understanding the global structure and semantic information of the image; (2) By increasing the size of the convolution kernel, the large kernel convolution module can extract richer feature information, which has a positive effect on improving the recognition performance and generalization ability of the model. The convolution kernel of the large kernel convolution module is larger than the convolution kernel in the convolution layer, or the size of the convolution kernel in the large kernel convolution module is larger than the preset convolution kernel size.
[0061] In this embodiment, the expression of the multi-scale feature map is as follows:
[0062] U0=LKCM(X) Formula (2)
[0063] Among them, U0 represents a multi-scale feature map, LKCM represents a large kernel convolution module, and X represents a sample image, which is a scene image of rare animals on the road whose distance from the vehicle is greater than a preset distance threshold.
[0064] The multi-scale feature map is input into the neural network module to obtain a multi-layer feature map.
[0065] In this embodiment, the neural network module includes a convolution layer and a pooling layer, wherein the convolution layer is used to extract local features of the multi-scale feature map to obtain a multi-layer local feature map, and the pooling layer is used to perform dimensionality reduction processing on the multi-layer local feature map to obtain a multi-layer feature map.
[0066] In this embodiment, taking the feature map of the i+1th layer as an example, the expression of the feature map of the i+1th layer is as follows:
[0067] U i+1 =U i *S i Formula (3)
[0068] Among them, U i+1 represents the feature map of the i+1th layer, U i represents the feature map of the i-th layer, S i It represents the superposition tensor of the tensor of the feature map of the i-th layer after two large kernel convolutions and the tensor of the feature map of the i-th layer after one large kernel convolution in the spatial dimension.
[0069] In this embodiment, the superposition tensor expression of the i-th layer feature map is as follows:
[0070]
[0071] Among them, S i represents the superposition tensor of the feature map of the i-th layer, represents a convolutional layer with a convolution kernel step size k and a dilation rate d, ω represents an adjustable parameter of the convolutional layer, U i Represents the feature map of the i-th layer.
[0072] In this embodiment, after obtaining the multi-layer feature map, the multi-layer feature map is input into the linear layer to obtain multiple feature vectors. Taking the i+1th layer feature map as an example, the expression of the feature vector corresponding to the i+1th layer feature map is as follows:
[0073] P i+1 =Linear(U i+1 ) Formula (5)
[0074] Among them, P i+1 represents the feature vector corresponding to the feature map of the i+1th layer, Linear(·) represents the linear layer, and U i+1 Represents the i+1th layer feature map.
[0075] In this embodiment, the multiple feature vectors include feature vectors corresponding to feature maps of all layers.
[0076] In one embodiment of the present application, the process of calculating the similarity loss value based on multiple similarities includes:
[0077] Multiple similarities are input into a preset contrast loss function to obtain multiple contrast similarities.
[0078] In this embodiment, after obtaining multiple similarities, the multiple similarities are arranged and stored in the form of a matrix to form a similarity matrix. The expression of the preset contrast loss function is as follows:
[0079]
[0080] Among them, loss a,b Indicates the comparison similarity, S a,b represents the similarity between the a-th transformed feature vector and the b-th transformed feature vector, ρ represents the adjustable temperature parameter that affects the distribution of the contrast similarity, S a,f is the similarity of the ath row and fth column in the similarity matrix, and 2N is the number of sample images input in a batch.
[0081] In this embodiment, the preset contrastive loss function is a self-supervised loss function for learning image representation, which utilizes a contrastive learning framework to learn a set of useful features that can capture semantic information from unlabeled images and optimize a pre-established detection model.
[0082] In this embodiment, after a plurality of comparative similarities are obtained, all comparative similarities are arranged and stored in a matrix form to obtain a comparative similarity matrix.
[0083] A part of the comparative similarities is selected from the multiple comparative similarities, and an average value of the selected part of the comparative similarities is used as the similarity loss value.
[0084] In this embodiment, the calculation formula of the similarity loss value is as follows:
[0085]
[0086] Where L represents the similarity loss value, 2N is the number of sample images input in a batch, loss(2f-1, 2f) represents the comparative similarity of the 2f-1th row and 2fth column in the comparative similarity matrix, and loss(2f, 2f-1) represents the comparative similarity of the 2fth row and 2f-1th column in the comparative similarity matrix.
[0087] In this embodiment, the contrast similarity of the 2f-1th row and the 2fth column and the contrast similarity of the 2fth row and the 2f-1th column are partial contrast similarities selected from the contrast similarity matrix, the selected partial contrast similarities are greater than the preset similarity threshold and the selected partial contrast similarities are calculated based on the feature vectors of different sample images.
[0088] In one embodiment of the present application, if the classification module includes a concatenation layer, a first encoding layer, an attention mechanism layer, and a second encoding layer, the multi-layer feature map, the first preset position vector, the second preset position vector, and the preset classification vector are input into the classification module, and the process of obtaining the adjusted classification vector includes:
[0089] Get the key-value pairs of the target animal and convert them into a category number vector.
[0090] In this embodiment, the key-value pair is used to represent the mapping relationship between the category of the target animal and the number of the target animal. For example, the category of the target animal is golden monkey, and the number of the golden monkey is set to 49, then the key-value pair is the mapping relationship between golden monkey and 49.
[0091] The multi-layer feature map, the first preset position vector, the second preset position vector, the preset classification vector, and the category number vector are input into the splicing layer to obtain a splicing vector.
[0092] In this embodiment, multiple layers of feature maps, a first preset position vector, a second preset position vector, a preset classification vector, and a category number vector are spliced to obtain multi-level feature map information and multi-scale feature map information. In the process of training a pre-established detection model, the pre-established detection model can dynamically assign attention weights to different parts of the feature map, so that the established detection model pays more attention to the feature map information that has an important impact on the classification vector.
[0093] The concatenated vector is input into the first encoding layer to obtain an encoded concatenated vector.
[0094] In this embodiment, the first coding layer may be implemented by a coding function of an encoder, or may be implemented by other methods, which are specifically limited here.
[0095] The encoded concatenated vector is input into the attention mechanism layer to obtain the weighted vector.
[0096] In this embodiment, the attention mechanism layer can use the global attention mechanism to calculate the encoded splicing vector, filter out key feature vectors from the encoded splicing vector, and automatically adjust the attention to the key feature vectors and non-key feature vectors, so that the output weighted vector can more accurately reflect the key feature information.
[0097] In this embodiment, the calculation formula of the attention mechanism layer is as follows:
[0098]
[0099] Among them, Attention(Q, K, V) represents the weighted vector output by the attention mechanism layer, Q represents the query matrix, K represents the key matrix, V represents the value matrix, and K T represents the transpose of the key matrix, d k Indicates the dimension of the key.
[0100] In this embodiment, the expression of the query matrix is as follows:
[0101] Q=W q Y formula (9)
[0102] Where Q represents the query matrix, W q represents the weight matrix set in the query matrix, and Y represents the encoded concatenated vector.
[0103] In this embodiment, the expression of the key matrix is as follows:
[0104] K=W k Y formula (10)
[0105] Where K represents the key matrix, W krepresents the weight matrix set in the key matrix, and Y represents the encoded concatenated vector.
[0106] In this embodiment, the expression of the value matrix is as follows:
[0107] V=W v Y formula (11)
[0108] Among them, V represents the value matrix, W v represents the weight matrix set in the value matrix, and Y represents the encoded concatenated vector.
[0109] The weighted vector is input into the second coding layer to obtain the encoded weighted vector.
[0110] In this embodiment, the second coding layer may be implemented by a coding function of an encoder, or may be implemented by other methods, which are specifically limited here.
[0111] The classification vector is extracted from the encoded weighted vector to obtain an adjusted classification vector.
[0112] In this embodiment, the adjusted classification vector can more accurately reflect the key feature vector and reduce the impact of non-key feature vectors on the classification result.
[0113] In one embodiment of the present application, the multi-layer feature map, the first preset position vector, the second preset position vector, the preset classification vector, and the category number vector are input into the splicing layer to obtain the splicing vector, which includes:
[0114] The first preset position vector is embedded in the category number vector to obtain the category number embedding vector.
[0115] In this embodiment, the first preset position vector is used to mark the initial position of the category number vector. When there are multiple category number vectors, the number of the first preset position vectors is consistent with the number of the category number vectors, that is, one first preset position vector is embedded in each category number vector.
[0116] In this embodiment, there are multiple category number embedding vectors. Taking the nth category number embedding vector as an example, the expression of the nth category number embedding vector is as follows:
[0117] P′ n =P n +w n1 Formula (12)
[0118] Among them, P′ n represents the embedding vector of the nth category number, P n represents the nth category number vector, w n1 Represents the first preset position vector embedded in the nth category number vector.
[0119] In this embodiment, the first preset position vector embedded in the nth category number vector and the first preset position vector embedded in the n-1th category number vector have a preset mapping relationship. The preset mapping relationship is represented by a cosine function.
[0120] The second preset position vector is embedded into the multi-layer feature map to obtain a multi-layer feature embedding vector.
[0121] In this embodiment, the number of the second preset position vectors includes multiple ones, and a second preset position vector is embedded in each layer of feature map. Taking the feature embedding vector of the nth layer as an example, the expression of the feature embedding vector of the nth layer is as follows:
[0122] E′ n =E n +w n2 Formula (13)
[0123] Among them, E′ n represents the n-th layer feature embedding vector, E n represents the feature map of the nth layer, w n2 Represents the second preset position vector embedded in the nth feature map.
[0124] In this embodiment, a second preset position vector embedded in the nth layer feature map has a preset mapping relationship with a second preset position vector embedded in the n-1th layer feature map, and the preset mapping relationship is represented by a cosine function.
[0125] The preset classification vector, the category number embedding vector, and the multi-layer feature embedding vector are concatenated to obtain a concatenated vector.
[0126] In this embodiment, the method of splicing the preset classification vector, the category number embedding vector, and the multi-layer feature embedding vector can be horizontal splicing or vertical splicing, which needs to be set according to the actual situation.
[0127] In one embodiment of the present application, the process of calculating the cross entropy loss value based on the adjusted classification vector includes:
[0128] The prediction probability of the sample category to which the sample image belongs is calculated through the adjusted classification vector to obtain multiple prediction probabilities.
[0129] In this embodiment, the process of calculating the predicted probability of the sample category to which the sample image belongs through the adjusted classification vector can be implemented by referring to the process of calculating the predicted probability of the sample category to which the sample image belongs based on the classification vector in relevant technical materials, and no specific limitation is made here.
[0130] Calculate the cross entropy of multiple predicted probabilities and get the cross entropy loss value.
[0131] In this embodiment, the calculation formula of the cross entropy loss value is as follows:
[0132]
[0133] Among them, L CE represents the cross entropy loss value, l represents the product of the total number of sample images and the total number of sample categories, and y oc Indicates the indicator symbol that the oth sample image corresponds to sample category c. When the oth sample image corresponds to sample category c, y oc is 1, when the oth sample image does not correspond to sample category c, y oc is 0, p oc Represents the observed probability that the oth sample image belongs to sample category c.
[0134] Figure 3 is a flow chart of an animal detection method shown in another exemplary embodiment of the present application, such as Figure 3 As shown in the figure, the animal detection method includes: (1) the vehicle-mounted camera obtains a scene image containing rare animals at a long distance; (2) an enhanced image of each scene image is obtained by image enhancement methods such as rotation and color change; (3) a feature extraction module (including a large kernel convolution module and a neural network module) is used to extract features of small targets of rare animals at a long distance to obtain a multi-layer feature map; (4) the similarity between the multi-layer feature maps is calculated, a preset contrast loss function is designed, a plurality of contrast similarities are calculated, and a similarity loss value is calculated by the plurality of contrast similarities, and a pre-established detection model is adjusted by the similarity loss value. (5) The multi-layer feature map, the first preset position vector, the second preset position vector, the preset classification vector, and the category number vector are concatenated, and the correlation between the multi-layer feature map and the category number vector is calculated using the global attention mechanism layer to obtain the adjusted classification vector; (6) Based on the adjusted classification vector, the predicted probability of the sample category to which the sample image belongs is calculated, and the cross entropy of multiple predicted probabilities is calculated to obtain the cross entropy loss value, and the pre-established detection model is updated through the cross entropy loss value; (7) The long-distance rare animals are accurately located and classified.
[0135] In this embodiment, since the vehicle-mounted camera has a long collection distance for rare animals, the size of the animal in the scene image of the rare animal is smaller than the preset size threshold. Therefore, the animal in the scene image of the rare animal is regarded as a small target of a rare animal or a small sample of a rare animal.
[0136] Figure 4 is a schematic diagram showing a combination of a feature extraction module and a linear layer as shown in an exemplary embodiment of the present application. Figure 4 As shown, the feature extraction module includes a large kernel convolution module and a neural network module. The sample image is input into the large kernel convolution module to obtain a multi-scale feature map. The multi-scale feature map is input into the neural network module to obtain a multi-layer feature map. The multi-layer feature map is input into the linear layer to obtain multiple feature vectors.
[0137] In this embodiment, LKCM represents a large kernel convolution module, and Spatial Selection is used to selectively process the spatial information of the sample image.
[0138] Figure 5 FIG. 1 is a schematic diagram showing a similarity matrix composed of multiple similarities according to an exemplary embodiment of the present application. Figure 5 As shown, 4 sample images are input in the same batch, and the 4 sample images are sorted by rows and columns respectively. By calculating the similarity between the feature vectors of the sample images in the rows and the feature vectors in the columns, multiple similarities (S11, S12, S13, S14, S21, S22, S23, S24, S31, S32, S33, S34, S41, S42, S43, S44) are obtained. Taking S11, S12, and S13 as examples, S11 represents the similarity between the feature vectors of the sample images in the first row and the feature vectors of the sample images in the first column, and S S12 represents the similarity between the feature vector of the sample image located in the first row and the feature vector of the sample image located in the second column, and S13 represents the similarity between the feature vector of the sample image located in the first row and the feature vector of the sample image located in the third column, wherein S11 is calculated from the feature vector of the same sample image, S12 is calculated from the feature vector of the rotated sample image and the unrotated sample image, and S13 is calculated from the feature vectors of two different sample images. Therefore, S11 is greater than S12, S12 is greater than S13, and the similarity is represented by the color depth.
[0139] In this embodiment, when calculating the similarity loss value, the selected partial contrast similarity is the contrast similarity calculated by S12, the comparison similarity calculated by S21, the comparison similarity calculated by S34 and the comparison similarity calculated by S43, so that the selected partial contrast similarity is greater than the preset similarity threshold and the selected partial contrast similarity is calculated based on the feature vectors of different sample images.
[0140] In this embodiment, a contrastive learning framework is used to learn a set of useful features that can capture semantic information from unlabeled images and optimize a pre-established detection model. There is no need to train images of a single species of animals or pre-label sample images, saving manpower, material resources and computing resources.
[0141] Figure 6 is a schematic diagram of a classification module shown in an exemplary embodiment of the present application, such as Figure 6 As shown, the classification module includes a splicing layer, a first encoding layer, an attention mechanism layer and a second encoding layer, wherein the splicing layer is used to splice the multi-layer feature map, the first preset position vector, the second preset position vector, the preset classification vector, and the category number vector to obtain a splicing vector; the first encoding layer is used to encode the splicing vector to obtain an encoded splicing vector; the attention mechanism layer is used to calculate the encoded splicing vector to obtain a weighted vector; the second encoding layer is used to encode the weighted vector to obtain the encoded weighted vector, and obtain the adjusted classification vector by extracting the classification vector from the encoded weighted vector.
[0142] In this embodiment, Encoder Layer1 represents the first encoding layer, and Encoder Layer2 represents the second encoding layer. CLS (classification) represents a classification vector, and the position vector includes a first preset position vector and a second preset position vector.
[0143] This application first uses a large-kernel convolution module for vehicle-mounted applications to extract features of small targets of long-distance rare animals in sample images taken during vehicle driving, so as to achieve multi-scale analysis of small targets at long distances and reduce the demand for computing resources in the model training process. Then, based on multiple similarities and preset contrast loss functions, the similarity loss value is calculated, and the pre-established detection model is trained by the similarity loss value to improve the distribution and aggregation of features of rare animals of the same category under small sample conditions; finally, the global attention mechanism is used to calculate the correlation between the feature map and the category number vector, guiding the animal detection model to accurately locate and classify unlabeled long-distance road rare animal samples.
[0144] The following describes an embodiment of the device of the present application, which can be used to perform the animal detection method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the animal detection method in the above embodiment of the present application.
[0145] Figure 7 is a block diagram of an animal detection device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The implementation environment shown is specifically configured in the vehicle controller 102. The device may also be applicable to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applicable.
[0146] like Figure 7 As shown, the exemplary animal detection device includes:
[0147] The image acquisition module 701 is used to acquire a scene image of a target animal in front of the vehicle.
[0148] The image detection module 702 is used to input the scene image as the image to be detected into the animal detection model to obtain the detection result.
[0149] In one embodiment of the present application, the target animals include animals on the road. The animals on the road include but are not limited to rare animals, which are nationally protected animals.
[0150] In this embodiment, the animal detection model is obtained by training a pre-built detection model based on sample images of the target animal, and the sample images include images at different angles and under different lighting conditions.
[0151] In this embodiment, a pre-constructed detection model is trained through images of target animals at different angles and under different lighting conditions, so that the trained animal detection model can adapt to changes in shooting angles and lighting conditions of animal images in scenarios where vehicles are traveling at high speeds, thereby improving the detection accuracy of animal images in scenarios where vehicles are traveling at high speeds; in addition, there is no need to train on images of a single type of animal, nor is there any need to pre-label sample images, thereby saving manpower, material resources and computing resources.
[0152] It should be noted that the animal detection device provided in the above embodiment and the animal detection method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the animal detection device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.
[0153] An embodiment of the present application also provides a vehicle-mounted networked control device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the vehicle-mounted networked control device implements the animal detection method provided in the above-mentioned embodiments.
[0154] An embodiment of the present application further provides a vehicle, the vehicle comprising an animal detection device as provided in the above embodiments or a vehicle-mounted networking control device as provided in the above embodiments.
[0155] Figure 8 The schematic diagram of the structure of the computer system suitable for implementing the vehicle networking control device of the embodiment of the present application is shown. It should be noted that: Figure 8The computer system 800 of the vehicle-mounted networking control device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0156] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 to the random access memory (RAM) 803, such as executing the method in the above embodiment. In the RAM 803, various programs and data required for system operation are also stored. The CPU 801, ROM 802 and RAM 803 are connected to each other through a bus 804. The input / output (I / O) interface 805 is also connected to the bus 804.
[0157] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.
[0158] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 809, and / or installed from a removable medium 811. When the computer program is executed by a central processing unit (CPU) 801, various functions defined in the system of the present application are executed.
[0159] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable medium or any combination of the above two. Computer-readable media can be, for example, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0160] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0161] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0162] Another aspect of the present application also provides a computer readable medium having a computer program stored thereon, which, when executed by a processor of a computer, causes the computer to execute the animal detection method as described above. The computer readable medium may be included in the vehicle-mounted networking control device described in the above embodiment, or may exist independently without being assembled into the vehicle-mounted networking control device.
[0163] Another aspect of the present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable medium. A processor of a computer device reads the computer instructions from the computer-readable medium, and the processor executes the computer instructions, so that the computer device executes the animal detection method provided in each of the above embodiments.
[0164] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.
Claims
1. An animal detection method, characterized in that: The method comprises: Acquire a scene image of a target animal in front of the vehicle, wherein the target animal includes an animal on the road; The scene image is input as the image to be detected into the animal detection model to obtain a detection result; the animal detection model is obtained by training a pre-constructed detection model based on sample images of the target animal, and the sample images include images at different angles and under different lighting conditions.
2. The animal detection method according to claim 1, characterized in that: If the pre-built detection model includes a feature extraction module, a linear layer and a classification module, the process of training the pre-built detection model based on the sample image of the target animal includes: Input the sample image into the feature extraction module to obtain a multi-layer feature map, and input the multi-layer feature map into the linear layer to obtain a plurality of feature vectors; Calculating similarities between the plurality of feature vectors, and calculating similarity loss values based on the plurality of similarities; Inputting the multiple layers of the feature map, the first preset position vector, the second preset position vector, and the preset classification vector into the classification module to obtain an adjusted classification vector, and calculating a cross entropy loss value based on the adjusted classification vector; The sum of the similarity loss value and the cross entropy loss value is taken as a loss result, and the parameters of the feature extraction module, the parameters of the linear layer, and the parameters of the classification module are adjusted according to the loss result; The adjusted feature extraction module, the adjusted linear layer, and the adjusted classification module are combined to obtain the animal detection model.
3. The animal detection method according to claim 2, characterized in that: If the feature extraction module includes a large kernel convolution module and a neural network module, the sample image is input into the feature extraction module to obtain a multi-layer feature map, which includes: Inputting the sample image into the large-kernel convolution module to obtain a multi-scale feature map; The multi-scale feature map is input into the neural network module to obtain multiple layers of the feature map; the neural network module includes a convolution layer and a pooling layer, and the convolution kernel of the large-core convolution module is larger than the convolution kernel in the convolution layer.
4. The animal detection method according to claim 2, characterized in that: Based on the multiple similarities, the process of calculating the similarity loss value includes: Inputting the plurality of similarities into a preset comparison loss function to obtain a plurality of comparison similarities; A part of the comparative similarities is selected from the multiple comparative similarities, and an average value of the selected part of the comparative similarities is used as the similarity loss value.
5. The animal detection method according to claim 2, characterized in that: If the classification module includes a concatenation layer, a first encoding layer, an attention mechanism layer, and a second encoding layer, the multi-layer feature map, the first preset position vector, the second preset position vector, and the preset classification vector are input into the classification module to obtain an adjusted classification vector, which includes: Obtaining a key-value pair of the target animal, and converting the key-value pair into a category number vector; the key-value pair is used to represent the mapping relationship between the category of the target animal and the number of the target animal; Input the multiple layers of the feature map, the first preset position vector, the second preset position vector, the preset classification vector, and the category number vector into the splicing layer to obtain a splicing vector; Inputting the splicing vector into the first coding layer to obtain an encoded splicing vector; Inputting the encoded concatenated vector into the attention mechanism layer to obtain a weighted vector; Inputting the weighted vector into the second coding layer to obtain a coded weighted vector; A classification vector is extracted from the encoded weighted vector to obtain the adjusted classification vector.
6. The animal detection method according to claim 5, characterized in that: The process of inputting the multiple layers of the feature map, the first preset position vector, the second preset position vector, the preset classification vector, and the category number vector into the splicing layer to obtain the splicing vector includes: Embedding the first preset position vector into the category number vector to obtain a category number embedding vector; Embedding the second preset position vector into multiple layers of the feature map to obtain a multi-layer feature embedding vector; The preset classification vector, the category number embedding vector, and multiple layers of the feature embedding vector are concatenated to obtain the concatenated vector.
7. The animal detection method according to claim 2, characterized in that: Based on the adjusted classification vector, the process of calculating the cross entropy loss value includes: Calculating the prediction probability of the sample category to which the sample image belongs by using the adjusted classification vector to obtain multiple prediction probabilities; Calculate the cross entropy of the multiple predicted probabilities to obtain the cross entropy loss value.
8. An animal detection device, characterized in that: The device comprises: An image acquisition module, used to acquire a scene image of a target animal in front of the vehicle, wherein the target animal includes an animal on the road; The image detection module is used to input the scene image as the image to be detected into the animal detection model to obtain the detection result; the animal detection model is obtained by training a pre-constructed detection model based on the sample images of the target animal, and the sample images include images at different angles and under different lighting conditions.
9. A vehicle-mounted networking control device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the vehicle-mounted networking control device to implement the animal detection method as described in any one of claims 1 to 7.
10. A vehicle, characterized in that: The vehicle includes the animal detection device as claimed in claim 8 or the vehicle-mounted networking control device as claimed in claim 9.
