Image recognition method and device based on mid-high point video

Abstract

The embodiment of the invention discloses an image recognition method and device based on a mid-high point video, and belongs to the technical field of artificial intelligence and big data. Comprising the following steps: acquiring an image in a mid-high point video, and preprocessing the image to obtain a to-be-recognized image; extracting features of the to-be-recognized image by using the trained preliminary target detection model to obtain a feature map of the to-be-recognized image, and amplifying the feature map through a camera to obtain an amplified first image; and carrying out classification identification on the first image through a trained refined target detection model to obtain an identification result, and determining a target type of a predetermined target object of the to-be-identified image based on the identification result. According to the invention, the accuracy of small target recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and big data technology, and in particular to an image recognition method and device based on mid- and high-point videos. Background Art

[0002] Current deep learning-based object detection models, such as the You Only Look Once (YOLO) series of object detection algorithms, have been widely used in various scenarios, especially when the target is large and has obvious features, showing good detection and classification performance.

[0003] In the related art, when detecting tiny targets from a high altitude bird's-eye view, traditional target detection models often find it difficult to accurately identify the types of tiny targets due to limitations in viewing angle and resolution, resulting in low accuracy in identifying small targets. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image recognition method, device, electronic device and storage medium based on mid- and high-point videos, so as to solve the problem of low accuracy in identifying small targets.

[0005] To solve the above technical problems, the embodiments of the present application are implemented as follows: In a first aspect, an embodiment of the present application provides an image recognition method based on mid-high point videos, comprising: acquiring images in mid-high point videos, preprocessing the images to obtain images to be recognized; using a trained preliminary target detection model to extract features of the images to be recognized, obtaining a feature map of the images to be recognized, and amplifying the feature map through a camera to obtain an enlarged first image; classifying and recognizing the first image through a trained refined target detection model to obtain a recognition result, and determining the target type of a predetermined target object of the image to be recognized based on the recognition result.

[0006] In the second aspect, an embodiment of the present application provides an image recognition device based on mid-high point videos, including: an acquisition module, used to acquire images in mid-high point videos, pre-process the images, and obtain images to be recognized; a preliminary detection module, used to use a trained preliminary target detection model to extract features of the image to be recognized, obtain a feature map of the image to be recognized, and amplify the feature map through a camera to obtain an enlarged first image; a fine detection module, used to classify and recognize the first image through a trained fine target detection model to obtain a recognition result, and determine the target type of the predetermined target object of the image to be recognized based on the recognition result.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being used to call and execute the computer program from the memory to implement the above-mentioned image recognition method based on mid- and high-point videos.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program can be executed by a processor to implement the above-mentioned image recognition method based on mid- and high-position videos.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the above-mentioned image recognition method based on mid- and high-point videos.

[0010] By adopting the technical solution of the embodiment of the present application, the image in the mid-high point video is obtained, and the image is preprocessed to obtain the image to be identified; the features of the image to be identified are extracted by using the trained preliminary target detection model to obtain the feature map of the image to be identified, and the feature map is amplified by the camera to obtain the enlarged first image; the first image is classified and identified by the trained refined target detection model to obtain the recognition result, and the target type of the predetermined target object of the image to be identified is determined based on the recognition result. It can be seen that the features of the image to be identified corresponding to the image in the mid-high point video are extracted by the trained preliminary target detection model to obtain an accurate feature map, and the first image after the feature map is amplified is classified and identified by the trained refined target detection model to obtain the target type of the predetermined target object of the image to be identified. It can be used to perform secondary detection on the tiny images in the mid-high point video, thereby determining the position and type of the predetermined target object, improving the accuracy of tiny target detection under the high-altitude bird's-eye view, and solving the problem of low accuracy in small target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 It is a structural schematic diagram of an image recognition system based on mid- and high-point videos according to an embodiment of the present application; Figure 2is a schematic flow chart of an image recognition method based on mid- and high-position videos according to an embodiment of the present application; Figure 3 is a schematic flow chart of a scene of an image recognition method based on mid- and high-position videos according to an embodiment of the present application; Figure 4 is a schematic block diagram of an image recognition device based on mid- and high-position videos according to an embodiment of the present application; Figure 5 It is a schematic diagram of the hardware structure of an image recognition device based on mid- and high-point videos according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] The embodiments of the present application provide an image recognition method, device, electronic device and storage medium based on mid- and high-position videos, so as to solve the problem of low accuracy in identifying small targets.

[0014] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0015] The image recognition method based on mid- and high-position video provided in the embodiment of the present application can be executed by an electronic device, or by software installed in the electronic device. Specifically, the electronic device can be a terminal device or a server device. Among them, the terminal device can include a smart phone, a laptop computer, a smart wearable device, a vehicle terminal, etc., and the server device can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0016] In the following, in conjunction with the accompanying drawings, an image recognition method based on mid- and high-point videos provided by an embodiment of the present application is described in detail through specific embodiments and their application scenarios.

[0017] Figure 1 is a structural diagram of an image recognition system based on mid- and high-position videos according to an embodiment of the present application. Figure 1As shown, it includes: a preliminary target detection module, an image magnification module, a refined target detection module and a new evaluation index module. Specifically, the preliminary target detection module includes obtaining images in the mid- and high-point videos to obtain images to be processed, and inputting the preliminary target detection model, using the preliminary target detection model to extract features of the image to be processed to obtain a feature map; inputting the feature map into the image magnification module, and obtaining a first image by 3D magnifying the feature map; inputting the first image into the refined target detection model in the refined target detection module, classifying and identifying the first image, and obtaining a recognition result; using the evaluation function based on the bulldozer Wasserstein distance in the new evaluation index module, the distance between the predicted bounding box of the feature map and the real bounding box of the predetermined target object at the corresponding position in the image to be processed is evaluated for detection accuracy.

[0018] Figure 2 A schematic flow chart of an image recognition method based on mid- and high-position videos provided by an embodiment of the present invention is shown, and the method comprises the following steps: S202, obtaining images in the mid- and high-position video, and preprocessing the images to obtain images to be recognized.

[0019] The mid-high position refers to the height of the camera relative to the object being photographed during the shooting process. The object photographed from a high altitude is relatively small in the image of the mid-high position video, that is, the target detection object is small.

[0020] As an example, the video frame images acquired in real time by cameras at mid- and high-positions, where the cameras are installed on higher towers, provide high-definition pictures from a bird's-eye view. The images acquired from the high-definition pictures from a bird's-eye view are preprocessed to obtain images to be identified, and the images to be identified may include migratory birds.

[0021] S204, using the trained preliminary target detection model, extracting features of the image to be identified, obtaining a feature map of the image to be identified, and amplifying the feature map through a camera to obtain an enlarged first image.

[0022] The trained preliminary target detection models include: an improved target detection version 8 (You Only Look Once version 8, YOLOv8) model, which is a deep learning model for target detection.

[0023] The features of the image to be identified are extracted through the trained preliminary target detection model to obtain a feature map. Specifically, the features of the image to be identified are extracted through the improved YOLOv8 model to obtain a feature map. Based on the feature map, the preliminary recognition result, predicted bounding box and preliminary confidence score of the feature map type corresponding to the image to be identified can be output.

[0024] Using the camera's 3D magnification function to magnify the identified feature map in order to obtain more detailed information is crucial for identifying the subtle features of the image to be identified. Specifically, the system intelligently magnifies the feature map to ensure that the intended target object occupies more pixels in the first image after magnification, thereby improving the accuracy of subsequent classification.

[0025] Following the example in S202, by extracting the features of the image to be identified, a feature map of the image to be identified is obtained, and the feature map may be an image of the specific position of the migratory bird in the image to be identified. By enlarging the feature map, a first image is obtained, which facilitates the subsequent classification and identification of the migratory bird.

[0026] S206, classifying and identifying the first image through the trained refined target detection model to obtain a recognition result, and determining the target type of the predetermined target object of the image to be identified based on the recognition result.

[0027] The refined target detection model after training includes: the You Only Look Once version8n (YOLOv8n) model with high accuracy after training but unchanged network structure, which is a model in the YOLOv8 series and has the fastest processing speed.

[0028] Specifically, the first image is input into the trained refined target detection model, and the target object in the first image is classified and identified to obtain a recognition result, and the target type of the predetermined target object of the image to be identified is determined through the recognition result. The recognition result includes: the type of the first image after the first image is classified and identified. The type is used as the target type of the predetermined target object of the image to be identified.

[0029] The predetermined target object may be a small target detection object such as a migratory bird or a puppy in an image of a mid- to high-position video, without any specific limitation.

[0030] Continuing with the above example, the first image after the feature map including the specific location of the migratory birds is enlarged and input into the trained refined target detection model. The trained refined target detection model can distinguish the types of migratory birds. For example, migratory birds can be divided into summer migratory birds, migratory birds, geese, etc.

[0031] By adopting the technical solution of the embodiment of the present application, the image in the mid-high point video is obtained, and the image is preprocessed to obtain the image to be identified; the features of the image to be identified are extracted by using the trained preliminary target detection model to obtain the feature map of the image to be identified, and the feature map is amplified by the camera to obtain the enlarged first image; the first image is classified and identified by the trained refined target detection model to obtain the recognition result, and the target type of the predetermined target object of the image to be identified is determined based on the recognition result. It can be seen that the features of the image to be identified corresponding to the image in the mid-high point video are extracted by the trained preliminary target detection model to obtain an accurate feature map, and the first image after the feature map is amplified is classified and identified by the trained refined target detection model to obtain the target type of the predetermined target object of the image to be identified. It can be used to perform secondary detection on the tiny images in the mid-high point video, thereby determining the position and type of the predetermined target object, improving the accuracy of tiny target detection under the high-altitude bird's-eye view, and solving the problem of low accuracy in small target recognition.

[0032] In one embodiment, the trained preliminary target detection model is used to extract features of the image to be identified and obtain a feature map of the image to be identified (ie, S204), and the following steps A1-A4 may be performed: Step A1, using the attention mechanism in the trained preliminary target detection model to group the images to be identified, and obtain the sub-features corresponding to the images to be identified in each group.

[0033] Attention mechanisms include: Shuffle Attention (SA) for computer vision tasks, which can be used as SA modules. Randomness is introduced by grouping and random arrangement, and cross-attention calculations are performed on features at different scales, thereby improving the model's expressiveness and ability to handle complex scenes.

[0034] The Shuffle Attention mechanism is introduced into the preliminary target detection model after training to form the SA module. The SA module can group the images to be identified, and each group contains sub-features corresponding to the images to be identified.

[0035] Specifically, the SA attention mechanism is used to process the sub-features in each group in parallel through channel segmentation. For example, the image to be identified is segmented according to the corresponding number of channels to obtain multiple groups. The channels of each group represent a specific feature, and each group includes corresponding sub-features.

[0036] In step A2, a global average pooling process is performed on each sub-feature through the channel attention branch in the attention mechanism to obtain the channel weight corresponding to the sub-feature, and a channel attention feature map is obtained according to the channel weight.

[0037] The channel attention branch is used to focus on the correlation between different channels in the image to be processed. It means that in the SA attention mechanism, different channel weights are assigned to each channel to emphasize the channels that contribute most to the task and suppress irrelevant or redundant channels, thereby improving the performance of the preliminary target detection model.

[0038] Global average pooling is a pooling operation that aggregates the information of the entire sub-feature map.

[0039] Specifically, the sub-feature map is compressed through global average pooling to obtain the average pooled feature value of each channel, and the channel-level statistical information, namely the global feature vector, is generated. The global feature vector is transformed nonlinearly through several fully connected layers, usually including a ReLU activation function and a siqmoid activation function. The output value is compressed between 0 and 1, and a weight vector value, namely the channel weight, with the same length as the number of input channels is generated. The channel weight is multiplied by the original sub-feature map to obtain the channel attention feature map.

[0040] In step A3, each sub-feature is group-normalized through the spatial attention branch in the attention mechanism to obtain a spatial attention feature map.

[0041] The spatial attention branch emphasizes or suppresses information in a specific area by assigning different weights to the position of each sub-feature. Generally, a spatial transformation neural network is used, which can transform various deformation data in space and automatically capture important regional features.

[0042] Group normalization refers to adjusting the mean of the input data to 0 and the variance to 1, which can reduce problems such as gradient disappearance during training and thus improve the stability of training.

[0043] Specifically, the spatial attention mechanism automatically captures important regional features by performing spatial transformation on each sub-feature, thereby helping the model to better understand and process each sub-feature data. It can ensure that the image to be processed can still obtain the same result as the original image before the operation after operations such as cropping, translation or rotation. First, global average pooling and maximum pooling are performed in the channel dimension to splice the generated sub-feature maps. On the spliced ​​sub-feature maps, the convolution layer is used to perform dimensionality reduction and normalization processing to obtain the spatial weight. The spatial weight is multiplied by the original sub-feature map to obtain the spatial attention feature map.

[0044] Step A4, concatenating the channel attention feature map and the spatial attention feature map to obtain a feature map of the image to be recognized.

[0045] The channel attention feature map and the spatial attention feature map are spliced ​​together. Specifically, feature splicing is a commonly used feature fusion method in deep learning, which combines feature information from different branches or different levels by splicing different feature maps together according to a certain dimension (usually the channel dimension).

[0046] By splicing the channel attention feature map and the spatial attention feature map, the feature map of the image to be identified is obtained. The position of the target object in the image to be identified in the mid- and high-level video is determined based on the feature map and the attention mechanism, and the preliminary recognition result, predicted bounding box and preliminary confidence score corresponding to the predetermined target object in the image to be identified can be preliminarily identified.

[0047] Continuing with the above example, the image to be identified can be an image including migratory birds. By processing the image to be identified, a feature map including a migratory bird picture is obtained. Based on the feature map, the trained preliminary target detection model can output preliminary recognition results, predicted bounding boxes and a lower preliminary confidence score for the migratory bird species based on the feature map of the migratory bird picture. However, the output feature map is sufficient to support the next step of image enlargement processing.

[0048] In this embodiment, the channel attention branch and the spatial attention branch in the attention mechanism in the trained preliminary target detection model are used to determine the channel attention feature map and the spatial attention feature map respectively, and the channel attention feature map and the spatial attention feature map are spliced ​​to obtain the feature map of the image to be identified, which can aggregate all sub-features and use the "channel shuffle" operation to achieve information exchange between different sub-features. Based on the obtained feature map, the type of target object in the image to be processed can be preliminarily identified, especially in small targets and complex backgrounds, showing higher accuracy and robustness.

[0049] In one embodiment, after extracting the features of the image to be identified using the trained preliminary target detection model and obtaining the feature map of the image to be identified (ie, S204), the following steps B1-B3 may be performed: Step B1, obtaining the predicted bounding box of the feature map and the real bounding box of the image to be identified.

[0050] Based on the trained preliminary target detection model, i.e., the improved YOLOv8 model, a feature map is obtained according to step A4, and a predicted bounding box of the feature map is determined, where the predicted bounding box is the bounding box corresponding to the obtained feature map. The true bounding box of the image to be identified is the bounding box corresponding to the predetermined target object in the image to be identified.

[0051] Step B2, based on the predicted bounding box and the true bounding box, using the evaluation function, calculates the distance between the predicted bounding box and the true bounding box.

[0052] The evaluation functions include an exact evaluation function based on the Wasserstein distance, which is used to indicate how similar two distributions are.

[0053] Specifically, after obtaining the feature map, in order to further improve the accuracy of the improved YOLOv8 model in detecting small targets, the Wasserstein distance is introduced as a new evaluation indicator. Compared with the traditional intersection over union (IoU) indicator, the Wasserstein distance is more suitable for evaluating the similarity of bounding boxes of small targets, reducing the detection error caused by the small target. Based on the calculation of the evaluation function Wasserstein, the distance between the predicted bounding box and the true bounding box is obtained.

[0054] In step B3, the predicted bounding box of the feature map is verified based on the distance.

[0055] Based on the obtained distance, the predicted bounding box of the feature map obtained by the trained preliminary target detection model is preliminarily verified to reduce the impact of position deviation on detection accuracy. In addition, based on the feedback of Wasserstein distance, the model can be further optimized to improve the overall detection accuracy and robustness, ensuring efficient recognition in complex environments.

[0056] In this embodiment, after obtaining the feature map of the image to be processed through the trained preliminary target detection model, the predicted bounding box and the true bounding box of the feature map are verified, and the distance between the predicted bounding box and the true bounding box is calculated based on the evaluation function. Based on the distance, accuracy feedback is obtained to reduce the impact of position deviation on detection accuracy and ensure efficient identification of the position of the feature map in complex environments.

[0057] In one embodiment, based on the predicted bounding box and the true bounding box, the distance between the predicted bounding box and the true bounding box is calculated using an evaluation function (ie, step B2), and the following steps C1-C2 may also be performed: Step C1, according to the predicted bounding box and the true bounding box, using a two-dimensional Gaussian distribution method, determines the predicted distribution data of the predicted bounding box and the true distribution data of the true bounding box.

[0058] Based on the improved YOLOv8 model, the predicted bounding box and the true bounding box of the feature map are obtained.

[0059] The two-dimensional Gaussian distribution is used to describe the joint probability distribution of two random variables.

[0060] As an example, when the ground-truth bounding box is , predict bounding box , model it as a two-dimensional Gaussian distribution, and obtain the real distribution data N( , ) and the predicted distribution data N( , ).

[0061] in, is the center point of the bounding box, is the covariance matrix, which represents the uncertainty of the bounding box, represents the center point of the ground-truth bounding box, represents the center point of the predicted bounding box, and is the covariance matrix of the true bounding box and the predicted bounding box.

[0062] Step C2, based on the predicted distribution data and the true distribution data, calculate the distance between the predicted bounding box and the true bounding box.

[0063] Specifically, the distance between the predicted bounding box and the true bounding box is the Wasserstein distance, It means that it can be defined as: + (

[0064] in: is the Euclidean distance between the center points of the two bounding boxes; and is the covariance matrix of the true bounding box and the predicted bounding box; Represents the trace of the matrix.

[0065] In this embodiment, the Wasserstein distance is calculated according to the two-dimensional Gaussian distribution method, and the distance between the predicted bounding box and the true bounding box is finally determined. According to the distance, the similarity between the feature map and the bounding box of the predetermined target object corresponding to the image to be processed is determined, which can be more suitable for calculating the similarity of the bounding boxes of small targets and improve the accuracy of the position calculation of the feature map.

[0066] In one embodiment, the training of the refined target detection model may be performed by performing the following steps D1-D4: Step D1, obtaining the categories to which multiple sample predetermined target objects belong, and based on the categories, obtaining multiple sample second images, wherein the sample second images include sample images corresponding to the categories.

[0067] Specifically, as an example, the refined target detection model includes the YOLOv8n model, the sample predetermined target object may be a type of migratory bird such as a wild goose, the category may be a migratory bird, the sample second image may be a sample image that is enlarged from multiple images of different types of migratory birds, and the multiple sample second images may be an enlarged data set of images of 200 types of migratory birds, which includes the categories of migratory birds and predicted bounding boxes.

[0068] Step D2: input multiple sample second images into the refined object detection model to be trained.

[0069] Step D3, obtaining sample recognition results of the multiple sample second images by performing block processing on the multiple sample second images.

[0070] The refined target detection model can complete the target image target detection, instance segmentation and image classification tasks, and the YOLOv8n model is used to process multiple sample second images in blocks to identify the sample recognition results of each sample second image. The sample recognition result is the sample image category to which the sample second image belongs after the sample second image is recognized based on the YOLOv8n model.

[0071] Step D4, based on the category of the target object predetermined by the sample and the sample recognition result, a refined target detection model is trained to obtain a trained refined target detection model.

[0072] According to the category of the target object of the sample and the sample recognition result, the parameter information of the refined target detection model is adjusted to make the target type of the category of the target object of the output sample consistent with the type recognized by the output sample recognition result. Among them, the YOLOv8n model has efficient detection capabilities and low computational overhead, and is more suitable for real-time detection tasks.

[0073] Following the example in step D1, the category of the sample predetermined target object is migratory birds. Based on the category, the enlarged sample second images of 200 types of migratory bird images are obtained as a data set, which are input into the YOLOv8n model to be trained. The YOLOv8n model without changing the network structure is used to process all the sample second images in the data set in blocks, and the sample target species of the sample second images are identified, and the recognition result is output. The recognition result is the species corresponding to the sample second image. By adjusting the parameters in the YOLOv8n model, the species identified in the output sample recognition result is consistent with the sample target species of the sample predetermined target object. For example, if the type of the input sample predetermined target object is a wild goose among migratory birds, then the recognition result is that the type of the sample predetermined target object is a wild goose among migratory birds.

[0074] There are no specific limitations on the intended target objects of the samples and their corresponding categories.

[0075] In this embodiment, by acquiring multiple sample second images of the same category as the sample predetermined target object, the refined target detection model is trained, and the trained refined target detection model can identify the specific type of the sample predetermined target object in the category to which it belongs. Among them, the trained YOLOv8n model can output the final recognition result and confidence score of the predetermined target object.

[0076] In one embodiment, the training of the preliminary object detection model may be performed by performing the following steps E1-E5: Step E1, obtaining a plurality of sample images to be identified and corresponding sample predetermined target objects, and inputting the sample images to be identified into a preliminary target detection model to be trained.

[0077] The sample image to be identified is an image obtained by preprocessing the image in the mid- and high-point video.

[0078] The sample predetermined target object is a target image to be identified in the sample to-be-identified image.

[0079] The preliminary target detection model to be trained is the YOLOv8 model, which can be improved.

[0080] Step E2, grouping the sample images to be identified through the attention mechanism in the preliminary target detection model to be trained to obtain sample sub-features corresponding to the sample images to be identified in each group.

[0081] By introducing the SA attention mechanism into the YOLOv8 model to be trained, the sample images to be identified are grouped and processed to obtain the sample sub-features corresponding to the sample images to be identified in each group.

[0082] Step E3, through the channel attention branch and the spatial attention branch, the sample sub-features are calculated and processed to obtain a sample feature map.

[0083] Through the channel attention branch in the SA attention mechanism, the sample sub-features are globally averaged pooled to obtain the channel weights corresponding to the sub-features. According to the channel weights, the sample channel attention feature map is obtained; through the spatial attention branch in the SA attention mechanism, each sample sub-feature is group normalized to obtain the sample spatial attention feature map.

[0084] The sample channel attention feature map and the sample space attention feature map are concatenated to obtain the sample feature map.

[0085] Step E4: verify the predicted bounding box of the sample feature map according to the evaluation function.

[0086] Based on the obtained sample feature map, the YOLOv8 model to be trained outputs the sample feature map, the corresponding sample prediction bounding box and the sample preliminary confidence score.

[0087] Based on the evaluation function, namely the Wasserstein distance method, the sample prediction bounding box of the obtained sample feature map is verified. Specifically, the sample Wasserstein distance between the sample prediction bounding box and the sample true bounding box is calculated by the two-dimensional Gaussian distribution method. According to the sample Wasserstein distance, the similarity between the sample prediction bounding box and the sample true bounding box is verified. If the similarity is very low, it means that the position of the sample feature map obtained may be inaccurate.

[0088] Step E5, based on the sample feature map, the sample predetermined target object and the loss function of the preliminary target detection, adjust the parameters in the preliminary target detection model to be trained to obtain the trained preliminary target detection model.

[0089] Among them, the loss function includes a distance loss function, and the distance loss function includes the distance between the predicted bounding box and the true bounding box.

[0090] Specifically, the loss function in the preliminary target detection model to be trained previously includes classification loss, positioning loss and confidence loss, and the positioning loss is replaced by a distance loss function, that is, the positioning loss function is replaced by a function of Wasserstein distance.

[0091] The loss function in the preliminary target detection model to be trained is specifically expressed as follows: L=

[0092] in, is the classification loss, which is used to measure the difference between the predicted category based on the feature map prediction and the true category of the intended target object (usually using the cross entropy loss); is the confidence loss, which is used to measure the difference between the initial confidence score of the predicted box containing the predetermined target object and the actual situation; W2(Bt,Bp) is the distance loss based on the Wasserstein distance. , , They are weight coefficients, which are used to balance different parts in the loss function.

[0093] The sample feature map output by the preliminary target detection model to be trained is compared with the obtained sample predetermined target object, and the parameters in the preliminary target detection model to be trained, such as the loss function of the preliminary target detection, are adjusted, so that the predicted bounding box of the sample feature map output by the preliminary target detection model to be trained is close to the real bounding box of the sample predetermined target object, and the predicted category corresponding to the sample feature map is close to the real category, and the preliminary confidence of the sample feature map and the sample predetermined target object is closer.

[0094] In this embodiment, by introducing the SA attention mechanism and the evaluation function based on the Wasserstein distance method into the preliminary target detection model to be trained, and adding the distance loss function, the preliminary target detection model to be trained is trained to obtain the trained preliminary target detection model. The trained preliminary target detection model can more accurately output the sample feature map, the predicted bounding box of the sample feature map, the preliminary recognition result and the preliminary confidence score.

[0095] In one embodiment, an image in a mid-high position video is obtained, and the image is preprocessed to obtain an image to be identified (ie, S202), and the following steps F1-F2 may also be performed: Step F1, obtaining an image in a mid- and high-position video, and performing detail enhancement and noise interference reduction processing on the image to obtain a processed image.

[0096] Image enhancement includes: enhancing image details by adjusting brightness and contrast.

[0097] Reducing noise interference, also known as noise reduction processing, includes: using filters (such as Gaussian filtering) to reduce noise interference in the image to improve the visual effect of the image and make it clearer and cleaner.

[0098] Step F2, normalize the scale of the processed image to obtain an image to be recognized that meets the input size of the trained preliminary target detection model.

[0099] Scale normalization includes: obtaining an image to be processed by adjusting the size of the processed image to adapt to the input size requirement of the YOLOv8 model improved by the trained preliminary target detection model set.

[0100] In this embodiment, an image is obtained from a mid-high point video, and details are enhanced and noise interference is reduced to obtain a processed image, and the processed image is scale-normalized to obtain an image to be identified that meets the input size of the trained preliminary target detection model. Images in mid-high point videos often have complex backgrounds, and clear images to be processed can be obtained through preprocessing.

[0101] Figure 3is a schematic flow chart of a scene of an image recognition method based on mid- and high-position videos according to another embodiment of the present application, such as Figure 3 As shown, the method comprises the following steps: S301, acquiring images in the mid- and high-position video, and performing detail enhancement and noise interference reduction processing on the images to obtain processed images.

[0102] S302, performing scale normalization processing on the processed image to obtain an image to be recognized that meets the input size of the trained preliminary target detection model.

[0103] S303, inputting the image to be identified into the trained preliminary target detection model, using the SA attention mechanism in the trained preliminary target detection model to group the image to be identified, and obtaining the sub-features corresponding to the image to be identified in each group.

[0104] S304, obtaining a channel attention feature map through the channel attention branch in the attention mechanism; and obtaining a spatial attention feature map based on the spatial attention branch in the attention mechanism.

[0105] S305, concatenating the channel attention feature map and the spatial attention feature map to obtain a feature map of the image to be recognized.

[0106] S306, enlarging the feature image through a camera to obtain an enlarged first image.

[0107] S307, inputting the first image into the trained refined object detection model, and classifying and identifying the first image based on the trained refined object detection model to obtain a recognition result.

[0108] S308: Determine the target type of the predetermined target object of the image to be identified based on the recognition result.

[0109] The specific process from S301 to S308 has been described in detail in the above embodiment and will not be repeated here.

[0110] By adopting the technical solution of the embodiment of the present application, the image in the mid-high point video is obtained, and the image is preprocessed to obtain the image to be identified; the features of the image to be identified are extracted by using the trained preliminary target detection model to obtain the feature map of the image to be identified, and the feature map is amplified by the camera to obtain the enlarged first image; the first image is classified and identified by the trained refined target detection model to obtain the recognition result, and the target type of the predetermined target object of the image to be identified is determined based on the recognition result. It can be seen that the features of the image to be identified corresponding to the image in the mid-high point video are extracted by the trained preliminary target detection model to obtain an accurate feature map, and the first image after the feature map is amplified is classified and identified by the trained refined target detection model to obtain the target type of the predetermined target object of the image to be identified. It can be used to perform secondary detection on the tiny images in the mid-high point video, thereby determining the position and type of the predetermined target object, improving the accuracy of tiny target detection under the high-altitude bird's-eye view, and solving the problem of low accuracy in small target recognition.

[0111] In summary, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0112] The above is an image recognition method based on mid-high point video provided by an embodiment of the present application. Based on the same idea, an image recognition device based on mid-high point video is also provided by an embodiment of the present application.

[0113] Figure 4 is a schematic diagram of the structure of an image recognition device based on mid- and high-position video according to an embodiment of the present invention. Figure 4 As shown, the image recognition device based on the mid-high point video includes: an acquisition module 41, a preliminary detection module 42, and a fine detection module 43: The acquisition module 41 is used to acquire images in the mid- and high-position videos, and pre-process the images to obtain images to be recognized; A preliminary detection module 42 is used to extract features of the image to be identified using the trained preliminary target detection model to obtain a feature map of the image to be identified, and to amplify the feature map through a camera to obtain an amplified first image; The fine detection module 43 is used to classify and identify the first image through the trained fine target detection model to obtain a recognition result, and determine the target type of the predetermined target object of the image to be identified based on the recognition result.

[0114] In one embodiment, the preliminary detection module 42 specifically includes: A grouping unit is used to group the images to be identified by using the attention mechanism in the trained preliminary target detection model to obtain sub-features corresponding to the images to be identified in each group; The channel unit is used to perform global average pooling on each sub-feature through the channel attention branch in the attention mechanism to obtain the channel weight corresponding to the sub-feature, and obtain the channel attention feature map according to the channel weight; The spatial unit is used to perform group normalization on each sub-feature through the spatial attention branch in the attention mechanism to obtain a spatial attention feature map; The splicing unit is used to splice the channel attention feature map and the spatial attention feature map to obtain a feature map of the image to be recognized.

[0115] In one embodiment, the device also includes a new evaluation index module, which is specifically used to obtain a predicted bounding box of a feature map and a true bounding box of an image to be identified; based on the predicted bounding box and the true bounding box, an evaluation function is used to calculate the distance between the predicted bounding box and the true bounding box; based on the distance, the predicted bounding box of the feature map is verified.

[0116] In one embodiment, the new evaluation index module is further specifically used to determine the predicted distribution data of the predicted bounding box and the real distribution data of the real bounding box based on the predicted bounding box and the real bounding box using a two-dimensional Gaussian distribution method; based on the predicted distribution data and the real distribution data, calculate the distance between the predicted bounding box and the real bounding box.

[0117] In one embodiment, the device further includes a training module for refining the target detection model, specifically including: A first acquisition unit is used to acquire the categories to which the plurality of sample predetermined target objects belong, and acquire a plurality of sample second images based on the categories to which they belong; the sample second images include sample images corresponding to the categories to which they belong; An input unit, used to input a plurality of sample second images into a refined object detection model to be trained; A first processing unit, configured to obtain sample recognition results of the plurality of sample second images by performing block processing on the plurality of sample second images; The first training unit is used to train a refined target detection model based on the category of the target object predetermined by the sample and the sample recognition result to obtain a trained refined target detection model.

[0118] In one embodiment, the device further includes a training module for a preliminary target detection model, specifically including: A second acquisition unit is used to acquire a plurality of sample images to be identified and corresponding sample predetermined target objects, and input the sample images to be identified into a preliminary target detection model to be trained; The second processing unit is used to group the sample images to be identified by using the attention mechanism in the preliminary target detection model to be trained, and obtain sample sub-features corresponding to the sample images to be identified in each group; A computing unit, used to compute and process the sample sub-features through a channel attention branch and a spatial attention branch to obtain a sample feature map; An evaluation unit, used to verify the predicted bounding box of the sample feature map according to the evaluation function; The second training unit is used to adjust the parameters in the preliminary target detection model to be trained based on the sample feature map, the sample predetermined target object and the loss function of the preliminary target detection to obtain the trained preliminary target detection model, wherein the loss function includes a distance loss function, and the distance loss function includes the distance between the predicted bounding box and the real bounding box.

[0119] In one embodiment, the acquisition module 41 is specifically used to acquire images in the mid- and high-position videos, and perform detail enhancement and noise interference reduction processing on the images to obtain processed images; the processed images are scale-normalized to obtain images to be identified that meet the input size of the trained preliminary target detection model.

[0120] By adopting the technical solution of the embodiment of the present application, the image in the mid-high point video is obtained, and the image is preprocessed to obtain the image to be identified; the features of the image to be identified are extracted by using the trained preliminary target detection model to obtain the feature map of the image to be identified, and the feature map is amplified by the camera to obtain the enlarged first image; the first image is classified and identified by the trained refined target detection model to obtain the recognition result, and the target type of the predetermined target object of the image to be identified is determined based on the recognition result. It can be seen that the features of the image to be identified corresponding to the image in the mid-high point video are extracted by the trained preliminary target detection model to obtain an accurate feature map, and the first image after the feature map is amplified is classified and identified by the trained refined target detection model to obtain the target type of the predetermined target object of the image to be identified. It can be used to perform secondary detection on the tiny images in the mid-high point video, thereby determining the position and type of the predetermined target object, improving the accuracy of tiny target detection under the high-altitude bird's-eye view, and solving the problem of low accuracy in small target recognition.

[0121] Those skilled in the art should understand that Figure 4The image recognition device based on mid-high point videos can be used to implement the image recognition method based on mid-high point videos described above, and the detailed description thereof should be similar to the description of the method part above. To avoid tediousness, it will not be repeated here.

[0122] Based on the same technical concept, an embodiment of the present application further provides an electronic device, which is used to execute the above-mentioned image recognition method based on mid- and high-position videos. Figure 5 A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call a computer program stored in the memory 530 and executable on the processor 510 to perform the following steps: Obtain images from the mid- and high-level video, pre-process the images, and obtain images to be identified; Using the trained preliminary target detection model, extract the features of the image to be identified to obtain a feature map of the image to be identified, and amplify the feature map through a camera to obtain an enlarged first image; The first image is classified and recognized by the trained refined target detection model to obtain a recognition result, and the target type of the predetermined target object of the image to be recognized is determined based on the recognition result.

[0123] By adopting the technical solution of the embodiment of the present application, the image in the mid-high point video is obtained, and the image is preprocessed to obtain the image to be identified; the features of the image to be identified are extracted by using the trained preliminary target detection model to obtain the feature map of the image to be identified, and the feature map is amplified by the camera to obtain the enlarged first image; the first image is classified and identified by the trained refined target detection model to obtain the recognition result, and the target type of the predetermined target object of the image to be identified is determined based on the recognition result. It can be seen that the features of the image to be identified corresponding to the image in the mid-high point video are extracted by the trained preliminary target detection model to obtain an accurate feature map, and the first image after the feature map is amplified is classified and identified by the trained refined target detection model to obtain the target type of the predetermined target object of the image to be identified. It can be used to perform secondary detection on the tiny images in the mid-high point video, thereby determining the position and type of the predetermined target object, improving the accuracy of tiny target detection under the high-altitude bird's-eye view, and solving the problem of low accuracy in small target recognition.

[0124] The specific execution steps can refer to the various steps of the above-mentioned embodiment of the image recognition method based on mid- and high-point videos, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0125] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices except terminals.

[0126] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0127] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0128] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0129] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image recognition method embodiment based on mid- and high-point videos is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0130] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0131] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image recognition method embodiment based on mid- and high-point videos, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0132] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0133] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the processor is used to run the program or instructions to implement the various processes of the above-mentioned product recommendation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0134] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0135] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0136] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image recognition method based on mid- and high-level video, characterized in that: The method comprises: Acquire images in the mid- and high-position video, and preprocess the images to obtain images to be identified; Using the trained preliminary target detection model, extracting features of the image to be identified, obtaining a feature map of the image to be identified, and amplifying the feature map through a camera to obtain an enlarged first image; The first image is classified and recognized by the trained refined target detection model to obtain a recognition result, and the target type of the predetermined target object of the image to be recognized is determined based on the recognition result.

2. The method according to claim 1, characterized in that The method of extracting features of the image to be identified by using the trained preliminary target detection model to obtain a feature map of the image to be identified includes: The images to be identified are grouped using the attention mechanism in the trained preliminary target detection model to obtain sub-features corresponding to the images to be identified in each group; Through the channel attention branch in the attention mechanism, a global average pooling process is performed on each of the sub-features to obtain a channel weight corresponding to the sub-feature, and a channel attention feature map is obtained according to the channel weight; By using the spatial attention branch in the attention mechanism, each of the sub-features is group normalized to obtain a spatial attention feature map; The channel attention feature map and the spatial attention feature map are spliced ​​to obtain the feature map of the image to be identified.

3. The method according to claim 2, characterized in that After extracting the features of the image to be identified by using the trained preliminary target detection model and obtaining the feature map of the image to be identified, the method includes: Obtaining a predicted bounding box of the feature map and a real bounding box of the image to be identified; According to the predicted bounding box and the true bounding box, using an evaluation function, calculating the distance between the predicted bounding box and the true bounding box; Based on the distance, the predicted bounding box of the feature map is verified.

4. The method according to claim 3, characterized in that: The calculating the distance between the predicted bounding box and the true bounding box by using an evaluation function according to the predicted bounding box and the true bounding box includes: According to the predicted bounding box and the true bounding box, using a two-dimensional Gaussian distribution method, determining predicted distribution data of the predicted bounding box and true distribution data of the true bounding box; The distance between the predicted bounding box and the true bounding box is calculated based on the predicted distribution data and the true distribution data.

5. The method according to claim 1, characterized in that The training of the refined target detection model includes: Acquire the categories to which a plurality of sample predetermined target objects belong, and based on the categories, acquire a plurality of sample second images, wherein the sample second images include sample images corresponding to the categories; Inputting a plurality of the sample second images into the refined object detection model to be trained; Obtaining sample recognition results of the plurality of sample second images by performing block processing on the plurality of sample second images; Based on the category to which the predetermined target object of the sample belongs and the sample recognition result, the refined target detection model is trained to obtain the trained refined target detection model.

6. The method according to claim 3, characterized in that The training of the preliminary target detection model includes: Acquire a plurality of sample images to be identified and corresponding sample predetermined target objects, and input the sample images to be identified into the preliminary target detection model to be trained; The sample images to be identified are grouped and processed by the attention mechanism in the preliminary target detection model to be trained to obtain sample sub-features corresponding to the sample images to be identified in each group; The sample sub-features are calculated and processed by the channel attention branch and the spatial attention branch to obtain a sample feature map; Verifying the predicted bounding box of the sample feature map according to the evaluation function; Based on the sample feature map, the sample predetermined target object and the loss function of the preliminary target detection, the parameters in the preliminary target detection model to be trained are adjusted to obtain the trained preliminary target detection model, wherein the loss function includes a distance loss function, and the distance loss function includes the distance between the predicted bounding box and the true bounding box.

7. The method according to claim 1, characterized in that The step of acquiring images in the mid- and high-position video and preprocessing the images to obtain images to be identified includes: Acquire the image in the mid-high point video, and perform detail enhancement and noise interference reduction processing on the image to obtain a processed image; The processed image is scale-normalized to obtain the image to be identified that meets the input size of the trained preliminary target detection model.

8. An image recognition device based on mid- and high-level video, characterized in that: The device comprises: An acquisition module is used to acquire images in the mid- and high-position videos, and pre-process the images to obtain images to be identified; A preliminary detection module, used to extract features of the image to be identified by using the trained preliminary target detection model, obtain a feature map of the image to be identified, and amplify the feature map through a camera to obtain an enlarged first image; The refined detection module is used to classify and identify the first image through a trained refined target detection model to obtain a recognition result, and determine the target type of the predetermined target object of the image to be identified based on the recognition result.

9. An electronic device, characterized in that: It includes a processor and a memory electrically connected to the processor, the memory stores a computer program, and the processor is used to call and execute the computer program from the memory to implement an image recognition method based on mid- and high-point videos as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium is used to store a computer program, and the computer program can be executed by a processor to implement an image recognition method based on mid- and high-position videos as described in any one of claims 1 to 7.

Images