Image-based multi-target detection method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a kind of based on image multi-target detection method and device, electronic equipment, storage medium, belong to financial science and technology technical field.The method comprises: the initial image is carried out frequency domain transformation and obtains image frequency domain feature;Based on image frequency domain feature, initial image is enhanced and obtains enhanced image;Based on multi-scale feature fusion submodel, enhanced image is handled and obtains multi-scale fusion feature map;Based on key point detection submodel, multi-scale fusion feature map is detected and obtains key point detection data;Based on key point position data, candidate key point is calculated and obtains key point distance similarity;Based on predicted key point category and key point distance similarity, candidate key point is calculated and obtains key point semantic similarity;Based on key point grouping submodel, key point semantic similarity, key point distance similarity and candidate key point are grouped and obtain target key point group.The accuracy of the multi-target detection based on image can be improved by the embodiments of the present application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of financial technology, and in particular to a multi-target detection method and device based on images, an electronic device and a storage medium. BACKGROUND

[0002] Multi-target detection based on images is a method of automatically identifying multiple targets from images. In the insurance claim settlement scenario of financial technology, the multi-target detection method can be used to identify the situation of multiple targets such as vehicles, pedestrians, roads and obstacles in the video taken by the vehicle to be claimed, so as to determine a specific claim settlement scheme. Due to problems such as occlusion, background interference and illumination, the image may have overexposed or dark areas, and the target in the image may be occluded by other targets, resulting in target mis-detection or missed detection.

[0003] Therefore, the related art usually only considers target detection in a single target scenario, and does not deeply understand the association between other target information in the image and the association information of the target to be identified when performing multi-target detection, so as to be easily affected by problems such as occlusion, background interference and illumination, thereby reducing the detection accuracy of multiple targets in the image. Therefore, how to improve the accuracy of multi-target detection based on images has become a technical problem to be solved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a multi-target detection method and device based on images, an electronic device and a storage medium, which can improve the accuracy of multi-target detection based on images.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application provides a multi-target detection method based on images, which comprises:

[0006] Obtaining an initial image, the initial image comprising multiple detection targets;

[0007] Performing frequency domain transformation on the initial image to obtain image frequency domain features;

[0008] Performing image enhancement on the initial image based on the image frequency domain features to obtain an enhanced image;

[0009] Inputting the enhanced sample image into a pre-trained multi-target detection model, the multi-target detection model comprising a multi-scale feature fusion sub-model, a key point detection sub-model and a key point grouping sub-model;

[0010] Performing multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map;

[0011] perform key point detection on the multi-scale fusion feature map based on the key point detection sub-model to obtain key point detection data, the key point detection data including predicted key point categories and key point position data of candidate key points, the candidate key points being used to indicate one key point of the detection target;

[0012] perform distance similarity calculation on the candidate key points based on the key point position data to obtain key point distance similarity;

[0013] perform semantic similarity calculation on the candidate key points based on the predicted key point categories and the key point distance similarity to obtain key point semantic similarity;

[0014] perform key point grouping on the key point semantic similarity, the key point distance similarity and the candidate key points based on the key point grouping sub-model to obtain a target key point group, the key points included in the target key point group being used to indicate the same detection target.

[0015] In some embodiments, the image enhancement based on the image frequency domain feature on the initial image to obtain an enhanced image, comprising:

[0016] perform target key feature extraction on the image frequency domain feature to obtain a frequency domain target key feature;

[0017] perform frequency domain inverse transformation on the frequency domain target key feature based on the initial image to obtain the enhanced image.

[0018] In some embodiments, the key point detection based on the key point detection sub-model on the multi-scale fusion feature map to obtain key point detection data, comprising:

[0019] perform category prediction on the fusion feature points in the multi-scale fusion feature map to obtain a probability prediction map of a preset category number, for a preset category, the corresponding probability prediction map including a predicted probability value, the predicted probability value being used to indicate a probability that the fusion feature point belongs to the preset category;

[0020] determine the candidate key points from the fusion feature points based on a peak value extraction algorithm and the predicted probability value, and take the preset category as the predicted key point categories of the candidate key points;

[0021] perform position data extraction on the candidate key points based on the multi-scale fusion feature map to obtain the key point position data;

[0022] determine the key point detection data based on the predicted key point categories and the key point position data.

[0023] In some embodiments, the multi-scale feature fusion sub-model comprises a multi-scale feature extraction layer, a self-attention layer, a cross-attention layer, and a feature fusion layer.

[0024] The multi-scale feature fusion processing of the enhanced image based on the multi-scale feature fusion sub-model comprises:

[0025] The multi-scale feature extraction of the enhanced image based on the multi-scale feature extraction layer comprises obtaining a plurality of enhanced feature maps of different resolutions.

[0026] The feature map connection of the plurality of enhanced feature maps of different resolutions based on the preset pyramid structure comprises obtaining a target pyramid structure, wherein the target pyramid structure comprises a candidate feature map and adjacent feature maps of the candidate feature map.

[0027] The self-attention processing of the candidate feature map based on the self-attention layer comprises obtaining a self-attention weight.

[0028] The feature value adjustment of the candidate feature map based on the self-attention weight comprises obtaining an adjusted feature map.

[0029] The cross-attention processing of the adjusted feature map and the adjacent feature maps of the candidate feature map based on the cross-attention layer comprises obtaining a cross-attention weight.

[0030] The feature value adjustment of the adjusted feature map based on the cross-attention weight comprises obtaining a target feature map.

[0031] The feature fusion of the target feature map in the target pyramid structure based on the feature fusion layer comprises obtaining the multi-scale fusion feature map.

[0032] In some embodiments, the semantic similarity calculation of the candidate key point based on the predicted key point category and the key point distance similarity comprises:

[0033] Selecting a first key point and a second key point from the candidate key point.

[0034] The category similarity calculation of the first predicted key point category of the first key point and the second predicted key point category of the second key point comprises obtaining a category similarity.

[0035] Obtaining a first category description text of the first predicted key point category and a second category description text of the second predicted key point category.

[0036] The text similarity calculation of the first category description text and the second category description text comprises obtaining a text similarity.

[0037] The distance similarity difference is obtained by subtracting the preset distance similarity threshold from the key point distance similarity;

[0038] The first weight of the category similarity and the second weight of the text similarity are determined based on the distance similarity difference;

[0039] The semantic similarity is obtained by weighting and calculating based on the category similarity, the first weight, the text similarity and the second weight.

[0040] In some embodiments, before the enhanced sample image is input into the pre-trained multi-target detection model, the method further comprises: constructing the multi-target detection model, specifically comprising:

[0041] Obtain training samples, the training samples comprising sample images and sample target key point groups, the sample images comprising multiple sample detection targets;

[0042] Input the sample images into an initial detection model, the initial detection model comprising the multi-scale feature fusion sub-model, the key point detection sub-model and the key point grouping sub-model;

[0043] Perform multi-scale feature fusion processing on the sample images based on the multi-scale feature fusion sub-model to obtain sample multi-scale fusion feature maps;

[0044] Perform key point detection on the sample multi-scale fusion feature maps based on the key point detection sub-model to obtain sample key point detection data, the sample key point detection data comprising sample predicted key point categories of sample candidate key points and sample key point position data, the sample candidate key points being used to indicate a key point of the sample detection target;

[0045] Perform semantic similarity calculation on the sample candidate key points based on the sample predicted key point categories and the sample key point position data to obtain sample key point semantic similarity;

[0046] Perform distance similarity calculation on the sample candidate key points based on the sample key point position data to obtain sample key point distance similarity;

[0047] Perform key point grouping on the sample key point semantic similarity, the sample key point distance similarity and the sample candidate key points based on the key point grouping sub-model to obtain a predicted target key point group, the key points contained in the predicted target key point group being used to indicate the same sample detection target;

[0048] perform loss value calculation based on the sample image, the number of sample detection targets, the sample target key point group, and the predicted target key point group, to obtain a model loss value;

[0049] perform parameter adjustment on the initial detection model based on the model loss value, to obtain the multi-target detection model.

[0050] In some embodiments, the performing loss value calculation based on the sample image, the number of sample detection targets, the sample target key point group, and the predicted target key point group, to obtain a model loss value comprises:

[0051] extracting sample target key points and first key point position data of the sample target key points from the sample target key point group;

[0052] extracting predicted target key points and second key point position data of the predicted target key points from the predicted target key point group;

[0053] performing loss calculation based on the first key point position data and the second key point position data of the same sample detection target, to obtain a prediction loss value;

[0054] performing key point conflict calculation based on the first key point position data and the second key point position data of any two different sample detection targets, to obtain a key point conflict value;

[0055] performing model loss calculation based on the prediction loss value and the key point conflict value, to obtain the model loss value.

[0056] To achieve the above object, a second aspect of the embodiments of the present application proposes a multi-target detection device based on images, which comprises:

[0057] an acquisition module configured to acquire an initial image, wherein the initial image comprises a plurality of detection targets;

[0058] a transformation module configured to perform frequency domain transformation on the initial image, to obtain image frequency domain features;

[0059] an enhancement module configured to perform image enhancement on the initial image based on the image frequency domain features, to obtain an enhanced image;

[0060] an input module configured to input the enhanced sample image into a pre-trained multi-target detection model, wherein the multi-target detection model comprises a multi-scale feature fusion sub-model, a key point detection sub-model, and a key point grouping sub-model;

[0061] a fusion module configured to perform multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model, to obtain a multi-scale fused feature map.

[0062] detecting a key point based on the multi-scale fusion feature map by using the key point detection sub-model to obtain key point detection data, the key point detection data including predicted key point categories and key point position data of candidate key points, the candidate key points being used to indicate a key point of the detection target;

[0063] a first calculation module configured to calculate a distance similarity of the candidate key points based on the key point position data to obtain a key point distance similarity;

[0064] a second calculation module configured to calculate a semantic similarity of the candidate key points based on the predicted key point categories and the key point distance similarity to obtain a key point semantic similarity;

[0065] a grouping module configured to group the key point semantic similarity, the key point distance similarity and the candidate key points based on the key point grouping sub-model to obtain a target key point group, the target key point group including key points used to indicate the same detection target.

[0066] To achieve the above object, a third aspect of embodiments of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method of the first aspect when executing the computer program.

[0067] To achieve the above object, a fourth aspect of embodiments of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program implementing the method of the first aspect when executed by a processor.

[0068] The embodiment of the application provides a kind of based on image multi-target detection method and device, electronic equipment, storage medium, first, obtain initial image, the initial image includes multiple detection targets;Image frequency domain feature is obtained by carrying out frequency domain transformation to initial image;Based on image frequency domain feature, image enhancement is carried out to initial image, and enhanced image is obtained;Further, enhanced sample image is input into pre-trained multi-target detection model, and the multi-target detection model includes multi-scale feature fusion submodel, key point detection submodel and key point grouping submodel;Based on multi-scale feature fusion submodel, multi-scale feature fusion processing is carried out to enhanced image, and multi-scale fusion feature map is obtained;Based on key point detection submodel, key point detection is carried out to multi-scale fusion feature map, and key point detection data is obtained, the key point detection data includes the predicted key point category and key point position data of candidate key point, and candidate key point is used to indicate the key point of detection target;Further, based on key point position data, distance similarity calculation is carried out to candidate key point, and key point distance similarity is obtained;Based on predicted key point category and key point distance similarity, semantic similarity calculation is carried out to candidate key point, and key point semantic similarity is obtained;Further, based on key point grouping submodel, key point semantic similarity, key point distance similarity and candidate key point are grouped, and target key point group is obtained, and the key point contained in the target key point group is used to indicate the same detection target.Compared with the related art, the association information between other target information in the image and the to-be-identified target is not deeply understood, so that the influence of problems such as shielding, background interference and illumination is easily received, the initial image is enhanced by the image frequency domain feature after frequency domain transformation, and the feature information related to the target key point is enhanced by combining multi-scale feature fusion and key point detection, the association information between other target information in the image and the to-be-identified target can be deeply understood, the feature expression and key point detection accuracy are improved, and more accurate key point grouping is realized based on key point distance similarity and key point semantic similarity.Therefore, the embodiment of the application can effectively improve the accuracy of multi-target detection based on image. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 is a flow chart of the multi-target detection method based on image provided by the embodiment of the application;

[0070] Figure 2 is Figure 1 a flow chart of step S130 in

[0071] Figure 3 is Figure 1 a flow chart of step S150 in

[0072] Figure 4 is Figure 1 a flow chart of step S160 in

[0073] Figure 5 is Figure 1 a flowchart of step S180 in

[0074] Figure 6 is a flowchart of constructing a multi-target detection model provided by an embodiment of the present application;

[0075] Figure 7 is Figure 6 a flowchart of step S680 in

[0076] Figure 8 is a structural schematic diagram of a multi-target detection device based on images provided by an embodiment of the present application;

[0077] Figure 9 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0079] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0080] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0081] First, the terms involved in the present application are analyzed:

[0082] Artificial Intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; Artificial intelligence is a branch of computer science, artificial intelligence tries to understand the essence of intelligence, and produce a new intelligent machine that can react in a similar way to human intelligence, including robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results.

[0083] Kalman Filter: is a recursive algorithm used to estimate the state of a dynamic system, widely used in control systems, navigation, signal processing and other fields. Kalman filter can provide optimal estimation of system state by combining system mathematical model and observation data to minimize the mean square error of estimation.

[0084] Image-based multi-target detection is a method of automatically identifying multiple targets from an image. In the insurance claim scenario of financial technology, the multi-target detection method can be used to identify the situation of multiple targets such as vehicles, pedestrians, roads, obstacles, etc. in the video taken by the vehicle to be claimed, so as to determine the specific claim scheme. However, when one or more targets are partially blocked by other targets, it may not be able to accurately detect the key points of the blocked target. When there is a light problem of overexposure or overdarkness in the image, it may not be able to guarantee the stability and robustness of key point detection. In addition, when there is a background interference problem of objects similar or confusing to the human body that needs to be identified in the image, it may not be able to avoid false detection or missed detection. Therefore, due to the problems of occlusion, background interference and light, it is easy to affect the accuracy of multi-target detection.

[0085] Therefore, the related art usually only considers target detection in a single target scene, and does not deeply understand the association information between other target information in the image and the target to be identified when performing multi-target detection, so it is easy to be affected by the problems of occlusion, background interference and light, reducing the detection accuracy of multiple targets in the image. Therefore, how to improve the accuracy of image-based multi-target detection has become a technical problem to be solved.

[0086] Therefore, the related art usually only considers target detection in a single target scene, and does not deeply understand the association information between other target information in the image and the target to be identified when performing multi-target detection, so it is easy to be affected by the problems of occlusion, background interference and light, reducing the detection accuracy of multiple targets in the image. Therefore, how to improve the accuracy of image-based multi-target detection has become a technical problem to be solved.

[0087] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving an environment, acquiring knowledge and using the knowledge to obtain optimal results.

[0088] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0089] The image-based multi-target detection method provided by the embodiments of the present application relates to the field of artificial intelligence. The image-based multi-target detection method provided by the embodiments of the present application can be applied in a terminal, can also be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (CDN) and big data and artificial intelligence platform, etc.; and the software can be an application for implementing the image-based multi-target detection method, etc., but is not limited to the above forms.

[0090] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0091] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the object, such as object attribute information, object action data, and object location information, the permission or consent of the object is obtained first. Moreover, the collection, use, and processing of such data comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the object, the separate permission or separate consent of the object is obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the object, the necessary object-related data for enabling the embodiments of the present application to normally operate is obtained.

[0092] Please refer to Figure 1 , Figure 1 is an optional flowchart of the image-based multi-target detection method provided by the embodiments of the present application. Figure 1 The method in the embodiments of the present application can specifically include but is not limited to steps S110 to S190, which are described below in combination with Figure 1 The nine steps are described in detail.

[0093] Step S110, obtaining an initial image;

[0094] Step S120, performing frequency domain transformation on the initial image to obtain image frequency domain features;

[0095] Step S130, performing image enhancement on the initial image based on the image frequency domain features to obtain an enhanced image;

[0096] Step S140, inputting the enhanced sample image into a pre-trained multi-target detection model;

[0097] Step S150, performing multi-scale feature fusion processing on the enhanced image based on a multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map;

[0098] Step S160, performing key point detection on the multi-scale fusion feature map based on a key point detection sub-model to obtain key point detection data;

[0099] Step S170, performing distance similarity calculation on the candidate key points based on the key point location data to obtain key point distance similarity;

[0100] Step S180, performing semantic similarity calculation on the candidate key points based on the predicted key point category and the key point distance similarity to obtain key point semantic similarity;

[0101] Step S190, performing key point grouping on the key point semantic similarity, the key point distance similarity, and the candidate key points based on a key point grouping sub-model to obtain a target key point group.

[0102] In steps S110-S190 of some embodiments, compared with the related art, the association information between other target information in the image and the to-be-identified target is not deeply understood, so that the influence of problems such as occlusion, background interference, and illumination is easily received, the image enhancement is performed on the initial image by using the image frequency domain features after the frequency domain transformation, and the feature information related to the target key points is enhanced by combining the multi-scale feature fusion and the key point detection, so that the association information between other target information in the image and the to-be-identified target can be deeply understood, the feature expression and the key point detection accuracy can be improved, and more accurate key point grouping can be realized based on the key point distance similarity and the key point semantic similarity. Therefore, the embodiments of the present application can effectively improve the accuracy of the multi-target detection based on the image.

[0103] In step S110 of some embodiments, the initial image refers to an image that needs to be subjected to target detection, and the initial image can include a plurality of detection targets that need to be detected. The detection target can be a person, a vehicle, an animal, or the like in the image, and is not specifically limited. The initial image can be any frame image in an input video segment to be detected, or can be an input image to be detected, and is not specifically limited here.

[0104] It should be noted that in the unmanned aerial vehicle application scenario, the initial image of the embodiments of the present application can also be an image collected by a camera mounted on a target unmanned aerial vehicle. The camera can perform optical zooming on the camera through camera calibration, that is, setting the focal length, camera intrinsic matrix, distortion coefficient, and other parameters of the camera under different zooming multiples, to collect images under different resolutions and pixels, so as to automatically adjust the pixels of the collected images.

[0105] It should be noted that after obtaining the initial image, the present application can first perform image preprocessing on the initial image. The image preprocessing can include image denoising (such as using a Gaussian filter, a mean filter, or the like to remove noise in the image), enhancing contrast (such as using histogram equalization, contrast limited adaptive histogram equalization (CLAHE), or the like to improve the contrast of the image and make the details more obvious), adjusting brightness (that is, adjusting according to the overall brightness level of the image to ensure that the image is not too dark or too bright, so as to facilitate subsequent processing), and the like, to update the initial image, thereby effectively improving the quality of the initial image and further improving the accuracy of the multi-target detection in the initial image.

[0106] In step S120 of some embodiments, since the related art usually only uses the information in the spatial domain for key point detection, the information in the frequency domain is not fully utilized for key point detection. Based on this, after obtaining the initial image, the present application performs frequency domain transformation on the initial image to obtain image frequency domain features. Specifically, the present application can convert the image from the spatial domain to the frequency domain through Fourier transformation (such as fast Fourier transformation, two-dimensional discrete Fourier transformation, etc.), wavelet transformation, etc., so that the image can be decomposed into sinusoidal components of different frequencies and directions, and the obtained image frequency domain features can better reveal the frequency information of the image.

[0107] In step S130 of some embodiments, further, the initial image can be image enhanced based on the obtained image frequency domain features, such as enhancing specific frequency components through a filter, so as to enhance the clarity, contrast, etc. of the key points related to the detection target in the image.

[0108] Please refer to Figure 2 , Figure 2 is an optional flowchart of step S130 provided by the embodiments of the present application. In some embodiments of the present application, step S130 can specifically include but is not limited to steps S210 to S220, which will be described in detail below. Figure 2 .

[0109] Step S210, target key feature extraction is performed on the image frequency domain features to obtain frequency domain target key features;

[0110] Step S220, frequency domain inverse transformation is performed on the frequency domain target key features based on the initial image to obtain an enhanced image.

[0111] In step S210 of some embodiments, after obtaining the image frequency domain features in the frequency domain, the features related to the key points of the detection target (such as human body) can be extracted first to obtain the frequency domain target key features, and the frequency domain target key features are used to represent the features related to the key points of the detection target (such as human body) extracted from the image frequency domain features. The frequency domain target key features can reflect the core features such as edges, textures, shapes, etc. of the detection target in the initial image, and these features can include the amplitude and phase information of the frequency components. In addition, the specific extraction method of the target key feature extraction can be basic threshold judgment, selection of different frequency components, etc., or application of machine learning and deep learning technology to automatically identify the most representative frequency domain features. No specific limitation is made.

[0112] It should be noted that in the frequency domain, human key points (such as joints, facial features, etc.) correspond to some specific frequency components. By analyzing the energy distribution in the frequency domain, the frequency components related to the human key points can be identified. Therefore, the method of extracting target key features from image frequency domain features can include: calculating the energy in a specific frequency range in the frequency domain as a key point related feature. Filtering the frequency domain to retain the frequency components related to the key points and extracting features such as amplitude and phase. Using multi-resolution analysis methods such as wavelet transform, extract key point related features at different scales, and combine the extracted key point related features into a feature vector as the representation of the image / video frame, i.e. obtain the frequency domain target key features. These feature vectors can be used for subsequent human pose estimation, action recognition, etc.

[0113] In step S220 of some embodiments, since the subsequent information extraction also needs to use the image, it is necessary to further perform frequency domain inverse transform on the frequency domain target key features based on the initial image, convert the frequency domain back to the spatial domain, and obtain an enhanced image after enhancing the initial image. Wherein, the frequency domain inverse transform can include using inverse Fourier transform to reconstruct the frequency domain features into the enhanced image. In this way, the enhanced image will retain the key features of the detection target, while possibly reducing noise or irrelevant information, making the detection target more prominent in the enhanced image.

[0114] In the above embodiments, the present application enhances the initial image by transforming the initial image in the frequency domain and the spatial domain, and extracting frequency domain features related to the key points of the detection target in the frequency domain. This method combines the advantages of frequency domain analysis, can enhance the feature area related to the key points, suppress irrelevant or interfering feature areas, effectively handle factors such as occlusion, illumination, background interference, etc., and improve the feature expression of the image.

[0115] In step S140 of some embodiments, after obtaining the enhanced image, in order to accurately detect the specific key points of each detection target in the enhanced image and accurately locate the position of the detection target, the present application can input the enhanced image into a pre-trained multi-target detection model. The multi-target detection model of the present application can be constructed based on attention mechanism, multi-scale feature fusion and key point recognition grouping, and specifically can include a multi-scale feature fusion sub-model, a key point detection sub-model and a key point grouping sub-model to improve the accuracy and robustness of key point detection.

[0116] In step S150 of some embodiments, after inputting the multi-target detection model, the enhanced image can be first subjected to multi-scale feature fusion processing based on a multi-scale feature fusion sub-model, different size convolution kernels or down-sampling rates are used to obtain feature maps of different resolutions, and feature map fusion is performed to obtain a multi-scale fusion feature map. The multi-scale feature fusion sub-model of the present application can include a multi-scale feature extraction layer, a self-attention layer, a cross-attention layer, and a feature fusion layer. In other words, the multi-scale feature fusion sub-model constructed by the present application combines attention mechanism and multi-scale feature fusion to improve the performance of key point detection, which can effectively utilize the context information in the multi-target scene, enhance the feature region related to the key point, and integrate the feature information of different scales and different domains to improve the quality and efficiency of feature expression.

[0117] It should be noted that the present application integrates feature maps of different scales by stacking feature maps to form a comprehensive feature representation, which contains both detailed image information and high-level semantic features, and can enhance the ability of the network in processing multi-scale targets, so that it can better capture different levels of information in the image.

[0118] Please refer to Figure 3 , Figure 3 is an optional flowchart of step S150 provided by the embodiments of the present application. In some embodiments of the present application, step S150 can specifically include but is not limited to steps S310 to S370, which will be described in detail below in combination with Figure 3 The seven steps will be described in detail.

[0119] Step S310, multi-scale feature extraction is performed on the enhanced image based on a multi-scale feature extraction layer to obtain a plurality of enhanced feature maps of different resolutions;

[0120] Step S320, the plurality of enhanced feature maps of different resolutions are subjected to feature map connection based on a preset pyramid structure to obtain a target pyramid structure;

[0121] Step S330, self-attention processing is performed on the candidate feature map based on a self-attention layer to obtain a self-attention weight;

[0122] Step S340, the candidate feature map is subjected to feature value adjustment based on the self-attention weight to obtain an adjusted feature map;

[0123] Step S350, cross-attention processing is performed on adjacent feature maps of the adjusted feature map and the candidate feature map based on a cross-attention layer to obtain a cross-attention weight;

[0124] Step S360, the adjusted feature map is subjected to feature value adjustment based on the cross-attention weight to obtain a target feature map;

[0125] At step S370, the feature fusion layer is used to fuse the target feature maps in the target pyramid structure to obtain a multi-scale fused feature map.

[0126] At step S310 of some embodiments, in the multi-scale feature fusion sub-model, rich feature representations can be extracted from the input enhanced image based on a multi-scale feature extraction layer, i.e., the multi-scale feature extraction layer can be constructed by using convolution kernels of different sizes or a hierarchical structure with different down-sampling rates, so as to obtain a plurality of enhanced feature maps with different resolutions by considering information of different scales, and the goal is to capture details and global information in the image.

[0127] At step S320 of some embodiments, further, the plurality of enhanced feature maps with different resolutions obtained can be connected or stacked according to a preset pyramid structure to obtain a target pyramid structure. The target pyramid structure includes a candidate feature map and a neighboring feature map of the candidate feature map. That is, by constructing a pyramid structure, different scale feature maps can be effectively managed. The preset pyramid structure refers to a pyramid structure constructed in advance based on actual needs, which can include feature map size requirements corresponding to each level, so that the model can effectively utilize feature information of different resolutions in the subsequent processing stage.

[0128] At step S330 of some embodiments, further, the stacked feature maps can be processed by an attention mechanism, and specifically, the similarity between each position in the candidate feature map can be calculated by a self-attention layer, i.e., a self-attention layer is used to calculate the correlation between each pixel and other pixels in each feature map to generate a corresponding self-attention weight. In this way, the model can automatically identify and emphasize important features or regions in the image, rather than uniformly paying attention to all positions.

[0129] At step S340 of some embodiments, further, the feature values in the candidate feature map are weighted and adjusted using the calculated self-attention weight, i.e., a weighted sum method can be used according to the obtained self-attention weight. At this time, features with high weights are enhanced, and features with low weights can be weakened, thereby obtaining a new feature map for subsequent processing.

[0130] At step S350 of some embodiments, further, the adjusted feature map is processed with its neighboring candidate feature map based on a cross-attention layer, i.e., a cross-attention layer is used to calculate the correlation between each pixel and other pixels in different feature maps, and the feature values of each pixel are adjusted according to the correlation weight (i.e., cross-attention weight). The cross-attention weight can reflect the relative importance between feature maps. The embodiments of the present application can establish effective connections between different feature maps through the cross-attention layer to promote the interaction of information.

[0131] In step S360 of some embodiments, further, the adjusted feature maps are adjusted again using the generated cross-attention weights, and the introduction of the cross-attention weights helps to fuse information from different feature maps to form more accurate and comprehensive feature representations to obtain the final target feature maps.

[0132] In step S370 of some embodiments, further, in the feature fusion layer, all target feature maps in the target pyramid structure are integrated. At this time, the feature fusion can specifically adopt addition, splicing or other fusion strategies, and finally generate a multi-scale fusion feature map, which contains rich multi-scale information and provides better support for subsequent tasks. That is, up-sampling processing can be used to up-sample the low-resolution target feature maps to high resolution, and the feature maps of different resolutions are weighted and summed according to certain weights to obtain a final multi-scale fusion feature map. The target feature maps obtained by the present application can contain 3 to 5 layers, and the resolution size ratio of adjacent layers is 2, and the specific value is related to the training data. Before fusion, it needs to be up-sampled to the same resolution, and generally up-sampled to the maximum resolution value.

[0133] In the above embodiments, the self-attention layer and the cross-attention layer only input differently, and the principles are the same. The self-attention layer is used to process a single feature map, and the output is the weight of the feature map, and the matrix size of the weight is the same as the feature map, which can be added directly. The input of the cross-attention layer is multiple feature maps, and the size of different feature maps can be adjusted in the middle calculation process. The present application can be based on the multi-scale feature extraction layer, the self-attention layer, the cross-attention layer and the feature fusion layer to jointly constitute a complex and efficient feature extraction and processing flow. Through multi-scale analysis, self-attention and cross-attention mechanisms and feature fusion, a better multi-scale fusion feature map is finally obtained, which can better improve the accuracy of multi-target detection based on images.

[0134] In step S160 of some embodiments, after obtaining the multi-scale fusion feature map, key point detection can be performed on the multi-scale fusion feature map to identify important points (such as corners, edges, etc. of objects) in the image. These key points are crucial for subsequent target recognition and grouping. The key point detection data includes predicted key point categories of candidate key points and key point position data, the candidate key points are used to indicate a key point (i.e. a pixel point) belonging to a detection target detected from the enhanced image, the predicted key point categories are used to represent the key point categories to which the corresponding candidate key points belong, and the key point position data is used to represent the pixel point position of the corresponding candidate key points in the enhanced image, i.e. the pixel coordinates.

[0135] Please refer to Figure 4 , Figure 4is an optional flowchart of step S160 provided by the embodiments of the present application. In some embodiments of the present application, step S160 can specifically include but is not limited to steps S410 to S440, which are described below in combination with Figure 4 The four steps are described in detail.

[0136] Step S410, class prediction is performed on the fusion feature points in the multi-scale fusion feature map to obtain a probability prediction map of a preset class number.

[0137] Step S420, candidate key points are determined from the fusion feature points based on a peak value extraction algorithm and a prediction probability value, and the preset class is taken as the prediction key point class of the candidate key points.

[0138] Step S430, position data of the candidate key points are extracted based on the multi-scale fusion feature map to obtain key point position data.

[0139] Step S440, key point detection data are determined based on the prediction key point class and the key point position data.

[0140] In step S410 of some embodiments, the multi-scale fusion feature map includes a plurality of fusion feature points (i.e., feature pixel points), and for a preset class, the corresponding probability prediction map (equivalent to a matrix) includes a prediction probability value, which is used to indicate the probability that the fusion feature point belongs to the preset class. That is, through class prediction, the probability that each pixel in the multi-scale fusion feature map belongs to which key point class can be predicted. The probability prediction map has the same size as the input feature map, and the class prediction can be specifically performed by a convolutional neural network, a deep learning network, etc.

[0141] It should be noted that the prediction key point class of the present application can be determined by a training sample set. If the detection target is a human body, the key point class can include corresponding classes of the five senses (such as the left eye, the mouth, etc.), various body joints (such as the left shoulder, the right shoulder, etc.), etc. For example, assuming that the key point class can include 17 classes, class prediction can be performed on the fusion feature points in the multi-scale fusion feature map to obtain 17 probability prediction maps. Moreover, each value in the probability prediction map represents the probability that the corresponding fusion feature point (i.e., pixel) in the multi-scale fusion feature map belongs to a certain key point class.

[0142] In step S420 of some embodiments, further, key points with high confidence can be identified from the probability prediction map. Specifically, a peak value extraction algorithm (such as non-maximum suppression) can be applied to identify local maximum values in the prediction probability map, which can represent candidate key points that can better represent the detection target, and the preset class is taken as the prediction key point class of the candidate key points.

[0143] In step S430 of some embodiments, further, by traversing the coordinates of the candidate key points, the corresponding position information (such as coordinates and possible offsets) in the multi-scale fused feature map (equivalent to the enhanced image) is extracted to obtain the key point position data of each candidate key point. Moreover, the accuracy of the coordinates can be improved by interpolation methods, for example, using the values of adjacent feature points to optimize the position data, ensuring the accuracy of the position extraction of the candidate key points.

[0144] Process: In step S440 of some embodiments, further, by the predicted key point class and key point position data corresponding to each candidate key point, the key point detection data of the candidate key point can be obtained, facilitating subsequent key point grouping recognition.

[0145] In step S170 of some embodiments, after determining the relevant data of the multiple candidate key points related to the detection target in the enhanced image, the distance similarity of the relative positions of the detected candidate key points can be further calculated, i.e., from the perspective of spatial distance, the key points that may belong to the same detection target are identified.

[0146] It should be noted that by calculating the similarity between the probability maps, it can be further determined which key points may belong to the same person. Common similarity measures include cosine similarity, correlation coefficient, etc., which can help distinguish which key points belong to the same category or the same person in semantics.

[0147] In step S180 of some embodiments, at the same time, the present application can consider the class of each candidate key point and the distance between them, i.e., the relevance between the enhanced candidate key points can be determined through similar classes and similar distances, i.e., from the perspective of semantic similarity, the relationship between these candidate key points and the detection target can be better understood.

[0148] Please refer to Figure 5 , Figure 5 is an optional flowchart of step S180 provided by the embodiments of the present application. In some embodiments of the present application, step S180 can include but is not limited to steps S510 to S570. The following will be described in detail in combination with Figure 5 The seven steps will be described in detail.

[0149] Step S510, selecting a first key point and a second key point from the candidate key points;

[0150] Step S520, performing class similarity calculation on the first predicted key point class of the first key point and the second predicted key point class of the second key point to obtain the class similarity;

[0151] Step S530, obtaining a first category description text of the first predicted key point category and a second category description text of the second predicted key point category;

[0152] Step S540, performing text similarity calculation on the first category description text and the second category description text to obtain a text similarity;

[0153] Step S550, performing difference calculation on the key point distance similarity and a preset distance similarity threshold to obtain a distance similarity difference;

[0154] Step S560, determining a first weight of the category similarity and a second weight of the text similarity based on the distance similarity difference;

[0155] Step S570, performing weighted sum calculation based on the category similarity, the first weight, the text similarity and the second weight to obtain a semantic similarity.

[0156] In step S510 of some embodiments, when performing semantic similarity calculation on any two candidate key points, two key points can be selected from the plurality of candidate key points for analysis.

[0157] It should be noted that the first key point and the second key point can also be selected from the candidate key point list according to certain standards (such as confidence, distance, etc.).

[0158] In step S520 of some embodiments, in order to quantify the similarity between the predicted categories of the first key point and the second key point, an appropriate similarity measure (such as cosine similarity, Jaccard similarity, etc.) can be used to compare the two categories and calculate the category similarity. This enables the correlation of the two key points at the category level to be evaluated.

[0159] In step S530 of some embodiments, in order to further classify categories to determine whether the corresponding first key point and second key point belong to the same detection target, the text description of each predicted key point category in the enhanced image can be obtained. These texts can contain attributes, feature words or other related information of the category, for example, if the predicted key point category is an eye, the obtained first category description text can be the shape, size, color of the surrounding area of the eye, etc. to distinguish different detection targets.

[0160] In step S540 of some embodiments, in order to evaluate the similarity between the category description texts, natural language processing techniques (such as cosine similarity, TF-IDF or text embedding) can be used to analyze and calculate the two category description texts to calculate the text similarity, in order to evaluate the semantic relationship of the categories at a deeper level.

[0161] In step S550 of some embodiments, to evaluate the influence of the spatial relationship between the selected first key point and the second key point on the semantic similarity, the application can first calculate the spatial distance between the first key point and the second key point, i.e., the key point distance similarity, and calculate the difference with the set distance threshold to obtain a distance similarity difference, to reflect the influence of the spatial configuration of the selected key points on their similarity.

[0162] In step S560 of some embodiments, further, the weights of the category similarity and the text similarity can be dynamically adjusted according to the size of the distance similarity difference. For example, the smaller the distance similarity difference, the greater the possibility that the two key points belong to different categories, and at this time the first weight of the category similarity can be increased, while the greater the distance similarity difference, the greater the possibility that the two key points belong to different detection targets, and at this time the second weight of the text similarity can be increased, to ensure that the weighting and calculation are more reasonable.

[0163] In step S570 of some embodiments, further, the calculated category similarity and text similarity are multiplied by the corresponding weights respectively, and then added to form a comprehensive semantic similarity score. This score can be used for subsequent similarity evaluation or decision-making, to help understand the relevance between key points.

[0164] In step S190 of some embodiments, further, the similarity information obtained above can be used for key point grouping by using a key point grouping sub-model, to provide more explicit output for target detection, and to form a target key point group corresponding to each detection target. Each target key point group contains key points indicating the same detection target. Specifically, the key point semantic similarity and the key point distance similarity can be combined to obtain a comprehensive score matrix using the weighted sum method. In this step, different weights can be assigned to different types of similarity according to their importance. Further, a clustering algorithm such as K-means, DBSCAN or hierarchical clustering is used to analyze the comprehensive score matrix, to identify key point groups with high similarity. The clustering results will identify a group of key points that are evaluated as possibly belonging to the same detection target.

[0165] It should be noted that after the target key point group corresponding to the same detection target is determined, the rectangular box information of the detection target in the initial image can be determined based on the target key point group, to locate the detection target. In addition, a unique identifier can be assigned to each detection target for user identification.

[0166] Please refer to Figure 6 , Figure 6is another optional flowchart of the image-based multi-target detection method provided by the embodiment of the present application. In some embodiments of the present application, before step S140, the multi-target detection method of the present application can further include: constructing a multi-target detection model, which can specifically include but is not limited to steps S610 to S690, which will be described below in combination with Figure 6 The nine steps will be described in detail.

[0167] Step S610, obtaining a training sample;

[0168] Step S620, inputting the sample image into an initial detection model;

[0169] Step S630, performing multi-scale feature fusion processing on the sample image based on a multi-scale feature fusion sub-model to obtain a sample multi-scale fusion feature map;

[0170] Step S640, performing key point detection on the sample multi-scale fusion feature map based on a key point detection sub-model to obtain sample key point detection data;

[0171] Step S650, performing semantic similarity calculation on the sample candidate key points based on the sample predicted key point category and the sample key point position data to obtain sample key point semantic similarity;

[0172] Step S660, performing distance similarity calculation on the sample candidate key points based on the sample key point position data to obtain sample key point distance similarity;

[0173] Step S670, performing key point grouping on the sample key point semantic similarity, the sample key point distance similarity and the sample candidate key points based on a key point grouping sub-model to obtain a predicted target key point group;

[0174] Step S680, performing loss value calculation based on the sample image, the number of sample detection targets, the sample target key point group and the predicted target key point group to obtain a model loss value;

[0175] Step S690, performing parameter adjustment on the initial detection model based on the model loss value to obtain a multi-target detection model.

[0176] In steps S610 to S670 of some embodiments, the training sample includes a sample image and a sample target key point group, the sample image includes a plurality of sample detection targets. The initial detection model includes a multi-scale feature fusion sub-model, a key point detection sub-model, and a key point grouping sub-model, the sample key point detection data includes sample predicted key point categories of sample candidate key points and sample key point position data, the sample candidate key points are used to indicate a key point of the sample detection target. The sample predicted key point category is equivalent to a category label, and the sample target key point group is equivalent to a combination of key points belonging to the same detection target which has been labeled from the sample image in advance. The specific processing process of steps S620 to S670 can refer to steps S140 to S190 of the above-mentioned embodiments, that is, the data in them are replaced by sample data. In order to save space, it will not be repeated here. The key points contained in the predicted target key point group are used to indicate the same sample detection target.

[0177] It should be noted that the specific categories of the sample detection targets in the present application can be set according to actual task requirements, so that the training sample set can be constructed based on continuous detection tasks of different categories, and the multi-target detection model of different scenes can be trained according to the training samples of different categories. For example, for a vehicle detection task, i.e. detecting vehicle targets. Therefore, different vehicle categories need to be distinguished, such as setting categories including small cars, trucks, buses, motorcycles, and bicycles, and setting corresponding category labels for each vehicle category. For example, for a human-vehicle action trajectory task, i.e. detecting people and vehicles at the same time, there is no need to subdivide vehicle categories, so the categories refer to two categories of people and vehicles, and corresponding category labels are set for the two categories of people and vehicles.

[0178] It should be noted that, in order to improve the recognition accuracy of the category targets to be detected, the present application can perform at least one of multi-scale transformation processing, image random cropping and splicing processing, image fusion processing, etc. on the collected sample images when constructing the training sample set, to obtain new sample images, thereby expanding the number of training samples.

[0179] In step S680 of some embodiments, further, the loss value can be calculated based on the sample image, the number of sample detection targets, the sample target key point group, and the predicted target key point group to obtain the model loss value, so as to train the initial detection model.

[0180] Please refer to Figure 7 , Figure 7 is an optional flowchart of step S680 provided by the embodiments of the present application. In some embodiments of the present application, step S680 can include but is not limited to steps S710 to S750. The following will be described in detail in combination with Figure 7 the five steps.

[0181] Step S710, extracting the sample target key points and the first key point position data of the sample target key points from the sample target key point group;

[0182] Step S720, extracting the prediction target key points and the second key point position data of the prediction target key points from the prediction target key point group;

[0183] Step S730, performing loss calculation based on the first key point position data and the second key point position data of the same sample detection target to obtain a prediction loss value;

[0184] Step S740, performing key point conflict calculation based on the first key point position data and the second key point position data of any two different sample detection targets to obtain a key point conflict value;

[0185] Step S750, performing model loss calculation based on the prediction loss value and the key point conflict value to obtain a model loss value.

[0186] In steps S710 to S730 of some embodiments, the present application can first extract the sample target key points and the first key point position data of the sample target key points from the sample target key point group, and extract the prediction target key points and the second key point position data of the prediction target key points from the prediction target key point group. Further, loss calculation can be performed based on the first key point position data and the second key point position data of the same sample detection target to obtain a prediction loss value, i.e., the difference between the key point position predicted by the model and the real sample key point position is calculated.

[0187] It should be noted that the loss function used by the present application to calculate the prediction loss value can be any one of 0-1 loss function, absolute value loss function, logarithmic loss function, exponential loss function, Hinge loss function, etc., which can be selected and adjusted according to actual needs, and will not be described here.

[0188] In step S740 of some embodiments, further, the present application adds an exclusive loss to the original loss function, which is used to measure whether the predicted key points of different targets overlap or conflict. Specifically, key point conflict calculation is performed based on the first key point position data and the second key point position data of any two different sample detection targets to obtain a key point conflict value. The key point conflict value is equivalent to a penalty term, and the specific calculation process can be seen from the following formula 1:

[0189]

[0190] In formula 1, L excdenotes a key point conflict value, K denotes a number of sample detection targets, N denotes a number of key point categories, and τ denotes a preset conflict threshold value, denotes a position of an i-th predicted target key point predicted in a k-th sample detection target, denotes a position of an i-th predicted target key point predicted in a k-th sample detection target.

[0191] In step S750 of some embodiments, further, model loss calculation is performed based on the prediction loss value and the key point conflict value, to obtain a model loss value. The specific formula for calculating the model loss value can be seen in formula 2 as follows:

[0192] L = a · L0 + β · L exc (Formula 2)

[0193] In formula 2, L exc denotes a key point conflict value, L denotes a model loss value, L0 denotes a prediction loss value, a denotes a weight corresponding to the prediction loss value, and β denotes a weight corresponding to the key point conflict value.

[0194] In step S690 of some embodiments, further, the initial detection model can be adjusted in parameters based on the model loss value, until a preset training end condition is reached to obtain a multi-target detection model. The model structure of the multi-target detection model is the same as that of the initial detection model. The preset end condition can be that a target detection accuracy rate reaches a preset accuracy rate threshold, or a model iteration number reaches a preset training number, and the like, which will not be repeated here.

[0195] It should be noted that the application can also perform key point trajectory detection on the key points in the grouped target key point group. A key point trajectory detection module is used to connect the key points of the same detection target according to the position changes and identifier matching between the key points in adjacent frames, and a Kalman filter is used to smooth and predict the position of each target key point, to realize trajectory detection of the multi-target key points.

[0196] It should be noted that the non-company software tools or components appearing in the embodiments of the application are only examples for introduction, and do not represent actual use.

[0197] The method for detecting multiple targets based on images provided in the embodiments of the present application can realize the detection of multiple target key points from a monocular image or video, without the use of additional sensors or devices, thereby reducing the cost and complexity of the system. Compared with the related art, the correlation between other target information in the image and the to-be-identified target information is not deeply understood, thereby being easily affected by problems such as occlusion, background interference and illumination. The embodiments of the present application can deeply understand the correlation between other target information in the image and the to-be-identified target information by performing image enhancement on the initial image based on the image frequency domain features after frequency domain transformation, and combining multi-scale feature fusion and key point detection to enhance the feature information related to the target key points, thereby improving the feature expression and key point detection accuracy, and realizing more accurate key point grouping based on the key point distance similarity and the key point semantic similarity. Therefore, the embodiments of the present application can effectively improve the accuracy of the multiple target detection based on images.

[0198] Referring to Figure 8 The embodiments of the present application also provide a device for detecting multiple targets based on images, which can implement the method for detecting multiple targets based on images described above, and the device comprises:

[0199] The acquisition module 810 is configured to acquire an initial image, and the initial image comprises multiple detection targets.

[0200] The transformation module 820 is configured to perform frequency domain transformation on the initial image to obtain image frequency domain features.

[0201] The enhancement module 830 is configured to perform image enhancement on the initial image based on the image frequency domain features to obtain an enhanced image.

[0202] The input module 840 is configured to input the enhanced sample image into a pre-trained multiple target detection model, and the multiple target detection model comprises a multi-scale feature fusion sub-model, a key point detection sub-model and a key point grouping sub-model.

[0203] The fusion module 850 is configured to perform multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map.

[0204] The detection module 860 is configured to perform key point detection on the multi-scale fusion feature map based on the key point detection sub-model to obtain key point detection data, and the key point detection data comprises predicted key point categories and key point position data of candidate key points, and the candidate key points are used to indicate a key point of the detection target.

[0205] The first calculation module 870 is configured to perform distance similarity calculation on the candidate key points based on the key point position data to obtain key point distance similarity.

[0206] The second calculation module 880 is configured to perform semantic similarity calculation on the candidate key points based on the predicted key point category and the key point distance similarity, to obtain key point semantic similarity.

[0207] The grouping module 890 is configured to perform key point grouping on the key point semantic similarity, the key point distance similarity and the candidate key points based on a key point grouping sub-model, to obtain a target key point group, and the key points included in the target key point group are used to indicate the same detection target.

[0208] The specific implementation of the image-based multi-target detection device provided in the embodiments of the present application is basically the same as the specific implementation of the image-based multi-target detection method described above, and thus will not be repeated here.

[0209] The embodiments of the present application further provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the image-based multi-target detection method described above when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0210] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0211] The processor 910 can be implemented in the form of a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present application.

[0212] The memory 920 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solutions provided in the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 920 and are called and executed by the processor 910 to implement the image-based multi-target detection method provided in the embodiments of the present application.

[0213] The input / output interface 930 is configured to realize information input and output.

[0214] The communication interface 940 is configured to realize the communication between the device and other devices. The communication can be realized through wired mode (for example, USB, network cable, etc.) or wireless mode (for example, mobile network, WIFI, Bluetooth, etc.).

[0215] The bus 950 is configured to transmit information between various components (for example, the processor 910, the memory 920, the input / output interface 930 and the communication interface 940) of the device.

[0216] The processor 910, the memory 920, the input / output interface 930 and the communication interface 940 are connected to each other through the bus 950.

[0217] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to realize the above-mentioned image-based multi-target detection method.

[0218] The memory is a non-transitory computer readable storage medium, which can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can further include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0219] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0220] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps or different steps.

[0221] The device embodiments described above are only schematic, and the units described as separate components can be or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to realize the purpose of the embodiments of the present application.

[0222] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the function modules / units in the system and the device can be implemented as software, firmware, hardware or appropriate combination thereof.

[0223] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover a changeable order, arrangement, grouping, composition or structure.

[0224] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0225] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0226] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0227] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0228] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical scheme of the present application or the part that contributes to the prior art or the whole or part of the technical scheme can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0229] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A multi-target detection method based on images, characterized in that, The method includes: Acquire an initial image, which includes multiple detection targets; The initial image is subjected to frequency domain transformation to obtain the image frequency domain features; Based on the image frequency domain features, the initial image is enhanced to obtain an enhanced image; The enhanced image is input into a pre-trained multi-object detection model, which includes a multi-scale feature fusion sub-model, a key point detection sub-model, and a key point grouping sub-model. Based on the multi-scale feature fusion sub-model, the enhanced image is subjected to multi-scale feature fusion processing to obtain a multi-scale fused feature map. Keypoint detection is performed on the multi-scale fused feature map based on the keypoint detection sub-model to obtain keypoint detection data. The keypoint detection data includes predicted keypoint categories and keypoint location data of candidate keypoints. The candidate keypoints are used to indicate a keypoint of the detected target. The keypoint detection process based on the keypoint detection sub-model to perform keypoint detection on the multi-scale fused feature map to obtain keypoint detection data includes: predicting the categories of fused feature points in the multi-scale fused feature map to obtain a probability prediction map with a preset number of categories. For a preset category, the corresponding probability prediction map includes a predicted probability value, which indicates the probability that the fused feature point belongs to the preset category; determining the candidate keypoints from the fused feature points based on the peak extraction algorithm and the predicted probability values, and using the preset category as the predicted keypoint category of the candidate keypoints; extracting location data from the candidate keypoints based on the multi-scale fused feature map to obtain keypoint location data; and determining the keypoint detection data based on the predicted keypoint categories and keypoint location data. Based on the key point location data, the distance similarity of the candidate key points is calculated to obtain the key point distance similarity. Based on the predicted keypoint category and the keypoint distance similarity, the semantic similarity of the candidate keypoints is calculated to obtain the keypoint semantic similarity. Based on the key point grouping sub-model, the key point semantic similarity, the key point distance similarity, and the candidate key points are grouped to obtain a target key point group. The key points contained in the target key point group are used to indicate the same detection target.

2. The method according to claim 1, characterized in that, The step of enhancing the initial image based on the image frequency domain features to obtain an enhanced image includes: The image frequency domain features are used to extract key target features to obtain frequency domain key target features; The enhanced image is obtained by performing an inverse frequency domain transformation on the key features of the target in the frequency domain based on the initial image.

3. The method according to claim 1, characterized in that, The multi-scale feature fusion sub-model includes a multi-scale feature extraction layer, a self-attention layer, a cross-attention layer, and a feature fusion layer; The process of performing multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fused feature map includes: Multi-scale feature extraction is performed on the enhanced image based on a multi-scale feature extraction layer to obtain multiple enhanced feature maps with different resolutions; Based on a preset pyramid structure, feature maps of multiple enhanced feature maps of different resolutions are connected to obtain a target pyramid structure, which includes candidate feature maps and adjacent feature maps of the candidate feature maps. The candidate feature map is processed by self-attention layer to obtain self-attention weights. The candidate feature map is adjusted based on the self-attention weights to obtain the adjusted feature map. Based on the cross-attention layer, cross-attention processing is performed on the adjacent feature maps of the adjusted feature map and the candidate feature map to obtain cross-attention weights; Based on the cross-attention weights, the feature values ​​of the adjusted feature map are adjusted to obtain the target feature map; Based on the feature fusion layer, the target feature map in the target pyramid structure is fused to obtain the multi-scale fused feature map.

4. The method according to claim 1, characterized in that, The step of calculating semantic similarity of the candidate keypoints based on the predicted keypoint category and the keypoint distance similarity to obtain the keypoint semantic similarity includes: Select a first key point and a second key point from the candidate key points; The category similarity is calculated between the first predicted key point category of the first key point and the second predicted key point category of the second key point to obtain the category similarity. Obtain the first category description text of the first predicted keypoint category and the second category description text of the second predicted keypoint category; The text similarity between the first category description text and the second category description text is calculated to obtain the text similarity. The difference between the key point distance similarity and the preset distance similarity threshold is calculated to obtain the distance similarity difference. The first weight of the category similarity and the second weight of the text similarity are determined based on the distance similarity difference; The semantic similarity is obtained by weighting and calculating the category similarity, the first weight, the text similarity, and the second weight.

5. The method according to any one of claims 1 to 4, characterized in that, Before inputting the enhanced image into the pre-trained multi-object detection model, the method further includes: constructing the multi-object detection model, specifically including: Acquire training samples, which include sample images and sample target key point groups, wherein the sample images include multiple sample detection targets; The sample image is input into the initial detection model, which includes the multi-scale feature fusion sub-model, the key point detection sub-model, and the key point grouping sub-model. Based on the multi-scale feature fusion sub-model, the sample image is subjected to multi-scale feature fusion processing to obtain a multi-scale fused feature map of the sample. Based on the key point detection sub-model, key point detection is performed on the multi-scale fusion feature map of the sample to obtain sample key point detection data. The sample key point detection data includes the sample predicted key point category and sample key point location data of the sample candidate key points. The sample candidate key points are used to indicate a key point of the sample detection target. Based on the sample predicted key point category and the sample key point location data, the semantic similarity of the sample candidate key points is calculated to obtain the sample key point semantic similarity. Based on the location data of the sample key points, the distance similarity of the sample candidate key points is calculated to obtain the distance similarity of the sample key points. Based on the key point grouping sub-model, the semantic similarity of the sample key points, the distance similarity of the sample key points, and the candidate key points of the sample are grouped into key point groups to obtain the predicted target key point group. The key points contained in the predicted target key point group are used to indicate the same sample detection target. Based on the sample image, the number of detected targets in the sample, the key point group of the sample target, and the key point group of the predicted target, the loss value is calculated to obtain the model loss value; The parameters of the initial detection model are adjusted based on the model loss value to obtain the multi-target detection model.

6. The method according to claim 5, characterized in that, The loss value is calculated based on the sample image, the number of detected targets, the key point group of the sample targets, and the key point group of the predicted targets to obtain the model loss value, including: Extract the sample target key points and the first key point location data of the sample target key points from the sample target key point group; Extract the location data of the predicted target key points and the second key points of the predicted target key points from the predicted target key point group; Loss calculation is performed based on the first key point location data and the second key point location data of the same sample detection target to obtain the predicted loss value; Based on the first key point location data and the second key point location data of any two different sample detection targets, key point conflict calculation is performed to obtain key point conflict value; The model loss is calculated based on the predicted loss value and the key point conflict value to obtain the model loss value.

7. An image-based multi-target detection device, characterized in that, The device includes: The acquisition module is used to acquire an initial image, which includes multiple detection targets; The transformation module is used to perform frequency domain transformation on the initial image to obtain the image frequency domain features; An enhancement module is used to enhance the initial image based on the image frequency domain features to obtain an enhanced image; The input module is used to input the enhanced image into a pre-trained multi-object detection model, which includes a multi-scale feature fusion sub-model, a key point detection sub-model, and a key point grouping sub-model. The fusion module is used to perform multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fused feature map. The detection module is used to perform keypoint detection on the multi-scale fusion feature map based on the keypoint detection sub-model to obtain keypoint detection data. The keypoint detection data includes predicted keypoint categories and keypoint location data of candidate keypoints. The candidate keypoints are used to indicate a keypoint of the detected target. The step of performing keypoint detection on the multi-scale fusion feature map based on the keypoint detection sub-model to obtain keypoint detection data includes: predicting the categories of fused feature points in the multi-scale fusion feature map to obtain a probability prediction map with a preset number of categories. For a preset category, the corresponding probability prediction map includes a predicted probability value, which indicates the probability that the fused feature point belongs to the preset category; determining the candidate keypoints from the fused feature points based on a peak extraction algorithm and the predicted probability values, and using the preset category as the predicted keypoint category of the candidate keypoints; extracting location data from the candidate keypoints based on the multi-scale fusion feature map to obtain keypoint location data; and determining the keypoint detection data based on the predicted keypoint categories and keypoint location data. The first calculation module is used to calculate the distance similarity of the candidate key points based on the key point location data, and obtain the key point distance similarity. The second calculation module is used to calculate the semantic similarity of the candidate key points based on the predicted key point category and the key point distance similarity, so as to obtain the key point semantic similarity. The grouping module is used to group the key points based on the semantic similarity of the key points, the distance similarity of the key points, and the candidate key points according to the key point grouping sub-model, so as to obtain a target key point group. The key points contained in the target key point group are used to indicate the same detection target.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images