Image-based multi-target detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image-based multi-target detection method and device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology. The method comprises the following steps: performing frequency domain transformation on an initial image to obtain an image frequency domain feature; enhancing the initial image based on the image frequency domain features to obtain an enhanced image; processing the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map; detecting the multi-scale fusion feature map based on a key point detection sub-model to obtain key point detection data; calculating the candidate key points based on the key point position data to obtain key point distance similarity; calculating the candidate key points based on the predicted key point category and the key point distance similarity to obtain key point semantic similarity; and based on the key point grouping sub-model, grouping the key point semantic similarity, the key point distance similarity and the candidate key points to obtain a target key point group. According to the embodiment of the invention, the accuracy of image-based multi-target detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to an image-based multi-target detection method and device, electronic device, and storage medium. Background Art

[0002] Image-based multi-target detection is a method that automatically identifies multiple targets from an image. In the insurance claims scenario of FinTech, multi-target detection methods can be used to identify multiple targets such as vehicles, pedestrians, roads, obstacles, etc. in the video captured by the vehicle to be claimed, so as to determine the specific claims plan. Due to occlusion, background interference, and lighting issues, the image may have overexposed or dark areas, and the target in the image may be blocked by other targets, resulting in false detection or missed detection of targets.

[0003] Based on this, related technologies usually only consider target detection in single-target scenarios, and when performing multi-target detection, they do not deeply understand the association information between other target information in the image and the target to be identified, which makes it easy to be affected by occlusion, background interference, and lighting problems, reducing the accuracy of detecting multiple targets in the image. Therefore, how to improve the accuracy of multi-target detection based on images has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an image-based multi-target detection method and device, electronic device, and storage medium, which can improve the accuracy of image-based multi-target detection.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes an image-based multi-target detection method, the method comprising:

[0006] Acquire an initial image, wherein the initial image includes a plurality of detection targets;

[0007] Performing frequency domain transformation on the initial image to obtain image frequency domain features;

[0008] Performing image enhancement on the initial image based on the image frequency domain features to obtain an enhanced image;

[0009] Inputting the enhanced sample image into a pre-trained multi-target detection model, wherein the multi-target detection model includes a multi-scale feature fusion sub-model, a key point detection sub-model, and a key point grouping sub-model;

[0010] Performing multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map;

[0011] Perform key point detection on the multi-scale fusion feature map based on the key point detection sub-model to obtain key point detection data, wherein the key point detection data includes predicted key point categories and key point position data of candidate key points, and the candidate key points are used to indicate a key point of the detection target;

[0012] Calculate the distance similarity of the candidate key points based on the key point position data to obtain the key point distance similarity;

[0013] Calculating semantic similarity of the candidate key points based on the predicted key point categories and the key point distance similarity to obtain key point semantic similarity;

[0014] The key point semantic similarity, the key point distance similarity and the candidate key points are grouped based on the key point grouping sub-model to obtain a target key point group, wherein the key points included in the target key point group are used to indicate the same detection target.

[0015] In some embodiments, the performing image enhancement on the initial image based on the image frequency domain features to obtain an enhanced image includes:

[0016] Extracting target key features from the image frequency domain features to obtain frequency domain target key features;

[0017] Based on the initial image, the frequency domain target key features are inversely transformed in the frequency domain to obtain the enhanced image.

[0018] In some embodiments, performing key point detection on the multi-scale fusion feature map based on the key point detection sub-model to obtain key point detection data includes:

[0019] Performing category prediction on the fused feature points in the multi-scale fused feature map to obtain a probability prediction map of a preset number of categories, wherein for a preset category, the corresponding probability prediction map includes a prediction probability value, and the prediction probability value is used to indicate the probability that the fused feature point belongs to the preset category;

[0020] Determine the candidate key point from the fused feature points based on a peak extraction algorithm and the predicted probability value, and use the preset category as the predicted key point category of the candidate key point;

[0021] Extracting position data of the candidate key points based on the multi-scale fusion feature map to obtain the key point position data;

[0022] The key point detection data is determined based on the predicted key point category and the key point position data.

[0023] In some embodiments, the multi-scale feature fusion sub-model includes a multi-scale feature extraction layer, a self-attention layer, a cross-attention layer, and a feature fusion layer;

[0024] The performing multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map includes:

[0025] Performing multi-scale feature extraction on the enhanced image based on a multi-scale feature extraction layer to obtain a plurality of enhanced feature maps with different resolutions;

[0026] Based on a preset pyramid structure, feature maps of the enhanced feature maps of multiple different resolutions are connected to obtain a target pyramid structure, wherein the target pyramid structure includes a candidate feature map and adjacent feature maps of the candidate feature map;

[0027] Performing self-attention processing on the candidate feature map based on the self-attention layer to obtain a self-attention weight;

[0028] Adjusting the feature value of the candidate feature map based on the self-attention weight to obtain an adjusted feature map;

[0029] Performing cross attention processing on the adjusted feature map and the adjacent feature map of the candidate feature map based on the cross attention layer to obtain a cross attention weight;

[0030] Adjusting the feature value of the adjusted feature map based on the cross attention weight to obtain a target feature map;

[0031] Based on the feature fusion layer, feature fusion is performed on the target feature map in the target pyramid structure to obtain the multi-scale fused feature map.

[0032] In some embodiments, the calculating of semantic similarity of the candidate key points based on the predicted key point categories and the key point distance similarity to obtain key point semantic similarity includes:

[0033] Selecting a first key point and a second key point from the candidate key points;

[0034] Calculating the category similarity of the first predicted key point category of the first key point and the second predicted key point category of the second key point to obtain the category similarity;

[0035] Acquire a first category description text of the first predicted key point category and a second category description text of the second predicted key point category;

[0036] Calculating text similarity between the first category description text and the second category description text to obtain text similarity;

[0037] Calculating the difference between the key point distance similarity and a preset distance similarity threshold to obtain a distance similarity difference;

[0038] Determine a first weight of the category similarity and a second weight of the text similarity based on the distance similarity difference;

[0039] The semantic similarity is obtained by performing a weighted sum calculation based on the category similarity, the first weight, the text similarity and the second weight.

[0040] In some embodiments, before inputting the enhanced sample image into a pre-trained multi-target detection model, the method further includes: constructing the multi-target detection model, specifically including:

[0041] Acquire a training sample, wherein the training sample includes a sample image and a sample target key point group, and the sample image includes a plurality of sample detection targets;

[0042] Inputting the sample image into an initial detection model, wherein the initial detection model includes the multi-scale feature fusion sub-model, the key point detection sub-model and the key point grouping sub-model;

[0043] Performing multi-scale feature fusion processing on the sample image based on the multi-scale feature fusion sub-model to obtain a sample multi-scale fusion feature map;

[0044] Based on the key point detection sub-model, key point detection is performed on the sample multi-scale fusion feature map to obtain sample key point detection data, wherein the sample key point detection data includes sample predicted key point categories and sample key point position data of sample candidate key points, and the sample candidate key points are used to indicate a key point of the sample detection target;

[0045] Calculating the semantic similarity of the sample candidate key points based on the sample prediction key point category and the sample key point position data to obtain the sample key point semantic similarity;

[0046] Calculating the distance similarity of the candidate key points of the sample based on the position data of the key points of the sample to obtain the distance similarity of the key points of the sample;

[0047] Based on the key point grouping sub-model, the sample key point semantic similarity, the sample key point distance similarity and the sample candidate key point are grouped into key points to obtain a predicted target key point group, wherein the key points included in the predicted target key point group are used to indicate the same sample detection target;

[0048] Calculate the loss value based on the sample image, the number of the sample detection targets, the sample target key point group and the predicted target key point group to obtain a model loss value;

[0049] The parameters of the initial detection model are adjusted based on the model loss value to obtain the multi-target detection model.

[0050] In some embodiments, the loss value is calculated based on the sample image, the number of the sample detection targets, the sample target key point group and the predicted target key point group to obtain the model loss value, including:

[0051] Extracting sample target key points and first key point position data of the sample target key points from the sample target key point group;

[0052] Extracting a predicted target key point and second key point position data of the predicted target key point from the predicted target key point group;

[0053] Perform loss calculation based on the first key point position data and the second key point position data of the same sample detection target to obtain a predicted loss value;

[0054] Perform key point conflict calculation based on the first key point position data and the second key point position data of any two different sample detection targets to obtain a key point conflict value;

[0055] A model loss calculation is performed based on the predicted loss value and the key point conflict value to obtain the model loss value.

[0056] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes an image-based multi-target detection device, the device comprising:

[0057] An acquisition module, used for acquiring an initial image, wherein the initial image includes a plurality of detection targets;

[0058] A transformation module, used for performing frequency domain transformation on the initial image to obtain image frequency domain features;

[0059] An enhancement module, used for performing image enhancement on the initial image based on the image frequency domain features to obtain an enhanced image;

[0060] An input module, used for inputting the enhanced sample image into a pre-trained multi-target detection model, wherein the multi-target detection model includes a multi-scale feature fusion sub-model, a key point detection sub-model and a key point grouping sub-model;

[0061] A fusion module, used for performing multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map;

[0062] A detection module, configured to perform key point detection on the multi-scale fusion feature map based on the key point detection sub-model to obtain key point detection data, wherein the key point detection data includes a predicted key point category and key point position data of a candidate key point, and the candidate key point is used to indicate a key point of the detection target;

[0063] A first calculation module, configured to calculate distance similarity of the candidate key points based on the key point position data to obtain key point distance similarity;

[0064] A second calculation module is used to calculate the semantic similarity of the candidate key points based on the predicted key point category and the key point distance similarity to obtain the key point semantic similarity;

[0065] A grouping module is used to group the key points based on the key point semantic similarity, the key point distance similarity and the candidate key points to obtain a target key point group, wherein the key points included in the target key point group are used to indicate the same detection target.

[0066] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0067] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0068] The present application proposes an image-based multi-target detection method and device, electronic device, and storage medium. First, an initial image is acquired, and the initial image includes multiple detection targets; the initial image is transformed in the frequency domain to obtain image frequency domain features; the initial image is enhanced based on the image frequency domain features to obtain an enhanced image; further, the enhanced sample image is input into a pre-trained multi-target detection model, and the multi-target detection model includes a multi-scale feature fusion sub-model, a key point detection sub-model, and a key point grouping sub-model; multi-scale feature fusion processing is performed on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map; and the multi-scale fusion feature map is processed based on the key point detection sub-model. Perform key point detection to obtain key point detection data, which includes predicted key point categories and key point position data of candidate key points, and the candidate key point is used to indicate a key point of the detection target; further, based on the key point position data, distance similarity is calculated for the candidate key points to obtain key point distance similarity; based on the predicted key point categories and key point distance similarity, semantic similarity is calculated for the candidate key points to obtain key point semantic similarity; further, based on the key point grouping sub-model, key point semantic similarity, key point distance similarity and candidate key points are grouped to obtain a target key point group, and the key points included in the target key point group are used to indicate the same detection target. Compared with the related art, which does not deeply understand the association information between other target information in the image and the target to be identified, and is easily affected by occlusion, background interference, and lighting problems, the present application enhances the initial image through the image frequency domain features after frequency domain transformation, and combines multi-scale feature fusion and key point detection to enhance the feature information related to the target key points, which can deeply understand the association information between other target information in the image and the target to be identified, improve the accuracy of feature expression and key point detection, and achieve more accurate key point grouping based on key point distance similarity and key point semantic similarity. Therefore, the embodiment of the present application can effectively improve the accuracy of image-based multi-target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flow chart of an image-based multi-target detection method provided in an embodiment of the present application;

[0070] Figure 2 yes Figure 1 A flowchart of step S130 in FIG.

[0071] Figure 3 yes Figure 1 A flowchart of step S150 in FIG.

[0072] Figure 4 yes Figure 1 A flowchart of step S160 in FIG.

[0073] Figure 5 yes Figure 1 A flowchart of step S180 in FIG.

[0074] Figure 6 It is a flow chart of building a multi-target detection model provided by an embodiment of the present application;

[0075] Figure 7 yes Figure 6 A flowchart of step S680 in FIG.

[0076] Figure 8 is a structural schematic diagram of an image-based multi-target detection device provided in an embodiment of the present application;

[0077] Fig. 9 It is a hardware structure diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0079] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0080] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0081] First, some nouns involved in this application are analyzed:

[0082] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0083] Kalman Filter: A recursive algorithm for estimating the state of a dynamic system, widely used in control systems, navigation, signal processing and other fields. Kalman filter can provide the best estimate of the system state by combining the mathematical model of the system with the observed data to minimize the mean square sum of the estimation error.

[0084] Image-based multi-target detection is a method that automatically identifies multiple targets from an image. In the insurance claims scenario of FinTech, the multi-target detection method can be used to identify multiple targets such as vehicles, pedestrians, roads, obstacles, etc. in the video captured by the vehicle to be claimed, so as to determine the specific claims plan. However, when one or more targets are partially obscured by other targets, it may not be possible to accurately detect the key points of the obscured targets. When there are lighting problems in overexposed or dark areas in the image, the stability and robustness of key point detection may not be guaranteed. In addition, when there are background interference problems with objects that are similar or confusing to the human body to be identified in the image, false detection or missed detection may not be avoided. Therefore, due to problems such as occlusion, background interference, and lighting, the accuracy of multi-target detection is easily affected.

[0085] Based on this, related technologies usually only consider target detection in single-target scenarios, and when performing multi-target detection, they do not deeply understand the association information between other target information in the image and the target to be identified, which makes it easy to be affected by occlusion, background interference, and lighting problems, reducing the accuracy of detecting multiple targets in the image. Therefore, how to improve the accuracy of multi-target detection based on images has become a technical problem that needs to be solved urgently.

[0086] Based on this, the embodiments of the present application provide an image-based multi-target detection method and device, an electronic device, and a storage medium, which can improve the accuracy of image-based multi-target detection.

[0087] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0088] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0089] The image-based multi-target detection method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The image-based multi-target detection method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or can be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and cloud servers that provide basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application that implements an image-based multi-target detection method, etc., but is not limited to the above forms.

[0090] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0091] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as object attribute information, object action data, and object location information, the permission or consent of the object will be obtained first. Moreover, the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the sensitive personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the object, the necessary object-related data for enabling the normal operation of the embodiment of the present application will be obtained.

[0092] See also Figure 1 , Figure 1 It is an optional flowchart of the image-based multi-target detection method provided in an embodiment of the present application. Figure 1 The method may specifically include but is not limited to steps S110 to S190. Figure 1 These nine steps are introduced in detail.

[0093] Step S110, obtaining an initial image;

[0094] Step S120, performing frequency domain transformation on the initial image to obtain image frequency domain features;

[0095] Step S130, performing image enhancement on the initial image based on the image frequency domain features to obtain an enhanced image;

[0096] Step S140, inputting the enhanced sample image into a pre-trained multi-target detection model;

[0097] Step S150, performing multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map;

[0098] Step S160, performing key point detection on the multi-scale fusion feature map based on the key point detection sub-model to obtain key point detection data;

[0099] Step S170, calculating the distance similarity of the candidate key points based on the key point position data to obtain the key point distance similarity;

[0100] Step S180, calculating the semantic similarity of the candidate key points based on the predicted key point categories and the key point distance similarity to obtain the key point semantic similarity;

[0101] Step S190 , grouping key points based on the key point semantic similarity, key point distance similarity and candidate key points to obtain a target key point group.

[0102] In steps S110 to S190 of some embodiments, compared to the related art, which does not deeply understand the association information between other target information in the image and the target to be identified, and is easily affected by occlusion, background interference, and lighting problems, the present application enhances the initial image through the image frequency domain features after frequency domain transformation, and combines multi-scale feature fusion and key point detection to enhance the feature information related to the target key points, which can deeply understand the association information between other target information in the image and the target to be identified, improve the accuracy of feature expression and key point detection, and achieve more accurate key point grouping based on key point distance similarity and key point semantic similarity. Therefore, the embodiments of the present application can effectively improve the accuracy of image-based multi-target detection.

[0103] In step S110 of some embodiments, the initial image refers to an image that needs to be detected, and the initial image may include multiple detection targets that need to be detected. The detection target may be a person, a vehicle, an animal, etc. in the image, without specific limitation. The initial image may be any frame image in the input video segment to be detected, or may be an input image to be detected, without specific limitation.

[0104] It should be noted that in the drone application scenario, the initial image of the embodiment of the present application may also be an image captured by an aerial camera set on the target drone. The aerial camera may perform optical zoom on the camera through camera calibration, that is, set the focal length, camera internal parameter matrix, distortion coefficient and other parameters of the camera at different zoom factors to capture images at different resolutions and pixels, thereby automatically adjusting the pixels of the captured image.

[0105] It should be noted that after obtaining the initial image, the present application can first perform image preprocessing on the initial image. The image preprocessing may include image denoising (such as using Gaussian filtering, mean filtering and other filters to remove noise in the image), contrast enhancement (such as using histogram equalization, contrast limited adaptive histogram equalization (CLAHE) and other technologies to enhance the contrast of the image and make the details more obvious), brightness adjustment (i.e., adjusting according to the overall brightness level of the image to ensure that the image is not too dark or too bright for subsequent processing), and other operations to update the initial image, thereby effectively improving the quality of the initial image, and then improving the accuracy of multi-target detection in the initial image.

[0106] In step S120 of some embodiments, since the relevant technology usually only uses the information in the spatial domain to detect key points, and does not fully utilize the information in the frequency domain to detect key points. Based on this, after obtaining the initial image, the present application will perform a frequency domain transform on the initial image to obtain the image frequency domain features. Among them, the present application can specifically convert the image from the spatial domain to the frequency domain through Fourier transform (such as fast Fourier transform, two-dimensional discrete Fourier transform, etc.), wavelet transform, etc., so that the image can be decomposed into sinusoidal wave components of different frequencies and directions, and the obtained image frequency domain features can better reveal the frequency information of the image.

[0107] In step S130 of some embodiments, the initial image can further be enhanced based on the obtained image frequency domain features. For example, a specific frequency component can be enhanced through a filter, thereby enhancing the clarity, contrast and other characteristics of key points related to the detection target in the image.

[0108] See also Figure 2 , Figure 2 is an optional flowchart of step S130 provided in an embodiment of the present application. In some embodiments of the present application, step S130 may specifically include but is not limited to steps S210 to S220. Figure 2 These two steps are introduced in detail.

[0109] Step S210, extracting target key features from the image frequency domain features to obtain frequency domain target key features;

[0110] Step S220, performing frequency domain inverse transformation on the frequency domain target key features based on the initial image to obtain an enhanced image.

[0111] In step S210 of some embodiments, after obtaining the image frequency domain features in the frequency domain, features related to the key points of the detection target (such as the human body) can be extracted therefrom to obtain the frequency domain target key features, and the frequency domain target key features are used to characterize the features related to the key points of the detection target (such as the human body) extracted from the image frequency domain features. Among them, the frequency domain target key features can reflect the core features such as the edge, texture, shape, etc. of the detection target in the initial image, and these features may include the amplitude and phase information of the frequency components. In addition, the specific extraction method of the target key feature extraction may be a basic threshold judgment, the selection of different frequency components, etc., or the application of machine learning and deep learning technology to automatically identify the most representative frequency domain features. No specific limitation is made.

[0112] It should be noted that in the frequency domain, human key points (such as joints, facial features, etc.) correspond to some specific frequency components. By analyzing the energy distribution in the frequency domain, the frequency components related to the human key points can be identified. Therefore, the method of extracting target key features from image frequency domain features may include: calculating the energy within a specific frequency range in the frequency domain as key point related features. Filtering the frequency domain, retaining the frequency components related to the key points, and extracting features such as amplitude and phase. Using multi-resolution analysis methods such as wavelet transform, extract key point related features at different scales, and combine the extracted key point related features into a feature vector as a representation of the image / video frame, that is, obtaining the frequency domain target key features. These feature vectors can be used for subsequent tasks such as human posture estimation and action recognition.

[0113] In step S220 of some embodiments, since the subsequent information extraction still needs to use the image, it is necessary to further perform a frequency domain inverse transform on the frequency domain target key features based on the initial image, convert the frequency domain back to the spatial domain, and obtain an enhanced image after enhancing the initial image. Among them, the frequency domain inverse transform may include using an inverse Fourier transform (Inverse Fourier Transform) to reconstruct the frequency domain features into an enhanced image. In this way, the enhanced image will retain the key features of the detection target, while possibly reducing noise or irrelevant information, making the detection target more prominent in the enhanced image.

[0114] In the above embodiment, the present application transforms the initial image in the frequency domain and the spatial domain, and extracts the frequency domain features related to the key points of the detection target in the frequency domain to enhance the initial image. This method combines the advantages of frequency domain analysis, can enhance the feature areas related to the key points, suppress irrelevant or interfering feature areas, effectively handle the influence of factors such as occlusion, illumination, and background interference, and improve the feature expression of the image.

[0115] In step S140 of some embodiments, after obtaining the enhanced image, in order to accurately detect the specific key points of each detection target in the enhanced image and accurately locate the position of the detection target, the present application may input the enhanced image into a pre-trained multi-target detection model. The multi-target detection model of the present application may be constructed based on an attention mechanism, multi-scale feature fusion, and key point recognition grouping, and may specifically include a multi-scale feature fusion sub-model, a key point detection sub-model, and a key point grouping sub-model to improve the accuracy and robustness of key point detection.

[0116] In step S150 of some embodiments, after inputting the multi-target detection model, the enhanced image can be first subjected to multi-scale feature fusion processing based on the multi-scale feature fusion sub-model, and feature maps of different resolutions can be obtained using convolution kernels of different sizes or downsampling rates, and feature map fusion is performed to obtain a multi-scale fusion feature map. The multi-scale feature fusion sub-model of the present application may include a multi-scale feature extraction layer, a self-attention layer, a cross-attention layer, and a feature fusion layer. In other words, the multi-scale feature fusion sub-model constructed in the present application combines the attention mechanism and multi-scale feature fusion to improve the performance of key point detection, can effectively utilize the contextual information in the multi-target scene, enhance the feature areas related to the key points, and integrate feature information of different scales and different domains to improve the quality and efficiency of feature expression.

[0117] It should be noted that the present application integrates feature maps of different scales by stacking feature maps to form a comprehensive feature representation, which not only contains the detailed information of the image, but also combines high-level semantic features, which can enhance the ability of the network in processing multi-scale targets and enable it to better capture different levels of information in the image.

[0118] See also Figure 3 , Figure 3 is an optional flowchart of step S150 provided in an embodiment of the present application. In some embodiments of the present application, step S150 may specifically include but is not limited to steps S310 to S370. Figure 3 These seven steps are introduced in detail.

[0119] Step S310, performing multi-scale feature extraction on the enhanced image based on the multi-scale feature extraction layer to obtain a plurality of enhanced feature maps with different resolutions;

[0120] Step S320, connecting the multiple enhanced feature maps of different resolutions based on a preset pyramid structure to obtain a target pyramid structure;

[0121] Step S330, performing self-attention processing on the candidate feature map based on the self-attention layer to obtain a self-attention weight;

[0122] Step S340, adjusting the feature value of the candidate feature map based on the self-attention weight to obtain an adjusted feature map;

[0123] Step S350, performing cross attention processing on the adjusted feature map and the adjacent feature maps of the candidate feature map based on the cross attention layer to obtain a cross attention weight;

[0124] Step S360, adjusting the feature value of the adjusted feature map based on the cross attention weight to obtain a target feature map;

[0125] Step S370, performing feature fusion on the target feature map in the target pyramid structure based on the feature fusion layer to obtain a multi-scale fused feature map.

[0126] In step S310 of some embodiments, in the multi-scale feature fusion sub-model, a rich feature representation can be first extracted from the input enhanced image based on the multi-scale feature extraction layer, that is, the multi-scale feature extraction layer can be constructed by using a hierarchical structure with different sizes of convolution kernels or downsampling rates, and by considering information at different scales, multiple enhanced feature maps of different resolutions are obtained, with the goal of capturing details and global information in the image.

[0127] In step S320 of some embodiments, further, the obtained multiple enhanced feature maps of different resolutions can be connected or stacked according to a preset pyramid structure to obtain a target pyramid structure. The target pyramid structure includes a candidate feature map and adjacent feature maps of the candidate feature map. In other words, feature maps of different scales can be effectively managed by constructing a pyramid structure. The preset pyramid structure refers to a pyramid structure pre-constructed based on actual needs, which can include feature map size requirements corresponding to each level, so that the model can effectively utilize feature information of different resolutions in subsequent processing stages.

[0128] In step S330 of some embodiments, the stacked feature maps may be further processed by an attention mechanism, and specifically, the similarity between the positions within the candidate feature maps may be calculated by a self-attention layer, that is, a self-attention layer is used to calculate the correlation between each pixel and other pixels in each feature map, and a corresponding self-attention weight is generated. In this way, the model can be helped to automatically identify and emphasize important features or areas in the image, rather than uniformly focusing on all positions.

[0129] In step S340 of some embodiments, the calculated self-attention weight is further used to perform weighted adjustment on the feature values ​​in the candidate feature map, that is, a weighted summation method can be adopted according to the obtained self-attention weight. At this time, features with high weights will be enhanced, and features with low weights may be weakened, thereby obtaining a new feature map for subsequent processing.

[0130] In step S350 of some embodiments, further, the adjusted feature map is processed with its adjacent candidate feature maps based on the cross attention layer, that is, a cross attention layer is used to calculate the correlation between each pixel and other pixels between different feature maps, and the feature value of each pixel is adjusted according to the correlation weight (i.e., the cross attention weight). The cross attention weight can reflect the relative importance between feature maps. The embodiments of the present application can establish effective connections between different feature maps through the cross attention layer to promote the interaction of information.

[0131] In step S360 of some embodiments, the adjusted feature map is further adjusted using the generated cross-attention weights, and the introduction of the cross-attention weights helps to fuse information from different feature maps to form a more accurate and comprehensive feature expression to obtain the final target feature map.

[0132] In step S370 of some embodiments, further, in the feature fusion layer, all target feature maps in the target pyramid structure are integrated. At this time, the feature fusion composition can specifically adopt addition, splicing or other fusion strategies to finally generate a multi-scale fusion feature map. This image contains rich multi-scale information and will provide better support for subsequent tasks. In other words, upsampling processing can be used to upsample the low-resolution target feature map to high resolution, and weighted sum the feature maps of different resolutions according to certain weights to obtain a final multi-scale fusion feature map. Among them, the target feature map obtained in the present application can include 3 to 5 layers, and the resolution size ratio of adjacent layers is 2. The specific value is related to the training data. It needs to be upsampled to the same resolution before fusion, generally upsampled to the maximum resolution value.

[0133] In the above embodiment, the self-attention layer and the cross-attention layer have only different inputs, and the principles are the same. The self-attention layer is used to process a single feature map, and the output is the weight of the feature map, and the matrix size of the weight is the same as the feature map, and they can be added. The input of the cross-attention layer is multiple feature maps, and the sizes of different feature maps can be adjusted in the intermediate calculation process. The present application can be based on a multi-scale feature extraction layer, a self-attention layer, a cross-attention layer and a feature fusion layer to form a complex and efficient feature extraction and processing process. Through multi-scale analysis, self-attention and cross-attention mechanisms and feature fusion, a better multi-scale fusion feature map is finally obtained, which can better improve the accuracy of image-based multi-target detection.

[0134] In step S160 of some embodiments, after obtaining the multi-scale fusion feature map, key point detection can be performed on the multi-scale fusion feature map to identify important points in the image (e.g., corners and edges of objects). These key points are essential for subsequent target recognition and grouping. The key point detection data includes the predicted key point category and key point position data of the candidate key point. The candidate key point is used to indicate a key point (i.e., pixel point) detected from the enhanced image and belonging to the detection target. The predicted key point category is used to characterize the key point category to which the corresponding candidate key point belongs. The key point position data is used to characterize the pixel point position of the corresponding candidate key point in the enhanced image, which can be the pixel coordinate.

[0135] See also Figure 4 , Figure 4is an optional flowchart of step S160 provided in an embodiment of the present application. In some embodiments of the present application, step S160 may specifically include but is not limited to steps S410 to S440. Figure 4 These four steps are introduced in detail.

[0136] Step S410, performing category prediction on the fused feature points in the multi-scale fused feature map to obtain a probability prediction map of a preset number of categories;

[0137] Step S420, determining candidate key points from the fused feature points based on the peak extraction algorithm and the predicted probability value, and using the preset category as the predicted key point category of the candidate key points;

[0138] Step S430, extracting position data of candidate key points based on the multi-scale fusion feature map to obtain key point position data;

[0139] Step S440, determining key point detection data based on the predicted key point category and key point position data.

[0140] In step S410 of some embodiments, the multi-scale fusion feature map includes multiple fusion feature points (i.e., feature pixel points), and for a preset category, the corresponding probability prediction map (equivalent to a matrix) includes a prediction probability value, which is used to indicate the probability that the fusion feature point belongs to the preset category. In other words, through category prediction, the probability of each pixel in the multi-scale fusion feature map belonging to which key point category can be predicted. Among them, the probability prediction map has the same size as the input feature map, and the category prediction can be specifically performed through a convolutional neural network, a deep learning network, etc.

[0141] It should be noted that the predicted key point categories of the present application can be determined by the training sample set. If the detection target is a human body, the key point categories may include categories corresponding to the five senses (such as the left eye, mouth, etc.), various body joints (such as the left shoulder, right shoulder, etc.). For example, assuming that the key point categories can include 17 categories, the category prediction of the fused feature points in the multi-scale fusion feature map can be performed to obtain 17 probability prediction maps. Moreover, each numerical value in the probability prediction map represents the probability that the corresponding fused feature point (i.e., pixel) in the multi-scale fusion feature map belongs to a certain key point category.

[0142] In step S420 of some embodiments, key points with high confidence can be further identified from the probability prediction map. Specifically, a peak extraction algorithm (such as non-maximum suppression) can be applied to identify local maxima in the prediction probability map, which can represent candidate key points that are more representative of the detection target, and the preset category is used as the predicted key point category of the candidate key point.

[0143] In step S430 of some embodiments, further, by traversing the coordinates of the candidate key points, the corresponding position information (such as coordinates and possible offsets) is extracted from the multi-scale fusion feature map (equivalent to the enhanced image), and the key point position data of each candidate key point is obtained. In addition, the accuracy of the coordinates can be improved by interpolation methods, such as using the values ​​of adjacent feature points to optimize the position data to ensure the accuracy of the position extraction of the candidate key points.

[0144] Process: In step S440 of some embodiments, further, the key point detection data of each candidate key point can be obtained through the predicted key point category and key point position data corresponding to the candidate key point, which is convenient for subsequent key point grouping and identification.

[0145] In step S170 of some embodiments, after determining the relevant data of multiple candidate key points related to the detection target in the enhanced image, the distance similarity of the detected candidate key points can be further calculated in relative position, that is, key points that may belong to the same detection target can be identified from the perspective of spatial distance.

[0146] It should be noted that by calculating the similarity between probability maps, we can further determine which key points may belong to the same person. Common similarity metrics include cosine similarity, correlation coefficient, etc. These metrics can help distinguish which key points belong to the same category or the same person semantically.

[0147] In step S180 of some embodiments, at the same time, the present application may consider the category of each candidate key point and the distance between them, that is, the correlation between the candidate key points can be enhanced through similar categories and similar distances, that is, the relationship between these candidate key points and the detection target can be better understood from the perspective of semantic similarity.

[0148] See also Figure 5 , Figure 5 is an optional flow chart of step S180 provided in an embodiment of the present application. In some embodiments of the present application, step S180 may specifically include but is not limited to steps S510 to S570. Figure 5 These seven steps are introduced in detail.

[0149] Step S510, selecting a first key point and a second key point from candidate key points;

[0150] Step S520, performing category similarity calculation on the first predicted key point category of the first key point and the second predicted key point category of the second key point to obtain category similarity;

[0151] Step S530, obtaining a first category description text of a first predicted key point category and a second category description text of a second predicted key point category;

[0152] Step S540, calculating text similarity between the first category description text and the second category description text to obtain text similarity;

[0153] Step S550, performing a difference calculation between the key point distance similarity and a preset distance similarity threshold to obtain a distance similarity difference;

[0154] Step S560, determining a first weight of category similarity and a second weight of text similarity based on the distance similarity difference;

[0155] Step S570, performing weighted sum calculation based on the category similarity, the first weight, the text similarity and the second weight to obtain the semantic similarity.

[0156] In step S510 of some embodiments, when calculating the semantic similarity between any two candidate key points, two key points may be selected from a plurality of candidate key points for analysis.

[0157] It should be noted that the present application may also select the first key point and the second key point from the candidate key point list according to certain standards (such as confidence, distance, etc.).

[0158] In step S520 of some embodiments, in order to quantify the similarity between the categories predicted by the first key point and the second key point, the present application may use an appropriate similarity metric (such as cosine similarity, Jaccard similarity, etc.) to compare the two categories and calculate the category similarity. This makes it possible to evaluate the relevance of the two key points at the category level.

[0159] In step S530 of some embodiments, in order to further classify the categories to determine whether the corresponding first key point and second key point belong to the same detection target, the text description of each predicted key point category in the enhanced image can be obtained respectively. These texts may contain the attributes, feature words or other relevant information of the category. For example, if the predicted key point category is eyes, the first category description text obtained may be the shape, size, color of the surrounding area, etc. of the eyes to distinguish different detection targets.

[0160] In step S540 of some embodiments, in order to evaluate the similarity between category description texts, natural language processing technology (such as cosine similarity, TF-IDF or text embedding) can be used to analyze and calculate the two category description texts, calculate the text similarity, and obtain a deeper semantic relationship between the categories.

[0161] In step S550 of some embodiments, in order to evaluate the impact of the spatial relationship between the selected first key point and the second key point on the semantic similarity, the present application may first calculate the spatial distance between the first key point and the second key point, that is, the key point distance similarity, and perform a difference calculation with the set distance threshold to obtain a distance similarity difference to reflect the impact of the spatial configuration of the selected key point on its similarity.

[0162] In step S560 of some embodiments, further, the weights of category similarity and text similarity can be dynamically adjusted according to the size of the distance similarity difference. For example, the smaller the distance similarity difference, the greater the possibility that the two key points belong to different categories. In this case, the first weight of category similarity can be increased, and the larger the distance similarity difference, the greater the possibility that the two key points belong to different detection targets. In this case, the second weight of text similarity can be increased to ensure that the weighted sum calculation is more reasonable.

[0163] In step S570 of some embodiments, the calculated category similarity and text similarity are further multiplied by corresponding weights, and then added to form a comprehensive semantic similarity score. This score can be used for subsequent similarity evaluation or decision-making to help understand the relevance between key points.

[0164] In step S190 of some embodiments, further, the key point grouping submodel can be used to group the similarity information obtained above for key points, so as to provide a more clear output for target detection, and form a target key point group corresponding to each detection target. Wherein, the key points contained in each target key point group are used to indicate the same detection target. Specifically, a weighted sum method can be used to obtain a comprehensive scoring matrix in combination with key point semantic similarity and key point distance similarity. In this step, different weights can be assigned to each type of similarity according to the importance of different similarities. Further, a clustering algorithm (such as K-means, DBSCAN or hierarchical clustering) is used to analyze the comprehensive scoring matrix to identify key point groups with high similarity. The results of these clusters will identify a group of key points, which are evaluated as possibly belonging to the same detection target.

[0165] It should be noted that after determining the target key point group corresponding to the same detection target, the rectangular frame information of the detection target in the initial image can be determined based on the target key point group to locate the detection target. In addition, a unique identifier can be assigned to each detection target for easy user identification.

[0166] See also Figure 6 , Figure 6is another optional flow chart of the image-based multi-target detection method provided by the embodiment of the present application. In some embodiments of the present application, before step S140, the multi-target detection method of the present application may also include: constructing a multi-target detection model, which may specifically include but is not limited to steps S610 to S690. Figure 6 These nine steps are introduced in detail.

[0167] Step S610, obtaining training samples;

[0168] Step S620, inputting the sample image into the initial detection model;

[0169] Step S630, performing multi-scale feature fusion processing on the sample image based on the multi-scale feature fusion sub-model to obtain a sample multi-scale fusion feature map;

[0170] Step S640, performing key point detection on the sample multi-scale fusion feature map based on the key point detection sub-model to obtain sample key point detection data;

[0171] Step S650, calculating the semantic similarity of the sample candidate key points based on the sample predicted key point category and the sample key point position data to obtain the sample key point semantic similarity;

[0172] Step S660, calculating the distance similarity of the sample candidate key points based on the sample key point position data to obtain the sample key point distance similarity;

[0173] Step S670, grouping the sample key point semantic similarity, the sample key point distance similarity and the sample candidate key point based on the key point grouping sub-model to obtain a predicted target key point group;

[0174] Step S680, calculating the loss value based on the sample image, the number of sample detection targets, the sample target key point group and the predicted target key point group to obtain a model loss value;

[0175] Step S690, adjusting the parameters of the initial detection model based on the model loss value to obtain a multi-target detection model.

[0176] In steps S610 to S670 of some embodiments, the training sample includes a sample image and a sample target key point group, and the sample image includes multiple sample detection targets. The initial detection model includes a multi-scale feature fusion sub-model, a key point detection sub-model and a key point grouping sub-model, and the sample key point detection data includes the sample predicted key point category and the sample key point position data of the sample candidate key point, and the sample candidate key point is used to indicate a key point of the sample detection target. The sample predicted key point category is equivalent to a category label, and the sample target key point group is equivalent to a combination of key points belonging to the same detection target that have been marked in advance from the sample image. Among them, the specific processing process of steps S620 to S670 can refer to steps S140 to S190 of the above embodiment, that is, the data therein is replaced with sample data, and in order to save space, it will not be repeated here. The key points contained in the predicted target key point group are used to indicate the same sample detection target.

[0177] It should be noted that the specific categories of the sample detection targets of this application can be set according to the actual task requirements, so that a training sample set can be constructed based on different categories of continuous detection tasks, and multi-target detection models for different scenarios can be trained based on different categories of training samples. For example, for vehicle detection tasks, it is necessary to detect vehicle targets. Therefore, it is necessary to distinguish different vehicle categories. For example, the categories can be set to include 5 categories: cars, trucks, buses, motorcycles, and bicycles, and corresponding category labels are set for each vehicle category. For another example, for the task of human-vehicle motion trajectory, it is necessary to detect both humans and vehicles at the same time. At this time, there is no need to subdivide the vehicle categories. The categories refer to humans and vehicles, and corresponding category labels are set for the two categories of humans and vehicles.

[0178] It should be noted that in order to improve the recognition accuracy of the category targets that need to be detected, when constructing a training sample set, the embodiments of the present application can perform at least one of multi-scale transformation processing, image random cropping and splicing processing, image fusion processing, etc. on the collected sample images to obtain new sample images, thereby expanding the number of training samples.

[0179] In step S680 of some embodiments, further, a loss value calculation can be performed based on the sample image, the number of sample detection targets, the sample target key point group and the predicted target key point group to obtain a model loss value to train the initial detection model.

[0180] See also Figure 7 , Figure 7 is an optional flow chart of step S680 provided in an embodiment of the present application. In some embodiments of the present application, step S680 may specifically include but is not limited to steps S710 to S750. Figure 7 These five steps are introduced in detail.

[0181] Step S710, extracting sample target key points and first key point position data of the sample target key points from the sample target key point group;

[0182] Step S720, extracting the predicted target key point and the second key point position data of the predicted target key point from the predicted target key point group;

[0183] Step S730, performing loss calculation based on the first key point position data and the second key point position data of the same sample detection target to obtain a predicted loss value;

[0184] Step S740, performing key point conflict calculation based on the first key point position data and the second key point position data of any two different sample detection targets to obtain a key point conflict value;

[0185] Step S750, performing model loss calculation based on the predicted loss value and the key point conflict value to obtain a model loss value.

[0186] In steps S710 to S730 of some embodiments, the present application may first extract the sample target key point and the first key point position data of the sample target key point from the sample target key point group, and extract the predicted target key point and the second key point position data of the predicted target key point from the predicted target key point group. Further, the loss calculation may be performed based on the first key point position data and the second key point position data of the same sample detection target to obtain the predicted loss value, that is, the difference between the key point position predicted by the calculation model and the actual sample key point position.

[0187] It should be noted that the loss function used in this application to calculate the predicted loss value can be any one of the 0-1 loss function, absolute value loss function, logarithmic loss function, exponential loss function, Hinge loss function, etc., which can be selected and adjusted according to actual needs, and will not be repeated here.

[0188] In step S740 of some embodiments, the present application further adds a mutual exclusion loss based on the original loss function to measure whether the key points predicted between different targets overlap or conflict. Specifically, the key point conflict calculation is performed based on the first key point position data and the second key point position data of any two different sample detection targets to obtain the key point conflict value. Among them, the key point conflict value is also equivalent to a penalty term, and the specific calculation process can be seen in the following formula 1:

[0189]

[0190] In formula 1, L excrepresents the key point conflict value, K represents the number of sample detection targets, N represents the number of key point categories and, τ represents the preset conflict threshold, represents the position of the i-th predicted target key point predicted in the k-th sample detection target, Represents the position of the i-th predicted target key point predicted in the l-th sample detection target.

[0191] In step S750 of some embodiments, further, a model loss calculation is performed based on the predicted loss value and the key point conflict value to obtain a model loss value. In addition, the specific formula for calculating the model loss value can be shown in the following formula 2:

[0192] L=α·L 0 +β·L exc (Formula 2)

[0193] In formula 2, L exc represents the key point conflict value, L represents the model loss value, and L 0 Represents the prediction loss value, α represents the weight corresponding to the prediction loss value, and β represents the weight corresponding to the key point conflict value.

[0194] In step S690 of some embodiments, further, the parameters of the initial detection model can be adjusted based on the model loss value until the preset training end condition is reached to obtain a multi-target detection model. The model structure of the multi-target detection model is the same as the model structure of the initial detection model. The preset end condition can be that the target detection accuracy reaches a preset accuracy threshold, or the number of model iterations reaches a preset number of training times, etc., which will not be repeated here.

[0195] It should be noted that the present application can also perform key point trajectory detection on the key points in the grouped target key point group, use a key point trajectory detection module to connect the key points of the same detection target according to the position changes and identifier matching between the key points in adjacent frames, and use a Kalman filter to smooth and predict the position of each target key point to achieve trajectory detection of multiple target key points.

[0196] It should be noted that the non-Company software tools or components appearing in the embodiments of the present application are merely examples and do not represent actual use.

[0197] The image-based multi-target detection method provided by the embodiment of the present application can realize the detection of multiple target key points from a monocular image or video without the use of additional sensors or equipment, thereby reducing the cost and complexity of the system. Compared with the related art, which does not deeply understand the association information of other target information in the image and the target to be identified, and is easily affected by problems such as occlusion, background interference and illumination, the present application enhances the image of the initial image through the image frequency domain features after frequency domain transformation, and combines multi-scale feature fusion and key point detection to enhance the feature information related to the target key points, which can deeply understand the association information of other target information in the image and the target to be identified, improve the accuracy of feature expression and key point detection, and realize more accurate key point grouping based on key point distance similarity and key point semantic similarity. Therefore, the embodiment of the present application can effectively improve the accuracy of image-based multi-target detection.

[0198] See also Figure 8 The embodiment of the present application further provides an image-based multi-target detection device, which can implement the above-mentioned image-based multi-target detection method, and the device includes:

[0199] An acquisition module 810 is used to acquire an initial image, where the initial image includes a plurality of detection targets;

[0200] The transformation module 820 is used to perform frequency domain transformation on the initial image to obtain image frequency domain features;

[0201] An enhancement module 830 is used to enhance the initial image based on the image frequency domain features to obtain an enhanced image;

[0202] An input module 840 is used to input the enhanced sample image into a pre-trained multi-target detection model, where the multi-target detection model includes a multi-scale feature fusion sub-model, a key point detection sub-model, and a key point grouping sub-model;

[0203] A fusion module 850 is used to perform multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map;

[0204] A detection module 860 is used to perform key point detection on the multi-scale fusion feature map based on the key point detection sub-model to obtain key point detection data, wherein the key point detection data includes a predicted key point category and key point position data of a candidate key point, and the candidate key point is used to indicate a key point of a detection target;

[0205] A first calculation module 870 is used to calculate the distance similarity of the candidate key points based on the key point position data to obtain the key point distance similarity;

[0206] A second calculation module 880 is used to calculate the semantic similarity of the candidate key points based on the predicted key point categories and the key point distance similarity to obtain the key point semantic similarity;

[0207] The grouping module 890 is used to group key points based on the key point semantic similarity, key point distance similarity and candidate key points to obtain a target key point group. The key points included in the target key point group are used to indicate the same detection target.

[0208] The specific implementation of the image-based multi-target detection device in the embodiment of the present application is basically the same as the specific implementation of the above-mentioned image-based multi-target detection method, and will not be repeated here.

[0209] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image-based multi-target detection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0210] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0211] The processor 910 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0212] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the image-based multi-target detection method of the embodiment of this application;

[0213] Input / output interface 930, used to implement information input and output;

[0214] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0215] bus 950 , which transmits information between the various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0216] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0217] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image-based multi-target detection method is implemented.

[0218] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0219] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0220] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0221] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0222] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0223] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0224] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0225] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0226] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0227] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0228] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0229] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A multi-target detection method based on an image, characterized in that: The method comprises: Acquire an initial image, wherein the initial image includes a plurality of detection targets; Performing frequency domain transformation on the initial image to obtain image frequency domain features; Performing image enhancement on the initial image based on the image frequency domain features to obtain an enhanced image; Inputting the enhanced sample image into a pre-trained multi-target detection model, wherein the multi-target detection model includes a multi-scale feature fusion sub-model, a key point detection sub-model, and a key point grouping sub-model; Performing multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map; Perform key point detection on the multi-scale fusion feature map based on the key point detection sub-model to obtain key point detection data, wherein the key point detection data includes predicted key point categories and key point position data of candidate key points, and the candidate key points are used to indicate a key point of the detection target; Calculate the distance similarity of the candidate key points based on the key point position data to obtain the key point distance similarity; Calculating semantic similarity of the candidate key points based on the predicted key point categories and the key point distance similarity to obtain key point semantic similarity; The key point semantic similarity, the key point distance similarity and the candidate key points are grouped based on the key point grouping sub-model to obtain a target key point group, wherein the key points included in the target key point group are used to indicate the same detection target.

2. The method according to claim 1, characterized in that The performing image enhancement on the initial image based on the image frequency domain feature to obtain an enhanced image includes: Extracting target key features from the image frequency domain features to obtain frequency domain target key features; Based on the initial image, the frequency domain target key features are inversely transformed in the frequency domain to obtain the enhanced image.

3. The method according to claim 1, characterized in that The performing key point detection on the multi-scale fusion feature map based on the key point detection sub-model to obtain key point detection data includes: Performing category prediction on the fused feature points in the multi-scale fused feature map to obtain a probability prediction map of a preset number of categories, wherein for a preset category, the corresponding probability prediction map includes a prediction probability value, and the prediction probability value is used to indicate the probability that the fused feature point belongs to the preset category; Determine the candidate key point from the fused feature points based on a peak extraction algorithm and the predicted probability value, and use the preset category as the predicted key point category of the candidate key point; Extracting position data of the candidate key points based on the multi-scale fusion feature map to obtain the key point position data; The key point detection data is determined based on the predicted key point category and the key point position data.

4. The method according to claim 1, characterized in that: The multi-scale feature fusion sub-model includes a multi-scale feature extraction layer, a self-attention layer, a cross-attention layer and a feature fusion layer; The performing multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map includes: Performing multi-scale feature extraction on the enhanced image based on a multi-scale feature extraction layer to obtain a plurality of enhanced feature maps with different resolutions; Based on a preset pyramid structure, feature maps of the enhanced feature maps of multiple different resolutions are connected to obtain a target pyramid structure, wherein the target pyramid structure includes a candidate feature map and adjacent feature maps of the candidate feature map; Performing self-attention processing on the candidate feature map based on the self-attention layer to obtain a self-attention weight; Adjusting the feature value of the candidate feature map based on the self-attention weight to obtain an adjusted feature map; Performing cross attention processing on the adjusted feature map and the adjacent feature map of the candidate feature map based on the cross attention layer to obtain a cross attention weight; Adjusting the feature value of the adjusted feature map based on the cross attention weight to obtain a target feature map; Based on the feature fusion layer, feature fusion is performed on the target feature map in the target pyramid structure to obtain the multi-scale fused feature map.

5. The method according to claim 1, characterized in that The calculating the semantic similarity of the candidate key points based on the predicted key point categories and the key point distance similarity to obtain the key point semantic similarity includes: Selecting a first key point and a second key point from the candidate key points; Calculating the category similarity of the first predicted key point category of the first key point and the second predicted key point category of the second key point to obtain the category similarity; Acquire a first category description text of the first predicted key point category and a second category description text of the second predicted key point category; Calculating text similarity between the first category description text and the second category description text to obtain text similarity; Calculating the difference between the key point distance similarity and a preset distance similarity threshold to obtain a distance similarity difference; Determine a first weight of the category similarity and a second weight of the text similarity based on the distance similarity difference; The semantic similarity is obtained by performing a weighted sum calculation based on the category similarity, the first weight, the text similarity and the second weight.

6. The method according to any one of claims 1 to 5, characterized in that: Before inputting the enhanced sample image into a pre-trained multi-target detection model, the method further includes: constructing the multi-target detection model, specifically including: Acquire a training sample, wherein the training sample includes a sample image and a sample target key point group, and the sample image includes a plurality of sample detection targets; Inputting the sample image into an initial detection model, wherein the initial detection model includes the multi-scale feature fusion sub-model, the key point detection sub-model and the key point grouping sub-model; Performing multi-scale feature fusion processing on the sample image based on the multi-scale feature fusion sub-model to obtain a sample multi-scale fusion feature map; Based on the key point detection sub-model, key point detection is performed on the sample multi-scale fusion feature map to obtain sample key point detection data, wherein the sample key point detection data includes sample predicted key point categories and sample key point position data of sample candidate key points, and the sample candidate key points are used to indicate a key point of the sample detection target; Calculating the semantic similarity of the sample candidate key points based on the sample prediction key point category and the sample key point position data to obtain the sample key point semantic similarity; Calculating the distance similarity of the candidate key points of the sample based on the position data of the key points of the sample to obtain the distance similarity of the key points of the sample; Based on the key point grouping sub-model, the sample key point semantic similarity, the sample key point distance similarity and the sample candidate key point are grouped into key points to obtain a predicted target key point group, wherein the key points included in the predicted target key point group are used to indicate the same sample detection target; Calculate the loss value based on the sample image, the number of the sample detection targets, the sample target key point group and the predicted target key point group to obtain a model loss value; The parameters of the initial detection model are adjusted based on the model loss value to obtain the multi-target detection model.

7. The method according to claim 6, characterized in that The calculating of the loss value based on the sample image, the number of the sample detection targets, the sample target key point group and the predicted target key point group to obtain the model loss value includes: Extracting sample target key points and first key point position data of the sample target key points from the sample target key point group; Extracting a predicted target key point and second key point position data of the predicted target key point from the predicted target key point group; Perform loss calculation based on the first key point position data and the second key point position data of the same sample detection target to obtain a predicted loss value; Perform key point conflict calculation based on the first key point position data and the second key point position data of any two different sample detection targets to obtain a key point conflict value; A model loss calculation is performed based on the predicted loss value and the key point conflict value to obtain the model loss value.

8. An image-based multi-target detection device, characterized in that: The device comprises: An acquisition module, used for acquiring an initial image, wherein the initial image includes a plurality of detection targets; A transformation module, used for performing frequency domain transformation on the initial image to obtain image frequency domain features; An enhancement module, used for performing image enhancement on the initial image based on the image frequency domain features to obtain an enhanced image; An input module, used for inputting the enhanced sample image into a pre-trained multi-target detection model, wherein the multi-target detection model includes a multi-scale feature fusion sub-model, a key point detection sub-model and a key point grouping sub-model; A fusion module, used for performing multi-scale feature fusion processing on the enhanced image based on the multi-scale feature fusion sub-model to obtain a multi-scale fusion feature map; A detection module, configured to perform key point detection on the multi-scale fusion feature map based on the key point detection sub-model to obtain key point detection data, wherein the key point detection data includes a predicted key point category and key point position data of a candidate key point, and the candidate key point is used to indicate a key point of the detection target; A first calculation module, configured to calculate distance similarity of the candidate key points based on the key point position data to obtain key point distance similarity; A second calculation module is used to calculate the semantic similarity of the candidate key points based on the predicted key point category and the key point distance similarity to obtain the key point semantic similarity; A grouping module is used to group the key points based on the key point semantic similarity, the key point distance similarity and the candidate key points to obtain a target key point group, wherein the key points included in the target key point group are used to indicate the same detection target.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images