Image detection method, image detection device and computer readable storage medium

By performing multi-level feature extraction and enhanced fusion on images, and combining the feature extraction and fusion units of the image detection network for parameter correction, the problem of low multi-task detection accuracy in the prior art is solved, and the accuracy of image detection is significantly improved.

CN113673578BActive Publication Date: 2024-11-19ZHEJIANG DAHUA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110851206.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-27
Publication Date
2024-11-19
Estimated Expiration
2041-07-27

AI Technical Summary

Technical Problem

In the existing image detection technology, multi-task detection methods lead to low detection accuracy because each task is independent and lacks effective feature fusion.

Method used

By using the image detection method, feature extraction of the image to be processed, feature maps at different levels are obtained, and fusion is performed, object information is determined by combining multiple feature maps, and parameter correction is performed using feature extraction units and feature fusion units in the image detection network. , improve detection accuracy.

Benefits of technology

Through the fusion of features at different levels, the accuracy of image detection is improved, the richness of feature information is enhanced, and the object information of the object to be processed can be determined more accurately.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113673578B_ABST
    Figure CN113673578B_ABST
Patent Text Reader

Abstract

The present application relates to the field of image detection technology, and discloses an image detection method, an image detection device, and a computer-readable storage medium. The method comprises: extracting features from an image to be processed containing an object to be processed to obtain a first feature map; extracting features of different levels from the first feature map to obtain multiple second feature maps corresponding to different levels; enhancing and fusing the multiple second feature maps to obtain a third feature map; and determining object information of the object to be processed based on the multiple second feature maps and the third feature map. In the above manner, the accuracy of image detection can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and in particular to an image detection method, an image detection device, and a computer-readable storage medium. Background Art

[0002] In image detection technology, there is a multi-task detection method, that is, the method can be used to complete multiple tasks at the same time.

[0003] The applicant has found through long-term research that although multiple tasks can be completed, each task is independent of each other. In essence, different branches are used to complete different tasks, which results in low detection accuracy. Summary of the invention

[0004] The main technical problem solved by the present application is to provide an image detection method, an image detection device and a computer-readable storage medium, which can improve the accuracy of image detection.

[0005] In order to solve the above problems, a technical solution adopted in the present application is to provide an image detection method, which includes: performing feature extraction on an image to be processed containing an object to be processed to obtain a first feature map; performing feature extraction at different levels on the first feature map to obtain multiple second feature maps corresponding to different levels; enhancing and fusing the multiple second feature maps to obtain a third feature map; and determining object information of the object to be processed based on the multiple second feature maps and the third feature map.

[0006] Among them, the multiple second feature maps include at least two of the following: a feature map used to characterize global information of the object to be processed; a feature map used to characterize component information of the object to be processed; and a feature map used to characterize key point information of the object to be processed.

[0007] The method comprises: performing feature extraction at different levels on the first feature map to obtain multiple second feature maps corresponding to different levels, including: inputting the first feature map into a trained image detection network; performing feature extraction on the first feature map based on multiple feature extraction units in the image detection network to obtain multiple second feature maps; and performing feature extraction at different levels on the first feature map by different feature extraction units.

[0008] Among them, the multiple feature extraction units include at least the following two: a first feature extraction unit including n deconvolution modules and n dilated convolutions; a second feature extraction unit including n / 2 deconvolution modules and 2n dilated convolutions; a third feature extraction unit including n / 4 deconvolution modules and 4n dilated convolutions; wherein n is a multiple of 4.

[0009] The method further includes: constructing an image detection network; wherein the image detection network includes at least feature extraction units and feature fusion units at different levels; inputting the first feature map into the image detection network to obtain second feature maps output by feature extraction units at different levels, and obtaining a third feature map output by the feature fusion unit; converting the second feature map into a first vector, and converting the third feature map into a second vector; determining a loss value based on the first vector and the second vector; and using the loss value to perform parameter correction on the image detection network.

[0010] Among them, feature extraction units at different levels include a first-level feature extraction unit, a second-level feature extraction unit and a third-level feature extraction unit; determining a loss value based on a first vector and a second vector includes: determining a first loss value according to a first vector converted from a second feature map output by the first-level feature extraction unit; determining a second loss value according to a first vector converted from a second feature map output by the second-level feature extraction unit; determining a third loss value according to a first vector converted from a second feature map output by the third-level feature extraction unit; determining a fourth loss value using the first loss value and the second loss value; determining a fifth loss value using the first loss value and the third loss value; determining a sixth loss value using the second loss value and the third loss value; determining a seventh loss value using the fourth loss value, the fifth loss value and the sixth loss value; and using the loss value to perform parameter correction on the image detection network, including: performing parameter correction on the image detection network using the first loss value, the second loss value, the third loss value and the seventh loss value.

[0011] The first loss value is determined according to the first vector converted from the second feature map output by the first-level feature extraction unit, including: using the prediction information represented by the first vector converted from the second feature map output by the first-level feature extraction unit and the real information corresponding to the first-level feature extraction unit to determine the first loss value;

[0012] Determining a second loss value according to a first vector converted from a second feature map output by a second-level feature extraction unit includes:

[0013] Determine a second loss value by using the prediction information represented by the first vector converted from the second feature map output by the second-level feature extraction unit and the real information corresponding to the second-level feature extraction unit;

[0014] The third loss value is determined according to the first vector converted from the second feature map output by the third-level feature extraction unit, including: determining the third loss value using the predicted information represented by the first vector converted from the second feature map output by the third-level feature extraction unit and the real information corresponding to the third-level feature extraction unit.

[0015] Among them, the seventh loss value is determined using the fourth loss value, the fifth loss value and the sixth loss value, including: weighting the fourth loss value using the first control parameter; weighting the fifth loss value using the second control parameter; weighting the sixth loss value using the third control parameter; and summing the weighted fourth loss value, fifth loss value and sixth loss value to obtain the seventh loss value.

[0016] In order to solve the above-mentioned problem, another technical solution adopted in the present application is to provide an image detection device, which includes a processor and a memory coupled to the processor; wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the method provided by the above-mentioned technical solution.

[0017] In order to solve the above problem, another technical solution adopted by the present application is to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the method provided by the above technical solution.

[0018] The beneficial effects of the present application are: different from the prior art, the image detection method, image detection device and computer-readable storage medium provided by the present application. The method includes: extracting features from an image to be processed containing an object to be processed to obtain a first feature map; extracting features of different levels from the first feature map to obtain multiple second feature maps corresponding to different levels; enhancing and fusing the multiple second feature maps to obtain a third feature map; and determining the object information of the object to be processed based on the multiple second feature maps and the third feature map. In the above manner, by fusing features of different levels, the fused features can have more feature information, thereby using the enhanced fused features and the previous features to determine the object information of the object to be processed, which can improve the accuracy of image detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0020] Figure 1 It is a flowchart of an embodiment of an image detection method provided by the present application;

[0021] Figure 2 is a flow chart of another embodiment of the image detection method provided by the present application;

[0022] Figure 3is a flow chart of another embodiment of the image detection method provided by the present application;

[0023] Figure 4 is a flow chart of another embodiment of the image detection method provided by the present application;

[0024] Figure 5 is a flowchart of an embodiment of step 44 provided by the present application;

[0025] Figure 6 is a flowchart of an embodiment of step 447 provided by the present application;

[0026] Figure 7 is a flow chart of another embodiment of the image detection method provided by the present application;

[0027] Figure 8-Figure 11 It is a schematic diagram of an application scenario of the image detection method provided in this application;

[0028] Fig.12 It is a structural schematic diagram of an embodiment of an image detection device provided by the present application;

[0029] Fig.13 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0031] The terms "first", "second", etc. in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0032] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0033] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of an image detection method provided by the present application. The method comprises:

[0034] Step 11: extract features from the image to be processed that contains the object to be processed to obtain a first feature map.

[0035] In some embodiments, the image to be processed may be acquired in real time by an image acquisition device. The image to be processed may also be acquired by the image acquisition device according to an acquisition rule, such as acquisition at a fixed time interval. The image to be processed may also be acquired in advance by the image acquisition device.

[0036] For example, the image acquisition device is set in a traffic scene, and the image to be processed will contain a series of traffic-related contents such as lane lines, zebra crossings, vehicles and pedestrians.

[0037] For another example, the image acquisition device is set in a target detection scene, and the image to be processed will contain a corresponding detection target or only background information. For example, the detection target may be an animal, a person, or an object.

[0038] As an embodiment, in the process of obtaining the first feature map, it is necessary to perform multiple convolution, pooling, correction and other operations on the image to be processed to obtain the first feature map corresponding to the image to be processed.

[0039] It can be understood that step 11 is a downsampling process.

[0040] In some embodiments, the feature information in the first feature map may represent basic features of an object in the image to be processed. For example, if the image to be processed includes a human body, these basic features may be basic information such as the texture and color of the human body. For example, if the image to be processed includes an object, these basic features may be basic information such as the texture and color of the object.

[0041] Step 12: Perform feature extraction at different levels on the first feature map to obtain multiple second feature maps corresponding to different levels.

[0042] In step 12, feature extraction units of different scales may be used to perform feature extraction on the first feature map to obtain at least two second feature maps corresponding to different levels.

[0043] Take the human body image as an example to illustrate:

[0044] Different levels of feature extraction can be performed according to the human body structure, for example, the human body contour is taken as the first level, the human body limbs are taken as the second level, and the human body key points are taken as the third level.

[0045] Feature extraction units with different structures are used to extract features. For example, human key point detection technology is used to accurately estimate n main key points of the human body in the image to be processed, which may include: left and right elbows, left and right wrists, left and right shoulders, head, neck, left and right ankles, left and right knees, left and right hips and soles of the feet. Human key point detection technology can be used to judge the state of the human body, the posture of the human body, etc.

[0046] Human body segmentation technology is used to accurately estimate the human body area and background area in the image to be processed, and detect the correct human body area. Human body segmentation technology can be applied to human behavior analysis, gait recognition, etc.

[0047] The human body parts analysis technology is used to accurately estimate the various limb parts of the human body in the image to be processed, including the head, left upper arm, left forearm, right upper arm, right forearm, upper body trunk, left thigh, left calf, right thigh, right calf and other major limb parts of the human body. The human body parts analysis technology can be applied to human behavior analysis, behavior recognition, etc.

[0048] Similarly, other objects can also be hierarchical in the above manner. For example, for animals, the animal outline can be used as the first level, the animal's limbs can be used as the second level, and the animal's key points can be used as the third level.

[0049] It can be understood that the above-mentioned feature extraction at different levels is performed simultaneously, which can reduce the time consumed in the processing process and improve the processing efficiency.

[0050] Step 13: Enhance and fuse multiple second feature maps to obtain a third feature map.

[0051] In step 13, the second feature maps at different levels may be mapped to form a third feature map based on the fusion of the second feature maps at different levels. The correspondence between features at different levels may be reflected in the third feature map, making the detection more accurate.

[0052] Step 14: Determine object information of the object to be processed according to the multiple second feature maps and the third feature maps.

[0053] At this time, the state of the object in the image to be processed can be determined according to the multiple second feature maps and the third feature map. Taking the human body as an example, whether there is human body data in the image to be processed can be given according to the second feature map and the third feature map. If there is, the behavior state of the human body in the image to be processed is determined. For example, it can be a standing state, a squatting state, a walking state, a running state, a falling state, etc.

[0054] In this embodiment, the network involved in feature extraction is pre-trained using corresponding object images.

[0055] In an application scenario, the networks involved in feature extraction may be the same image detection network or different image detection networks. For example, an image detection network is used to perform step 11, extract features from an input image to be processed containing an object to be processed, obtain a first feature map, and then input the first feature map into another image detection network. Step 12 may be the following process:

[0056] The first feature map is input into the trained image detection network; based on multiple feature extraction units in the image detection network, feature extraction is performed on the first feature map to obtain multiple second feature maps; wherein different feature extraction units perform feature extraction at different levels on the first feature map.

[0057] When the network involved in feature extraction is the same image detection network, the image to be processed containing the object to be processed is input into the image detection network, and the image detection network executes the above-mentioned process of step 11 to step 14.

[0058] In this embodiment, the second feature maps at different levels can complete different tasks at the same time, such as completing the target segmentation task, segmenting the background and the target object, completing the component segmentation of the target object, such as segmenting the components of the target object and marking them for display, and completing the key point detection of the target object.

[0059] In this embodiment, a first feature map is obtained by performing feature extraction on an image to be processed that includes an object to be processed; feature extraction at different levels is performed on the first feature map to obtain multiple second feature maps corresponding to different levels; the multiple second feature maps are enhanced and fused to obtain a third feature map; and according to the multiple second feature maps and the third feature map, object information of the object to be processed is determined, and features of different levels are fused so that the fused features have more feature information, thereby determining the object information of the object to be processed using the enhanced fused features and the previous features, thereby improving the accuracy of image detection.

[0060] See also Figure 2 , Figure 2 : is a flow chart of another embodiment of the image detection method provided by the present application. The method comprises:

[0061] Step 21: extract features from the image to be processed that contains the object to be processed to obtain a first feature map.

[0062] Step 21 has the same or similar technical solution as the above embodiment, and will not be described in detail here.

[0063] Step 22: Perform first-level feature extraction on the first feature map to obtain a first-level second feature map.

[0064] The second feature map of the first level is used to represent the global information of the object. Taking the human body as an example, the second feature map of the first level can represent the contour of the human body. Taking the object as an object as an example, the second feature map of the first level can represent the contour of the object, such as the contour of a chair or a car.

[0065] In this process, it is equivalent to using segmentation technology to accurately estimate the object area and background area in the image to be processed and detect the correct object area. Specifically, the target segmentation technology can be used to segment the pixels representing the object area and the background area in the first feature map.

[0066] Step 23: Perform second-level feature extraction on the first feature map to obtain a second feature map of the second level.

[0067] Among them, the second feature map of the second level is used to represent the component information of the object. Taking the human body as an example, the second feature map of the second level can represent the limbs of the human body, such as arms, legs, head, upper body, and lower body. Taking the object as an object as an example, the second feature map of the second level can represent the parts of the object. For example, if the object is a chair, the second feature map can represent the chair legs, chair back, etc.

[0068] In this process, it is equivalent to using component analysis technology to accurately estimate the various components of an object.

[0069] Step 24: Perform third-level feature extraction on the first feature map to obtain a second feature map of the third level.

[0070] Among them, the second feature map of the third level is used to characterize the key point information of the object. Taking the human body as an example, the second feature map of the third level can characterize the key points of the human body, such as left and right elbows, left and right wrists, left and right shoulders, head, neck, left and right ankles, left and right knees, left and right hips and soles of feet. Taking the object as an object as an example, the second feature map of the third level can characterize the key points of the object. For example, if the object is a chair, the second feature map can characterize the connection between the chair legs and the backrest, etc.

[0071] In this process, it is equivalent to using key point detection technology to accurately estimate the key points of an object.

[0072] Step 25: Enhance and fuse the second feature map of the first level, the second feature map of the second level, and the second feature map of the third level to obtain a third feature map.

[0073] Step 26: Determine object information in the image to be processed according to the multiple second feature maps and the third feature map.

[0074] Different detection tasks are completed according to the second feature maps at different levels, such as image background and object segmentation, object component segmentation, and object key point extraction.

[0075] By fusing features at different levels in the above manner, the fused features can have more feature information, thereby enhancing the fused features and previous features to determine the object information in the image to be processed, thereby improving the accuracy of image detection.

[0076] See also Figure 3 , Figure 3 : is a flow chart of another embodiment of the image detection method provided by the present application. The method comprises:

[0077] Step 31: extract features from the image to be processed that contains the object to be processed to obtain a first feature map.

[0078] Step 31 is the same or similar to the technical solution of any of the above embodiments and will not be repeated here.

[0079] In this embodiment, a feature extraction unit of a corresponding structure is used to extract features from the first feature map. For details, refer to steps 32 to 34.

[0080] Step 32: Use the first feature extraction unit to perform first-level feature extraction on the first feature map to obtain a first-level second feature map.

[0081] The second feature map of the first level is used to represent the global information of the object.

[0082] The first feature extraction unit includes n deconvolution modules and n dilated convolution modules.

[0083] The first feature map is a small-resolution image, which can be upsampled through n deconvolution modules and n dilated convolutions to restore it to a large-resolution image. The use of dilated convolution can increase the receptive field, facilitate the extraction of global information, and make the extracted features more accurate.

[0084] Step 33: Use the second feature extraction unit to perform second-level feature extraction on the first feature map to obtain a second-level second feature map.

[0085] The second feature map of the second level is used to characterize component information of the object.

[0086] The second feature extraction unit includes n / 2 deconvolution modules and 2n dilated convolutions.

[0087] The first feature map is a small-resolution image, which can be upsampled through n / 2 deconvolution modules and 2n dilated convolutions to restore it to a larger-resolution image. The use of dilated convolution can increase the receptive field, facilitate the extraction of local information, and make the extracted local features more accurate.

[0088] It can be understood that since step 34 is to extract local information, it is not necessary to have a higher resolution than the first feature extraction unit, so the number of deconvolution modules can be half of that of the first feature extraction unit.

[0089] Step 34: Use the third feature extraction unit to perform third-level feature extraction on the first feature map to obtain a third-level second feature map.

[0090] Among them, the second feature map of the third level is used to represent the key point information of the object.

[0091] The third feature extraction unit includes n / 4 deconvolution modules and 4n dilated convolutions, wherein n is a multiple of 4.

[0092] The first feature map is a small-resolution image, which can be upsampled by n / 4 deconvolution modules and 4n dilated convolutions to restore it to a large-resolution image relative to the first feature map. The use of dilated convolution can increase the receptive field, facilitate the extraction of local information, and make the extracted local features more accurate.

[0093] It can be understood that since step 34 is to extract the corresponding key point information and is more sensitive to local semantic information, it does not need to have a higher resolution than the first feature extraction unit, so the number of deconvolution modules can be one-fourth of the first feature extraction unit.

[0094] Step 35: Enhance and fuse the second feature map of the first level, the second feature map of the second level, and the second feature map of the third level to obtain a third feature map.

[0095] Step 36: Determine object information in the image to be processed according to the multiple second feature maps and the third feature map.

[0096] By fusing features at different levels in the above manner, the fused features can have more feature information, thereby utilizing the enhanced fused features and previous features to determine the object information of the object to be processed, thereby improving the accuracy of image detection.

[0097] See also Figure 4 , Figure 4 1 is a flow chart of another embodiment of the image detection method provided by the present application. This embodiment is used to introduce the training process of the image detection network involved in the above-mentioned image detection. The method includes:

[0098] Step 41: Build an image detection network.

[0099] Among them, the image detection network includes at least feature extraction units, feature fusion units, and fully connected units at different levels.

[0100] Among them, the common feature extraction unit is connected to the feature extraction units at different levels, and the output of the common feature extraction unit is used as the input of the feature extraction units at different levels. The output of the feature extraction units at different levels is used as the input of the feature fusion unit. And the feature extraction units at different levels are also connected to the fully connected unit, so that the fully connected unit predicts the output of the feature extraction unit at each level. The feature fusion unit is also connected to the fully connected unit, and the output of the feature fusion unit is used as the input of the fully connected unit.

[0101] Step 42: Input the first feature map into the image detection network to obtain the second feature maps output by the feature extraction units at different levels, and obtain the third feature map output by the feature fusion unit.

[0102] Step 43: Convert the second feature map into a first vector, and convert the third feature map into a second vector.

[0103] In some embodiments, before the training image is input into the image detection network, the training target on the training image needs to be annotated as real information. After annotating, the training image can be augmented, such as rotating and adjusting pixels, to expand the number of training images and improve the training quality.

[0104] The training image is then input into a common feature extraction unit of the image detection network. In the common feature extraction unit, features in the training image can be obtained by downsampling to form a first feature map. The common feature extraction unit can be constructed using multiple convolution modules and pooling modules.

[0105] Then the first feature map obtained at this time is used as the input of the feature extraction units at different levels, so that each feature extraction unit outputs the second feature map of the corresponding level, and the second feature map is represented by the first vector. Each second feature map is input into the feature fusion unit for coupling to obtain the third feature map, and the third feature map is represented by the second vector.

[0106] Step 44: Determine a loss value based on the first vector and the second vector.

[0107] At this time, each first vector is input to the fully connected layer for loss value calculation, and the second vector is input to the fully connected layer for loss value calculation.

[0108] In some embodiments, see Figure 5 , step 44 may be the following process:

[0109] Step 441: Determine a first loss value according to a first vector converted from a second feature map output by a first-level feature extraction unit.

[0110] Specifically, the first loss value is determined by using the prediction information represented by the first vector converted from the second feature map output by the first-level feature extraction unit and the real information corresponding to the first-level feature extraction unit.

[0111] The following formula can be used for calculation:

[0112]

[0113] Among them, Loss seg represents the first loss value, n represents the number of segmentation categories, P i Represents forecast information, Indicates true information.

[0114] Step 442: Determine a second loss value according to the first vector converted from the second feature map output by the second-level feature extraction unit.

[0115] Specifically, the second loss value is determined by utilizing the prediction information represented by the first vector converted from the second feature map output by the second-level feature extraction unit and the real information corresponding to the second-level feature extraction unit.

[0116] The following formula can be used for calculation:

[0117]

[0118] Among them, Loss parse represents the second loss value, n represents the number of component categories, P i Represents forecast information, represents the real information, and j represents the jth category among the n component categories. It can be understood that each feature extraction unit has corresponding real information and prediction information.

[0119] Step 443: Determine a third loss value according to the first vector converted from the second feature map output by the third-level feature extraction unit.

[0120] The third loss value is determined by using the prediction information represented by the first vector converted from the second feature map output by the third-level feature extraction unit and the real information corresponding to the third-level feature extraction unit.

[0121] The following formula can be used for calculation:

[0122]

[0123] Among them, Loss kp represents the third loss value, n represents the number of key points, P i Represents forecast information, Indicates true information.

[0124] Step 444: Determine a fourth loss value using the first loss value and the second loss value.

[0125] The following formula can be used for calculation:

[0126]

[0127] In this way, global information and local information supervise each other and promote each other. The global information makes the feature focus of local information in the transition stage more specific, while the features of local information make the global features more refined.

[0128] Step 445: Determine a fifth loss value using the first loss value and the third loss value.

[0129]

[0130] In this way, the global information and key point information supervise each other and promote each other. The global information makes the feature focus of the key point information more specific, while the features of the key point information make the global features more refined.

[0131] Step 446: Determine a sixth loss value using the second loss value and the third loss value.

[0132]

[0133] In this way, local information and key point information supervise each other and promote each other. Local information makes the feature focus of key point information more specific, while the feature of key point information makes local information more refined.

[0134] Step 447: Determine a seventh loss value using the fourth loss value, the fifth loss value, and the sixth loss value.

[0135] In some embodiments, see Figure 6 , step 447 may be as follows:

[0136] Step 4471: Use the first control parameter to weight the fourth loss value.

[0137] Step 4472: Use the second control parameter to weight the fifth loss value.

[0138] Step 4473: Use the third control parameter to weight the sixth loss value.

[0139] Step 4474: sum the weighted fourth loss value, fifth loss value and sixth loss value to obtain a seventh loss value.

[0140] Steps 4471 to 4474 can be expressed by the following formula:

[0141]

[0142] Among them, Loss 耦合 represents the seventh loss value, α represents the first control parameter, β represents the second control parameter, and γ represents the third control parameter.

[0143] The first control parameter, the second control parameter, and the third control parameter may be modified during parameter modification to participate in the next iterative training.

[0144] Step 45: Use the loss value to modify the parameters of the image detection network.

[0145] The first loss value, the second loss value, the third loss value and the seventh loss value are used to correct the parameters of the image detection network.

[0146] In this embodiment, the parameters can be corrected by back propagation to update the parameters in the common feature extraction unit, feature extraction units at different levels, feature fusion units, and fully connected units. Back propagation can accelerate the convergence speed of the image detection network and improve training efficiency.

[0147] According to the process of step 42 to step 44 above, the image detection network is iteratively trained multiple times using multiple first feature maps until the accuracy of the image detection network meets the requirements, and the trained image detection network can be used for image detection.

[0148] Through the above method, the three proposed multi-task learning branches can abstract object signs at different levels. Due to the mutual coupling relationship between the object skeleton features, the coupling relationship between multiple tasks can be made closer and more interrelated, achieving a tighter coupling, thereby improving the accuracy of image detection using the image detection network.

[0149] In other embodiments, the first feature map is also obtained in the image detection network, and the training process of the image detection network is as follows: construct an image detection network, and input the training image into the image detection network for training. Specifically, in the image detection network, feature extraction is performed on the training image to obtain a first feature map, and then feature extraction units at different levels are used to extract features from the first feature map to obtain at least two second feature maps, and then at least two second feature maps are feature fused in a feature fusion unit to obtain a third feature map. Vector transformation is performed based on at least two second feature maps and the third feature map, and the corresponding loss value is calculated using the transformed vector, and then parameter correction is performed according to the loss value.

[0150] In one application scenario, human body images are used as training images, combined with Figure 7 To explain:

[0151] First, the training image is input into the common feature layer to extract the first feature map mentioned in any of the above embodiments, and then the first feature map is input into the global feature extraction layer, the component analysis feature extraction layer, and the key point feature extraction layer respectively.

[0152] Among them, the global feature extraction layer can be used to extract the global contour of the human body in the training image, the component analysis feature extraction layer can be used to extract the contours of human body parts in the training image, such as arms, head, and legs, and the key point feature extraction layer can be used to extract the key points of the human body in the training image, such as eyes, nose, elbow joints, knee joints, ankles, and hand joints.

[0153] Then, a global loss calculation is performed on the second feature map output by the global feature extraction layer to obtain a first loss value.

[0154] The second feature map output by the component analysis feature extraction layer is used to perform component analysis loss calculation to obtain a second loss value.

[0155] The key point loss is calculated on the second feature map output by the key point feature extraction layer to obtain a third loss value.

[0156] Moreover, the second feature map output by the global feature extraction layer, the second feature map output by the key point feature extraction layer, and the second feature map output by the component analysis feature extraction layer are hierarchically mapped to obtain a third feature map, and the coupling loss is calculated for the third feature map to obtain a seventh loss value.

[0157] The first loss value, the second loss value, the third loss value and the seventh loss value are used to correct the parameters of the image detection network.

[0158] Combination Figure 8-Figure 11 As shown, Figure 8 is the original image, Fig. 9 It is the image information extracted by the global feature extraction layer. Fig.10 Image information extracted by the component analysis feature extraction layer. Fig.11 is the image information extracted using the key point feature extraction layer. Fig. 9 The human body segmentation features shown are defined as the global skeleton, Fig.10 The human body parts shown are defined as transition skeletons. Fig.11 As shown, the key point information of the human body is defined as a detailed skeleton. Obviously, from the perspective of skeleton features, the segmentation part provides global contour information, and the human body parts analysis provides further subdivided contours based on limb parts. The mutual supervision between the two can promote each other. The global feature focus of the transition stage is more specific, and the transition feature makes the global feature more refined. Similarly, global features and key point features also promote each other. Detailed features and transition features can also promote each other.

[0159] Therefore, based on the skeleton features of three different levels and relying on the skeleton, a skeleton coupling loss is constructed to promote the collaborative optimization learning of the three tasks, each of which can achieve better performance than training a single one or any two of them.

[0160] Therefore, after the training is completed, it can complete three different tasks when performing image detection with higher accuracy.

[0161] See also Fig.12 , Fig.12 1 is a schematic diagram of the structure of an embodiment of an image detection device provided by the present application. The image detection device 120 includes a processor 121 and a memory 122 coupled to the processor 121; wherein the memory 122 is used to store a computer program, and the processor 121 is used to execute the computer program to implement the following method:

[0162] Feature extraction is performed on an image to be processed containing an object to be processed to obtain a first feature map; feature extraction at different levels is performed on the first feature map to obtain multiple second feature maps corresponding to different levels; the multiple second feature maps are enhanced and fused to obtain a third feature map; object information of the object to be processed is determined based on the multiple second feature maps and the third feature map.

[0163] It can be understood that the processor 121 is also used to execute a computer program to implement the method of any of the above embodiments, which will not be elaborated here.

[0164] See also Fig.13 , Fig.13 1 is a schematic diagram of a computer readable storage medium according to an embodiment of the present application. The computer readable storage medium 130 is used to store a computer program 131. When the computer program 131 is executed by a processor, it is used to implement the following method:

[0165] Feature extraction is performed on an image to be processed containing an object to be processed to obtain a first feature map; feature extraction at different levels is performed on the first feature map to obtain multiple second feature maps corresponding to different levels; the multiple second feature maps are enhanced and fused to obtain a third feature map; object information of the object to be processed is determined based on the multiple second feature maps and the third feature map.

[0166] It can be understood that when the computer program 131 is executed by the processor, it is also used to implement the method of any of the above embodiments, which will not be elaborated here.

[0167] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program codes.

[0171] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image detection method, characterized in that: The method comprises: Extracting features from the image to be processed containing the object to be processed to obtain a first feature map; Inputting the first feature map into the trained image detection network; Based on multiple feature extraction units in the image detection network, feature extraction is performed on the first feature map to obtain multiple second feature maps; wherein different feature extraction units perform feature extraction at different levels on the first feature map; the multiple second feature maps include at least two of the following: a feature map for characterizing global information of the object to be processed; a feature map for characterizing component information of the object to be processed; and a feature map for characterizing key point information of the object to be processed; Performing enhanced fusion on the multiple second feature maps to obtain a third feature map; According to the plurality of second feature maps and the third feature map, object information of the object to be processed is determined.

2. The method according to claim 1, characterized in that The multiple feature extraction units include at least two of the following: A first feature extraction unit including n deconvolution modules and n dilated convolution modules; A second feature extraction unit including n / 2 deconvolution modules and 2n dilated convolutions; A third feature extraction unit including n / 4 deconvolution modules and 4n dilated convolutions; Here, n is a multiple of 4.

3. The method according to claim 1, characterized in that The method further comprises: Constructing the image detection network; wherein the image detection network at least includes feature extraction units and feature fusion units at different levels; Inputting the first feature map into the image detection network, obtaining the second feature maps respectively output by the feature extraction units at different levels, and obtaining the third feature map output by the feature fusion unit; Convert the second feature map into a first vector, and convert the third feature map into a second vector; Determine a loss value based on the first vector and the second vector; The loss value is used to modify parameters of the image detection network.

4. The method according to claim 3, characterized in that The feature extraction units at different levels include a first-level feature extraction unit, a second-level feature extraction unit, and a third-level feature extraction unit; The determining the loss value based on the first vector and the second vector includes: Determine a first loss value according to a first vector converted from the second feature map output by the first-level feature extraction unit; Determine a second loss value according to a first vector converted from the second feature map output by the second-level feature extraction unit; Determine a third loss value according to a first vector converted from the second feature map output by the third-level feature extraction unit; determining a fourth loss value using the first loss value and the second loss value; determining a fifth loss value using the first loss value and the third loss value; determining a sixth loss value using the second loss value and the third loss value; determining a seventh loss value using the fourth loss value, the fifth loss value and the sixth loss value; The using the loss value to perform parameter correction on the image detection network includes: The first loss value, the second loss value, the third loss value and the seventh loss value are used to perform parameter correction on the image detection network.

5. The method according to claim 4, characterized in that The determining a first loss value according to a first vector converted from the second feature map output by the first-level feature extraction unit includes: Determine the first loss value by using the prediction information represented by the first vector converted from the second feature map output by the first-level feature extraction unit and the real information corresponding to the first-level feature extraction unit; The determining a second loss value according to the first vector converted from the second feature map output by the second-level feature extraction unit includes: Determine the second loss value by using the prediction information represented by the first vector converted from the second feature map output by the second-level feature extraction unit and the real information corresponding to the second-level feature extraction unit; The determining a third loss value according to the first vector converted from the second feature map output by the third-level feature extraction unit includes: The third loss value is determined by using the prediction information represented by the first vector converted from the second feature map output by the third-level feature extraction unit and the real information corresponding to the third-level feature extraction unit.

6. The method according to claim 4, characterized in that The determining the seventh loss value by using the fourth loss value, the fifth loss value and the sixth loss value comprises: weighting the fourth loss value using the first control parameter; weighting the fifth loss value using a second control parameter; weighting the sixth loss value using a third control parameter; The seventh loss value is obtained by summing the weighted fourth loss value, the fifth loss value and the sixth loss value.

7. An image detection device, characterized in that: The image detection device includes a processor and a memory coupled to the processor; The memory is used to store a computer program, and the processor is used to execute the computer program to implement the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the computer program is used to implement the method according to any one of claims 1 to 6.