Object detection method, device, computer readable storage medium and computer equipment

Abstract

The present application relates to an object detection method, apparatus, computer-readable storage medium and computer equipment, the method comprising: obtaining an image to be detected; inputting the image to be detected into a target object detection model, the target object detection model extracting features from the image to be detected through a first feature extraction network set to obtain object state feature information, the first feature extraction network set comprising parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork; the target object detection model inputs the object state feature information into a second feature extraction network set to obtain position feature information corresponding to a target detection object in the image to be detected, the second feature extraction network set comprising parallel position feature extraction subnetworks and a second fusion subnetwork connected to each parallel position feature extraction subnetwork; and outputting a position area corresponding to the target detection object according to the position feature information.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an object detection method, apparatus, computer-readable storage medium, and computer equipment. Background Art

[0002] With the rapid development of computer technology, object detection models are usually used to detect the area where the target object is located in the image, for example, using object detection models to detect the specific location of the gesture in the image. However, in the prior art, it is difficult to detect smaller target objects in the image, such as small target objects in high-resolution images, using general object detection models, which easily leads to low accuracy of the object detection model. Summary of the invention

[0003] Based on this, it is necessary to provide an object detection method, apparatus, computer-readable storage medium and computer equipment that can ensure a balance between target object detection accuracy and target object computational complexity in response to the above technical problems.

[0004] An object detection method, comprising:

[0005] Acquire the image to be detected;

[0006] The image to be detected is input into the target object detection model, and the target object detection model extracts features from the image to be detected through a first feature extraction network set to obtain object state feature information, wherein the first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork;

[0007] The target object detection model inputs the object state feature information into the second feature extraction network set to obtain the position feature information corresponding to the target detection object in the image to be detected, and the second feature extraction network set includes parallel position feature extraction subnetworks and a second fusion subnetwork connected to each parallel position feature extraction subnetwork;

[0008] Output the location area corresponding to the target detection object according to the location feature information.

[0009] An object detection device, the device comprising:

[0010] An image acquisition module to be detected is used to acquire the image to be detected;

[0011] A target object detection model detection module, used for inputting an image to be detected into the target object detection model, wherein the target object detection model extracts features from the image to be detected through a first feature extraction network set to obtain object state feature information, wherein the first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork;

[0012] The target object detection model detection module is also used for the target object detection model to input the object state feature information into the second feature extraction network set to obtain the position feature information corresponding to the target detection object in the image to be detected, and the second feature extraction network set includes parallel position feature extraction sub-networks and a second fusion sub-network connected to each parallel position feature extraction sub-network;

[0013] The position area output module is used to output the position area corresponding to the target detection object according to the position feature information.

[0014] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the following steps are implemented:

[0015] Acquire the image to be detected;

[0016] The image to be detected is input into the target object detection model, and the target object detection model extracts features from the image to be detected through a first feature extraction network set to obtain object state feature information, wherein the first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork;

[0017] The target object detection model inputs the object state feature information into the second feature extraction network set to obtain the position feature information corresponding to the target detection object in the image to be detected, and the second feature extraction network set includes parallel position feature extraction subnetworks and a second fusion subnetwork connected to each parallel position feature extraction subnetwork;

[0018] Output the location area corresponding to the target detection object according to the location feature information.

[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the following steps:

[0020] Acquire the image to be detected;

[0021] The image to be detected is input into the target object detection model, and the target object detection model extracts features from the image to be detected through a first feature extraction network set to obtain object state feature information, wherein the first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork;

[0022] The target object detection model inputs the object state feature information into the second feature extraction network set to obtain the position feature information corresponding to the target detection object in the image to be detected, and the second feature extraction network set includes parallel position feature extraction subnetworks and a second fusion subnetwork connected to each parallel position feature extraction subnetwork;

[0023] Output the location area corresponding to the target detection object according to the location feature information.

[0024] The object detection method, device, computer-readable storage medium and computer equipment described above input the image to be detected into the trained target object detection model, the target object detection model obtains the object state feature information through the first feature extraction network set, the target object detection model inputs the object state feature information into the second feature extraction network set, the second feature extraction network set first determines the target detection object in the image to be detected according to the object state feature information, then extracts the position feature information corresponding to the target detection object, and finally outputs the position area corresponding to the target detection object according to the position feature information. Among them, the first feature extraction network set includes a parallel object state feature extraction subnetwork and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork, and the second feature extraction network set includes a parallel position feature extraction subnetwork and a second fusion subnetwork connected to each parallel position feature extraction subnetwork. By optimizing the network structure of the target object detection model, the target object detection model only calculates the feature information related to the target detection object in the image to be detected, reduces the calculation amount of the target object detection model, and ensures the detection accuracy of the target object detection model while improving the detection efficiency of the target object detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A diagram of an application environment of an object detection method in an embodiment;

[0026] Figure 2 is a schematic diagram of a flow chart of an object detection method in an embodiment;

[0027] Figure 3A A schematic diagram of a flow chart of a step of extracting object state feature information by a first feature extraction network set in one embodiment;

[0028] Figure 3BA schematic diagram of the structure of a first feature extraction network set in one embodiment;

[0029] Figure 4A A schematic diagram of a flow chart of a step of extracting location feature information by a second feature extraction network set in one embodiment;

[0030] Figure 4B A schematic diagram of the structure of a second feature extraction network set in one embodiment;

[0031] Figure 5A A schematic diagram of a flow chart of a step of determining a location area of ​​a target detection object in one embodiment;

[0032] Figure 5B A schematic diagram of an interface showing a detection result of an object detection method in an embodiment;

[0033] Figure 6 A schematic diagram of a flow chart of the training steps of a target object detection model in one embodiment;

[0034] Figure 7 is a flowchart of the training steps of the target object detection model in another embodiment;

[0035] Figure 8 is a schematic flow chart of the steps of acquiring a training image set in one embodiment;

[0036] Fig. 9 is a structural block diagram of an object detection device in one embodiment;

[0037] Fig.10 is a structural block diagram of a target object detection model detection module in one embodiment;

[0038] Fig.11 is a structural block diagram of a target object detection model detection module in another embodiment;

[0039] Fig.12 is a structural block diagram of a location area output module in one embodiment;

[0040] Fig.13 is a structural block diagram of an object detection device in another embodiment;

[0041] Fig.14 FIG. 4 is a structural block diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] Figure 1 FIG. 1 is an application environment diagram of an object detection method in an embodiment. Figure 1 , the object detection method is applied to an object detection system. The object detection system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal, and the mobile terminal can be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server 120 can be implemented by an independent server or a server cluster composed of multiple servers.

[0044] Specifically, the terminal 110 obtains the image to be detected and sends the image to be detected to the server 120. The server 120 inputs the image to be detected into the target object detection model, and the target object detection model extracts features from the image to be detected through the first feature extraction network set to obtain object state feature information, the first feature extraction network set includes a parallel object state feature extraction subnetwork and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork, and the target object detection model inputs the object state feature information into the second feature extraction network set to obtain the position feature information corresponding to the target detection object in the image to be detected, the second feature extraction network set includes a parallel position feature extraction subnetwork and a second fusion subnetwork connected to each parallel position feature extraction subnetwork, and outputs the position area corresponding to the target detection object according to the position feature information. Finally, the server 120 can send the position area to the terminal 110, and the terminal 110 can display the position area where the target detection object in the image to be detected is located.

[0045] like Figure 2 As shown, in one embodiment, a method for object detection is provided. This embodiment mainly applies the method to the above Figure 1 The terminal 110 or the server 120 in FIG. Figure 2 , the object detection method specifically includes the following steps:

[0046] Step 202: Acquire an image to be detected.

[0047] The images to be detected include but are not limited to pictures, photos, films, etc. They may be photos taken by a camera of a terminal, pictures obtained by screenshots of a terminal, or images uploaded by an application capable of uploading images, etc. If the execution subject is a server 120, the terminal may send these images to be detected to the server 120 for subsequent processing. The images to be detected here may be but are not limited to those used to detect small target objects in an image. For example, the image to be detected is a high-resolution image, which includes many small target objects. Since the small target objects in the high-resolution image to be detected are relatively inconspicuous in the high-resolution image, it is necessary to detect the location area where the small target objects in the high-resolution image to be detected are located through a target object detection model.

[0048] Step 204: input the image to be detected into the target object detection model. The target object detection model extracts features from the image to be detected through a first feature extraction network set to obtain object state feature information. The first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork.

[0049] Among them, the target object detection model is a network model used to detect the location area where the target object in the image to be detected is located. The target object detection model has been trained and can be used directly for detection. The target object detection model includes but is not limited to a first feature extraction network set and a second feature extraction network set. The first feature extraction network set is used to extract object state feature information, and the second feature extraction network set is used to extract object position feature information. The first feature extraction network set and the second feature extraction network set are connected, and the output of the first feature extraction network set is used as the input of the second feature extraction network set. Among them, the object state feature information here is feature information related to the state of the target detection object. The target detection object can be determined based on the object state feature information. The object state feature information can be information such as color and contour size.

[0050] The first feature extraction network set in the target object detection model includes but is not limited to a parallel object state feature extraction subnetwork and a first fusion subnetwork. The parallel object state feature extraction subnetworks have different feature extraction methods, different convolution kernel sizes between the object state feature extraction subnetworks, and the same step size of the object state feature extraction subnetworks.

[0051] Among them, the parallel mode of each object state feature extraction subnetwork can be customized, and the customization can be that the first object state feature extraction subnetwork is connected to the second object state feature extraction subnetwork and the third object state feature extraction subnetwork respectively, that is, the second object state feature extraction subnetwork and the third object state feature extraction subnetwork are connected in parallel, and the output of the first object state feature extraction subnetwork is used as the input of the second object state feature extraction subnetwork and the third object state feature extraction subnetwork. Among them, the number of object state feature extraction subnetworks can be set according to actual needs, and the best effect is to set the number of object state feature extraction subnetworks of the first feature extraction network set to 3.

[0052] Among them, the connection mode between the parallel object state feature extraction subnetworks and the first fusion subnetwork in the first feature extraction network set can be customized, and the customization can be to use the output of the parallel object state feature extraction subnetwork as the input of the first fusion subnetwork, and the first fusion subnetwork splices the outputs of all parallel object state feature extraction subnetworks to obtain object state feature information. For example, the object state feature extraction subnetwork includes the first object state feature extraction subnetwork, the second object state feature extraction subnetwork and the third object state feature extraction subnetwork, the output of the first object state feature extraction subnetwork is used as the input of the second object state feature extraction subnetwork and the third object state feature extraction subnetwork, and the output of the second object state feature extraction subnetwork and the third object state feature extraction subnetwork is used as the input of the first fusion subnetwork.

[0053] The parallel object state feature extraction subnetworks in the first feature extraction network set all include a downsampling process. The downsampling in the first feature extraction network set is to reduce the image so that the image fits the size of the display area and the complexity of the image is small.

[0054] In step 206, the target object detection model inputs the object state feature information into the second feature extraction network set to obtain the position feature information corresponding to the target detection object in the image to be detected. The second feature extraction network set includes parallel position feature extraction subnetworks and a second fusion subnetwork connected to each parallel position feature extraction subnetwork.

[0055] Among them, the second feature extraction network set here is used to extract object position feature information, and the object position feature information is feature information related to the position of the target detection object. The area where the target detection object is located can be determined based on the object position feature information. The object position feature information can be the area size, the center point position of the target detection object, etc.

[0056] Among them, since the target object detection model includes but is not limited to the first feature extraction network set and the second feature extraction network set, the first feature extraction network set and the second feature extraction network set are connected, and the output of the first feature extraction network set is used as the input of the second feature extraction network set. Therefore, the object state feature information output by the first feature extraction network set is input into the second feature extraction network set, and the second feature extraction network set can determine the target detection object in the image to be detected according to the object state feature information, and then extract the corresponding position feature information according to the target detection object. For example, the position state feature information output by the first feature extraction network set is the outline size and color of the hand, and the position state feature information output by the first feature extraction network set can be input into the second feature extraction network set, and the second feature extraction network set first determines that the target detection object is a hand according to the outline size and color of the hand, and then extracts the position feature information related to the hand, such as the center point position of the hand, the position area of ​​the hand, etc.

[0057] The second feature extraction network set includes but is not limited to parallel position feature extraction subnetworks and second fusion subnetworks connected to each parallel position feature extraction subnetwork. The parallel position feature extraction subnetworks have different feature extraction methods, different convolution kernel sizes between position feature extraction subnetworks, and the same step size for each position feature extraction subnetwork.

[0058] Among them, the parallel mode of each position feature extraction subnetwork can be customized, and the customization can be that the first position feature extraction subnetwork is connected to the second position feature extraction subnetwork, the third position feature extraction subnetwork and the fourth position feature extraction subnetwork respectively, that is, the second position feature extraction subnetwork, the third position feature extraction subnetwork and the fourth position feature extraction subnetwork are connected in parallel, and the output of the first position feature extraction subnetwork is used as the input of the second position feature extraction subnetwork, the third position feature extraction subnetwork and the fourth position feature extraction subnetwork. Among them, the number of position feature extraction subnetworks can be set according to actual needs, and the number of position feature extraction subnetworks of the second feature extraction network set is set to 4 for the best effect.

[0059] Among them, the connection mode between the parallel position feature extraction subnetworks and the second fusion subnetwork in the second feature extraction network set can be customized, and the customization can be to use the output of the parallel position feature extraction subnetwork as the input of the second fusion subnetwork, and the second fusion subnetwork splices the outputs of all parallel position feature extraction subnetworks to obtain position feature information. For example, the position feature extraction subnetwork includes the first position feature extraction subnetwork, the second position feature extraction subnetwork, the third position feature extraction subnetwork and the fourth position feature extraction subnetwork, the output of the first position feature extraction subnetwork is used as the input of the second position feature extraction subnetwork, the third position feature extraction subnetwork and the fourth position feature extraction subnetwork, and the output of the second position feature extraction subnetwork, the third position feature extraction subnetwork and the fourth position feature extraction subnetwork is used as the input of the second fusion subnetwork.

[0060] The parallel position feature extraction subnetworks in the second feature extraction network set all include a downsampling process. The downsampling in the second feature extraction network set is to reduce the image so that the image fits the size of the display area and the complexity of the image is small.

[0061] Step 208: output the location area corresponding to the target detection object according to the location feature information.

[0062] Among them, the position area refers to the specific area where the target detection object in the image to be detected is located. Specifically, after the target object detection model outputs the position feature information corresponding to the target detection object, since the position feature information is the position information related to the target detection object, the specific position area where the target detection object is located can be determined according to the position feature information, and finally the position area corresponding to the determined target detection object is output. Among them, the method of determining the position area corresponding to the target detection object according to the position feature information can be customized, and the customization can be to first determine the center point position of the target detection object, determine the corresponding position area range according to the outline size of the target detection object, and finally draw the position area of ​​the target detection object according to the center point position and the position area range.

[0063] The above object detection method inputs the image to be detected into the trained target object detection model, the target object detection model obtains the object state feature information through the first feature extraction network set, the target object detection model inputs the object state feature information into the second feature extraction network set, the second feature extraction network set first determines the target detection object in the image to be detected according to the object state feature information, then extracts the position feature information corresponding to the target detection object, and finally outputs the position area corresponding to the target detection object according to the position feature information. Among them, the first feature extraction network set includes a parallel object state feature extraction subnetwork and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork, and the second feature extraction network set includes a parallel position feature extraction subnetwork and a second fusion subnetwork connected to each parallel position feature extraction subnetwork. By optimizing the network structure of the target object detection model, the target object detection model only calculates the feature information related to the target detection object in the image to be detected, reduces the calculation amount of the target object detection model, and ensures the detection accuracy of the target object detection model while improving the detection efficiency of the target object detection model.

[0064] In one embodiment, Figure 3A As shown, the target object detection model extracts features from the image to be detected through the first feature extraction network set to obtain object state feature information, including:

[0065] Step 302: input the image to be detected into the first shared convolutional subnetwork to extract the first shared image features.

[0066] Among them, Figure 3B As shown, Figure 3B A schematic diagram showing the structure of a first feature extraction network set in one embodiment is shown. Figure 3BThe first feature extraction network set shown includes a first shared convolutional subnetwork, a first convolutional subnetwork, a first pooling subnetwork and a first fusion subnetwork, wherein the first shared convolutional subnetwork serves as the input of the first feature extraction network set, and the output of the first shared convolutional subnetwork serves as the input of the first convolutional subnetwork and the first pooling subnetwork, that is, the first convolutional subnetwork and the first pooling subnetwork are connected to the first shared convolutional subnetwork in parallel. The output of the first convolutional subnetwork and the first pooling subnetwork serves as the input of the first fusion subnetwork, and the first fusion subnetwork is the output of the first feature extraction network set, which is used to output object state feature information. Among them, the first shared convolutional subnetwork, the first convolutional subnetwork, and the first pooling subnetwork all include a downsampling process. Among them, the first shared convolutional subnetwork, the first convolutional subnetwork, and the first pooling subnetwork have different feature extraction methods, different convolutional kernel sizes, and the same step size. Among them, the convolution kernel size and step size of the first shared convolution subnetwork, the first convolution subnetwork, and the first pooling subnetwork can be set according to actual needs. The convolution kernel sizes of the first shared convolution subnetwork, the first convolution subnetwork, and the first pooling subnetwork are set to 3*3, 3*3, and 2*2 respectively, and the step sizes are all set to 2, which has the best effect in extracting object state feature information.

[0067] Among them, the first shared convolution subnetwork serves as the input network of the first feature extraction network set, and the image to be detected is input into the first shared convolution subnetwork. The first shared convolution subnetwork performs convolution feature extraction on the image to be detected to obtain the first shared image feature. The first shared convolution subnetwork here is used to extract the first shared image feature in the image to be detected. Since there are other irrelevant objects in addition to the target detection object in the image to be detected, it is necessary to first perform initial convolution feature extraction on the image to be detected through the first shared convolution subnetwork to obtain the first shared image feature. The first shared image feature here is the input image of the first convolution subnetwork and the first pooling subnetwork, and is an image feature shared by the first convolution subnetwork and the first pooling subnetwork.

[0068] Step 304: input the first shared image feature into a parallel first convolution subnetwork and a first pooling subnetwork, wherein the first shared convolution subnetwork, the first convolution subnetwork and the first pooling subnetwork include a downsampling process.

[0069] Step 306: The first convolution subnetwork outputs image convolution features, and the first pooling subnetwork outputs image detail information.

[0070] Among them, Figure 3BAs shown, the image to be detected is input into the first shared convolution subnetwork, and the first shared convolution subnetwork performs initial convolution feature extraction on the image to be detected to obtain the first shared image feature. The first shared image feature is input into the parallel first convolution subnetwork and the first pooling subnetwork. The first convolution subnetwork and the first pooling subnetwork have different feature extraction methods, different convolution kernel sizes, and the same step size. The parallel first convolution subnetwork and the first pooling subnetwork are set to be able to extract more state feature information in different aspects. Among them, in order to reduce the complexity of the image and better extract the state feature information of the object in the image, the downsampling process must be included in the first shared convolution subnetwork, the first convolution subnetwork, and the first pooling subnetwork.

[0071] Specifically, after the first shared image feature output by the first shared convolution subnetwork is input into the parallel first convolution subnetwork and the first pooling subnetwork, the first convolution subnetwork performs corresponding convolution feature extraction on the first shared image feature and outputs the corresponding image convolution feature. The image convolution feature here is obtained by the first convolution subnetwork performing convolution extraction on the first shared image feature through the corresponding feature extraction method. Similarly, the first pooling subnetwork performs corresponding feature extraction on the first shared image feature and outputs the corresponding image detail information. The image detail information here is obtained by the first pooling subnetwork performing convolution extraction on the first shared image feature through the corresponding feature extraction method. Among them, the first convolution subnetwork and the first pooling subnetwork both have corresponding feature extraction methods, which can extract different features from the first shared image feature.

[0072] Step 308: The image convolution features and the image detail information are concatenated through the first fusion sub-network to obtain object state feature information.

[0073] Among them, Figure 3B As shown, after the first convolution subnetwork and the first pooling subnetwork respectively output the image convolution features and the image detail information at the same time, the image convolution features and the image detail information are input into the first fusion subnetwork, and the first fusion subnetwork performs feature splicing on the image convolution features and the image detail information to obtain the object state feature information. Among them, the feature splicing method of the first fusion subnetwork can be customized, and the customization can be to list the image convolution features and the image detail information in a customized format to form the object state feature information, etc. That is, the object state feature information includes the image convolution features and the image detail information. Among them, the first fusion subnetwork can also customize the feature splicing method by fusing the image convolution features and the corresponding image detail information to obtain the object state feature information.

[0074] In one embodiment, Figure 4AAs shown, the target object detection model inputs the object state feature information into the second feature extraction network set to obtain the position feature information corresponding to the target detection object in the image to be detected, including:

[0075] Step 402: input the object state feature information into the second shared convolutional subnetwork to extract the second shared image feature.

[0076] Among them, Figure 4B As shown, Figure 4B A schematic diagram showing the structure of a second feature extraction network set in one embodiment is shown. Figure 4B The second feature extraction network set shown includes a second shared convolutional subnetwork, a second convolutional subnetwork, a third convolutional subnetwork, a second pooling subnetwork, and a second fusion subnetwork. Among them, the second shared convolutional subnetwork is used as the input of the second feature extraction network set, and the output of the second shared convolutional subnetwork is used as the input of the second convolutional subnetwork, the third convolutional subnetwork, and the second pooling subnetwork, that is, the second convolutional subnetwork, the third convolutional subnetwork, and the second pooling subnetwork are connected to the second shared convolutional subnetwork in parallel. The output of the second convolutional subnetwork, the third convolutional subnetwork, and the second pooling subnetwork is used as the input of the second fusion subnetwork, and the second fusion subnetwork is the output of the second feature extraction network set, which is used to output position feature information. Among them, the second shared convolutional subnetwork, the second convolutional subnetwork, the third convolutional subnetwork, and the second pooling subnetwork all include a downsampling process. Among them, the second shared convolutional subnetwork, the second convolutional subnetwork, the third convolutional subnetwork, and the second pooling subnetwork have different feature extraction methods, different convolution kernel sizes, and the same step size. Among them, the convolution kernel size and step size of the second shared convolution subnetwork, the second convolution subnetwork, the third convolution subnetwork, and the second pooling subnetwork can be set according to actual needs. The second shared convolution subnetwork, the second convolution subnetwork, the third convolution subnetwork, and the second pooling subnetwork are set to 1*1, 3*3, 5*5, and 2*2 respectively, and the step size is set to 1, which has the best effect in extracting position feature information.

[0077] Specifically, since the first feature extraction network set is connected to the second feature extraction network set, the output of the first feature extraction network set serves as the input of the second feature extraction network set, and the second shared convolution subnetwork serves as the input network of the second feature extraction network set, the object state feature information output by the first feature extraction network set is input into the second shared convolution subnetwork. The second shared convolution subnetwork performs convolution feature extraction on the object state feature information according to the corresponding feature extraction method and convolution kernel size to obtain the second shared image feature. Similarly, the second shared image feature is the input image of the second convolution subnetwork, the third convolution subnetwork, and the second pooling subnetwork, and is an image feature shared by the second convolution subnetwork, the third convolution subnetwork, and the second pooling subnetwork.

[0078] Step 404, input the second shared image feature to the parallel second convolution subnetwork, the third convolution subnetwork and the second pooling subnetwork, the feature extraction network scales of the second convolution subnetwork and the third convolution subnetwork are different, and the second shared convolution subnetwork, the second convolution subnetwork, the third convolution subnetwork and the second pooling subnetwork include a downsampling process.

[0079] Step 406: the second convolution subnetwork outputs the first position convolution feature, the third convolution subnetwork outputs the second position convolution feature, and the second pooling subnetwork outputs the position detail information.

[0080] Among them, Figure 4B As shown, the object state feature information is input into the second shared convolution subnetwork, and the second shared convolution subnetwork performs shared convolution feature extraction on the object state feature information to obtain the second shared image feature. The second shared image feature is input into the parallel second convolution subnetwork, the third convolution subnetwork, and the second pooling subnetwork, and the feature extraction network scales of the second convolution subnetwork, the third convolution subnetwork, and the second pooling subnetwork are different, wherein the different feature extraction network scales may be different feature extraction methods, different convolution kernel sizes, the same step size, etc. The parallel second convolution subnetwork, the third convolution subnetwork, and the second pooling subnetwork are set in order to be able to extract more different aspects of location information. Among them, in order to reduce the complexity of the image and better extract the location feature information in the image, it is necessary to include a downsampling process in the second shared convolution subnetwork, the second convolution subnetwork, the third convolution subnetwork, and the second pooling subnetwork.

[0081] Specifically, after the second shared image feature is input into the parallel second convolution subnetwork, the third convolution subnetwork and the second pooling subnetwork, the second convolution subnetwork performs corresponding convolution feature extraction on the second shared image feature and outputs the corresponding first position convolution feature. The first position convolution feature here is obtained by the second convolution subnetwork performing convolution feature extraction on the second shared image feature according to the corresponding feature extraction method and the convolution kernel size. Similarly, the third convolution subnetwork performs corresponding convolution feature extraction on the second shared image feature and outputs the corresponding second position convolution feature. The second position convolution feature here is obtained by the third convolution subnetwork performing convolution feature extraction on the second shared image feature according to the corresponding feature extraction method and convolution sum size. The second pooling subnetwork performs corresponding feature extraction on the second shared image feature and outputs the corresponding position detail information. The position detail information here is the information related to the position detail in the second shared image feature, which is specifically obtained by the second pooling subnetwork performing convolution calculation according to the corresponding feature extraction method and the convolution kernel size.

[0082] Step 408, the first position convolution feature, the second position convolution feature and the position detail information are concatenated through a second fusion sub-network to obtain position feature information.

[0083] Among them, Figure 4B As shown, after the second convolution subnetwork, the third convolution subnetwork and the second pooling subnetwork respectively output the first position convolution feature, the second position convolution feature and the position detail information at the same time, the first position convolution feature, the second position convolution feature and the position detail information are input into the second fusion subnetwork, and the second fusion subnetwork performs feature splicing on the first position convolution feature, the second position convolution feature and the position detail information to obtain the position feature information. Similarly, the feature splicing method of the second fusion subnetwork can be customized, and the customization can be to list the first position convolution feature, the second position convolution feature and the position detail information in a customized format to form the position feature information, etc. That is, the position feature information includes the first position convolution feature, the second position convolution feature and the position detail information. Among them, the second fusion subnetwork can also customize the feature splicing method according to the corresponding relationship between the first position convolution feature, the second position convolution feature and the position detail information to obtain the position feature information.

[0084] In one embodiment, Figure 5A As shown, the location area corresponding to the target detection object is output according to the location feature information, including:

[0085] Step 502: Determine the contour of the target detection object in the image to be detected according to the position feature information.

[0086] Among them, since the position feature information is obtained by extracting the object state feature information by the second feature extraction network set in the target object detection model, the position feature information includes but is not limited to the object state feature information. Specifically, the contour of the target detection object in the image to be detected can be determined based on the object state feature information in the position feature information. For example, the object state feature information is the color of the hand and the contour of the hand, and the position feature information output by the second feature extraction network set includes the color of the hand and the contour of the hand. According to the contour and color of the hand, it can be determined that the target detection object in the image to be detected is a hand.

[0087] Step 504: determine the region range of the target detection object according to the outline of the target detection object.

[0088] Among them, since the outlines of different target detection objects are of different sizes, the area ranges corresponding to the target detection objects will be different. The area range of the target detection object can be determined according to the outline of the target detection object, that is, the area range of the target detection object must include the target detection object. Among them, the principle for determining the area range of the target detection object can be that the larger the outline of the target detection object, the larger the area range of the target detection object.

[0089] Step 506: determine the center point position of the target detection object according to the position feature information.

[0090] The center point position here is the center position of the target detection object. The center point position of the target detection object may be the center of the area where the target detection object is located. Specifically, since the position feature information is information related to the position of the target detection object, such as the center point position of the target detection object, the location area where the target detection object is located, etc., the center point position of the target detection object can be determined according to the position feature information.

[0091] Step 508: Draw the location area and center position of the target detection object according to the area range and the center point position.

[0092] Specifically, after determining the center point position and the corresponding area range of the target detection object, in order to finally detect and display the location area and center position of the target detection object in the image to be detected, it is necessary to draw the corresponding location area and center position according to the area range and center point position of the target detection object. Among them, the center position of the target detection object is located in the location area of ​​the target detection object. For example, Figure 5B As shown, Figure 5B A schematic diagram of an interface showing a detection result of an object detection method in an embodiment is shown, Figure 5B The target detection object shown is a human hand. Therefore, the location area and center position of the hand are drawn according to the area range and the center point position. Figure 5B The area framed by the solid line is where the hand is located. Figure 5B The center point of the circle is the center position of the hand.

[0093] In one embodiment, Figure 6 As shown, the training steps of the target object detection model include:

[0094] Step 602: Acquire a training image set, where the training images in the training image set include annotated region positions corresponding to the training target objects.

[0095] Among them, the training image set is a training data set used to train the initial object detection model, and the training image set includes multiple training images, each training image includes a training target object, and each training target object is marked with a corresponding marked area position. Among them, the marked area position is used to mark the standard area position where the training target object in the training image is located, that is, the marked area position is the reference standard for the output result of the initial detection model. The marked area position is equivalent to the label of the training image, which is subsequently used as the basis for adjusting the model parameters of the initial object detection model together with the output result of the initial object detection model. It should be noted that the training image sets here are all cleaned, and the specific cleaning process of the training image sets will be specifically described in the subsequent embodiments, which will not be repeated here.

[0096] Step 604: input the training images in the training image set into the initial object detection model. The initial object detection model extracts features from the training images through a first feature extraction network set to obtain training object state feature information. The first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork.

[0097] The initial object detection model here refers to the object detection model that has been initialized by the original object detection model. The initialization process of the original object detection model will be specifically described in the subsequent embodiments, and will not be repeated here. Specifically, the training image in the training image set is used as the input data of the initial object detection model, and the first feature extraction network set of the initial object detection model extracts features from the training image to obtain the training object state feature information. Specifically, at least two parallel object state feature extraction subnetworks in the first feature extraction network set perform convolution feature extraction on the training image to obtain the corresponding object state convolution feature information, and then the object state convolution feature information output by each parallel object state feature extraction subnetwork is spliced ​​through the first fusion subnetwork to obtain the training object state feature information corresponding to the training image. The training object state feature information refers to the state information related to the training target object in the training image. The training target object in the training image can be determined according to the training object state feature information. The training object state feature information can be the color of the training target object, the outline size of the training target object, etc.

[0098] In one embodiment, the network structure of the first feature extraction network set in the initial object detection model can be as follows: Figure 3B The network structure of the first feature extraction network set shown in FIG. If the first feature extraction network set of the initial object detection model is Figure 3BThe first feature extraction network set shown in the figure includes a first shared convolutional subnetwork, a first convolutional subnetwork, a first pooling subnetwork and a first fusion subnetwork, wherein the first shared convolutional subnetwork is used as the input of the first feature extraction network set, and the output of the first shared convolutional subnetwork is used as the input of the first convolutional subnetwork and the first pooling subnetwork, that is, the first convolutional subnetwork and the first pooling subnetwork are connected to the first shared convolutional subnetwork in parallel. The output of the first convolutional subnetwork and the first pooling subnetwork is used as the input of the first fusion subnetwork, and the first fusion subnetwork is the output of the first feature extraction network set, which is used to output the state feature information of the training object.

[0099] Step 606: The initial object detection model inputs the training object state feature information into the second feature extraction network set to obtain the training position feature information corresponding to the training target object in the training image. The second feature extraction network set includes parallel position feature extraction subnetworks and a second fusion subnetwork connected to each parallel position feature extraction subnetwork.

[0100] Among them, the first feature extraction network set and the second feature extraction network set in the initial object detection model are connected, and the output of the first feature extraction network set is used as the input of the second feature extraction network set. Therefore, the initial object detection model inputs the training object state feature information output by the first feature extraction network set into the second feature extraction network set, and the second feature extraction network set extracts features from the training object state feature information to obtain training position feature information. Specifically, at least two parallel position feature extraction subnetworks in the second feature extraction network set perform convolution feature extraction on the training object state feature information to obtain corresponding training object position convolution feature information, and then the training object position convolution feature information output by each parallel position feature extraction subnetwork is spliced ​​through the second fusion subnetwork, thereby obtaining the training position feature information corresponding to the training target object in the training image.

[0101] In one embodiment, the network structure of the second feature extraction network set in the initial object detection model can be as follows: Figure 4B The network structure of the second feature extraction network set shown in FIG. If the second feature extraction network set of the initial object detection model is Figure 4BThe second feature extraction network set shown in FIG. 1 includes a second shared convolutional subnetwork, a second convolutional subnetwork, a third convolutional subnetwork, a second pooling subnetwork, and a second fusion subnetwork. The second shared convolutional subnetwork serves as the input of the second feature extraction network set, and the output of the second shared convolutional subnetwork serves as the input of the second convolutional subnetwork, the third convolutional subnetwork, and the second pooling subnetwork, that is, the second convolutional subnetwork, the third convolutional subnetwork, and the second pooling subnetwork are connected to the second shared convolutional subnetwork in parallel. The outputs of the second convolutional subnetwork, the third convolutional subnetwork, and the second pooling subnetwork serve as the input of the second fusion subnetwork, and the second fusion subnetwork is the output of the second feature extraction network set, which is used to output the training position feature information.

[0102] Step 608, adjusting the model parameters of the initial object detection model according to the training position feature information and the position of the marked area, until the training position feature information output by the initial object detection model meets the convergence condition, thereby obtaining a trained target object detection model.

[0103] Among them, since the training images in the training image set all include the annotated area positions corresponding to the training target objects, and the annotated area positions corresponding to the training target objects of the training images are the standard annotations of the training target objects, the training position feature information output by the initial object detection model is compared with the corresponding annotated area positions, and the model parameters of the initial object detection model are adjusted until the training position feature information output by the initial object detection model meets the convergence conditions, and the trained target object detection model is obtained. Among them, the convergence conditions can be customized, and the customization can be that the loss function calculated based on the training position feature information of the training image and the corresponding annotated area position reaches the minimum value, and then it can be considered that the convergence conditions are met. Among them, the smaller the loss function calculated based on the training position feature information of the training image and the corresponding annotated area position, the closer the training position feature information is to the annotated area position, that is, the more accurate the output result of the initial object detection model is.

[0104] In one embodiment, Figure 7 As shown, Figure 6 The training steps of the target object detection model shown also include:

[0105] Step 702: Obtain annotation data corresponding to the annotation area position corresponding to the training target object in the training image.

[0106] Among them, the initial object detection model is an object detection model that has been initialized. Therefore, in this embodiment, the initialization of the object detection model is mainly explained in detail. Specifically, since the training image includes the marked area position corresponding to the training target object, the annotation data corresponding to the marked area position corresponding to the training target object in the training image is obtained. Specifically, the marked area position corresponding to the training target object in the training image can be converted into corresponding coordinate data in a custom format, and the coordinate data is used as the annotation data. Among them, the coordinate data in the custom format can be (x, y, w, h), x represents the horizontal coordinate, y represents the vertical coordinate, w represents the width, and h represents the height. For example, the marked area position corresponding to the training image in a certain training image is converted into the corresponding annotation data (3, 5, 7, 9), where 3 and 5 represent the horizontal coordinate and the vertical coordinate, and 7 and 9 represent the width and height.

[0107] Step 704, calculating the initialization value corresponding to the object detection model based on the labeled data.

[0108] After obtaining the annotation data corresponding to the annotation area position corresponding to the training target object in the training image, the initialization value corresponding to the object detection model can be calculated according to the annotation data corresponding to each training image. The calculation method of the initialization value corresponding to the object detection model can be customized, and the customization can be to average the annotation data corresponding to each training image, or to perform clustering calculation on the annotation data corresponding to each training image, or to perform weighted average calculation on the annotation data corresponding to each training image, etc.

[0109] In one embodiment, after obtaining the annotation data corresponding to the annotation area position corresponding to the training target object in the training image, the annotation data corresponding to each training image is averaged, and the calculated average value of the annotation data corresponding to each training image is used as the initialization value corresponding to the object detection model.

[0110] Step 706, initializing the model parameters of the object detection model according to the initialization values ​​to obtain an initial object detection model.

[0111] Specifically, after the initialization value corresponding to the object detection model is calculated according to the labeled data, the model parameters of the object detection model are initialized according to the initialization value to obtain the initial object detection model. In this way, the target object detection model obtained by training the initialized initial object detection model will detect the target location area according to the initialization value, and specifically can find the detection of the target location area within a certain range of the initialization value, thereby improving the detection efficiency of the target location area. For example, if the initialization value is (5, 18, 35, 12), then the target object detection model finally trained will give priority to searching within the range of this initialization value (5, 18, 35, 12) when detecting the location area range where the target detection object is located, because the location area range where the target detection object is located is likely to be larger near the initialization value. Starting to search within the range of the initialization value (5, 18, 35, 12) can reduce the detection time of the target object detection model and improve the detection efficiency.

[0112] In one embodiment, Figure 8 As shown, a training image set is obtained, and the training images in the training image set include the annotated area positions corresponding to the training target objects, including:

[0113] Step 802: Acquire a sample image set, where the sample images in the sample image set include corresponding sample object areas.

[0114] Specifically, the sample image set is a set of collected sample images, wherein the sample images include sample object regions corresponding to sample objects, and the sample object regions are regions where sample objects are marked in the sample images. Sample images with marked sample object regions are grouped into a sample image set.

[0115] Step 804: obtaining preset screening rules, wherein the preset screening rules include matching relationships between sample objects in different area ranges and sample image processing rules.

[0116] Among them, the preset screening rules are used to screen the sample image set, and the preset screening rules can be customized. The customization can be determined according to business needs, or determined according to the sample object area in the sample image set, etc. The preset screening rules include the matching relationship between sample objects in different area ranges and sample image processing rules. Among them, the sample image processing rules are rules used to process sample objects in the corresponding area range, and the matching relationship between sample objects in different area ranges and sample image processing rules is one-to-one. For example, according to business needs, the preset screening rules include the first-level sample image processing rules, the second-level sample image processing rules, the third-level sample image processing rules and the fourth-level sample image processing rules. The sample object area range corresponding to the first-level sample image processing rules is: less than 80*80, the sample object area range corresponding to the second-level sample image processing rules is: 80*80-150*150, the sample object area range corresponding to the third-level sample image processing rules is: 150*150-300*300, and the sample object area range corresponding to the fourth-level sample image processing rules is: 300*300 or more.

[0117] Step 806, according to the range and matching relationship of the sample object area in the sample image, obtain the target sample data processing rules corresponding to each sample image, and perform sample image processing on the corresponding sample image according to the target sample data processing rules to obtain training data, wherein the training data includes the marked area position corresponding to the training target object.

[0118] Specifically, after obtaining the preset screening rules, the target sample data processing rules corresponding to the sample image are determined according to the range of the sample object area in the sample image and the preset screening rules, and then the corresponding sample image is processed according to the target sample data processing rules to obtain training data. Among them, the sample image processing includes but is not limited to filtering the sample image, or enlarging the sample image, or reducing the sample image, etc. Among them, enlarging or reducing the sample image is to enlarge or reduce the size of the sample image according to a preset multiple, and when the sample image is enlarged or reduced, the sample object area in the sample image will also be enlarged or reduced accordingly. The sample object area after enlargement and reduction is the corresponding annotation area position.

[0119] In one embodiment, the preset screening rules include first-level sample image processing rules, second-level sample image processing rules, third-level sample image processing rules and fourth-level sample image processing rules. According to the range and matching relationship of the sample object area in the sample image, the target sample data processing rules corresponding to each sample image are obtained, and the corresponding sample image is processed according to the target sample data processing rules to obtain training data. The training data includes the position of the annotated area corresponding to the training target object, including: when the sample object area in the sample image is smaller than the first area range corresponding to the first-level sample image processing rule, and / or when the sample object area in the sample image is larger than the fourth area range corresponding to the fourth-level sample image processing rule, the sample image is filtered; when the sample object area in the sample image is located in the second area range corresponding to the second-level sample image processing rule, a preset multiple is obtained, and part of the sample image is enlarged according to the preset multiple to obtain the enlarged sample image, and the enlarged sample image is composed of the training image set; when the sample object area in the sample image is located in the third area range corresponding to the third-level sample image processing rule, the sample image is directly composed of the training image set.

[0120] Among them, the preset screening rules determined according to business needs include first-level sample image processing rules, second-level sample image processing rules, third-level sample image processing rules and fourth-level sample image processing rules, wherein each sample image processing rule has a corresponding area range, and therefore the corresponding sample image processing rule can be determined according to the sample object area in the sample image.

[0121] In one embodiment, when the sample object area in the sample image is smaller than the first area range corresponding to the first-level sample image processing rule, it means that the sample object size in the sample image is too small, but in actual application scenarios, such small sample objects do not appear or do not need to be concerned about, and the learning of too small sample objects requires the consumption of the model capacity and resources of the initial object detection model, so the corresponding sample image is filtered. For example, if the first-level sample image processing rule is to filter the sample image, and the first area range corresponding to the first-level sample image processing rule is 80*80, when the sample object area in the sample image is smaller than 80*80, the sample image is filtered out.

[0122] In one embodiment, when the sample object area in the sample image is located in the second area range corresponding to the second-level sample image processing rule, it means that the sample object in the second area range is closer to the area range of the normal object, and the possibility of falling into the second area range in the actual application scenario is not great, but the sample images falling into the second area range can be used to enhance the diversity of the sample images. Specifically, for sample images falling into the second area range, a preset number of sample images can be randomly obtained from the sample images, a preset multiple can be obtained, and some sample images can be enlarged according to the preset multiple, and the enlarged sample images are added to the training image set. Among them, when the sample image is enlarged, the sample object area in the sample image is enlarged accordingly, and the enlarged sample object area is the marked area position in the training image. For example, the second area range can be 80*80-150*150.

[0123] In one embodiment, when the sample object area in the sample image is located in the third area range corresponding to the third-level sample image processing rule, it means that the sample images falling within the third area range belong to the vast majority, and in the actual application scenario, the possibility of falling within the third area range is relatively large. Therefore, the sample images falling within the third area range can be directly added to the training image set. Before adding to the training image set, the sample images falling within the third area range can be subjected to simple image processing, such as brightness, contrast, saturation, etc. adjustments. The size of the sample images within this area range is not processed in any way. For example, the third area range can be 150*150-300*300.

[0124] In one embodiment, when the sample object area in the sample image is larger than the fourth area range corresponding to the fourth level sample image processing rule, it means that the sample object size in the sample image is too large, and the possibility of falling into the fourth area range in the actual application scenario is extremely small, and the sample image falling into the fourth area range may be caused by incorrect sample object labeling or other reasons, so the sample image corresponding to the fourth area range needs to be filtered out. For example, if the fourth level sample image processing rule is to filter the sample image, and the fourth area range corresponding to the fourth level sample image processing rule is 300*300, when the sample object area in the sample image is larger than 300*300, the sample image is filtered out.

[0125] In a specific embodiment, an object detection method is provided, which specifically includes the following steps:

[0126] 1. Obtain the annotation data corresponding to the annotation area position corresponding to the training target object in the training image.

[0127] 2. Calculate the initialization value corresponding to the object detection model based on the labeled data.

[0128] 3. Initialize the model parameters of the object detection model according to the initialization values ​​to obtain the initial object detection model.

[0129] 4. Obtain a sample image set, where the sample images in the sample image set include corresponding sample object areas.

[0130] 5. Obtain preset screening rules, which include matching relationships between sample objects in different area ranges and sample image processing rules.

[0131] 6. According to the range and matching relationship of the sample object area in the sample image, obtain the target sample data processing rules corresponding to each sample image, and perform sample image processing on the corresponding sample image according to the target sample data processing rules to obtain training data, wherein the training data includes the marked area position corresponding to the training target object.

[0132] 7. Input the training images in the training image set into the initial object detection model. The initial object detection model extracts features from the training images through a first feature extraction network set to obtain training object state feature information. The first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork.

[0133] 8. The initial object detection model inputs the training object state feature information into the second feature extraction network set to obtain the training position feature information corresponding to the training target object in the training image. The second feature extraction network set includes parallel position feature extraction sub-networks and a second fusion sub-network connected to each parallel position feature extraction sub-network.

[0134] 9. Adjust the model parameters of the initial object detection model according to the training position feature information and the marked area position until the training position feature information output by the initial object detection model meets the convergence condition, thereby obtaining a trained target object detection model.

[0135] 10. Obtain the image to be detected.

[0136] 11. Input the image to be detected into the target object detection model. The target object detection model extracts features from the image to be detected through a first feature extraction network set to obtain object state feature information. The first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork.

[0137] 11-1. Input the image to be detected into the first shared convolutional subnetwork to extract the first shared image features.

[0138] 11-2. Input the first shared image feature into the parallel first convolution subnetwork and the first pooling subnetwork. The first shared convolution subnetwork, the first convolution subnetwork and the first pooling subnetwork include a downsampling process.

[0139] 11-3. The first convolution subnetwork outputs image convolution features, and the first pooling subnetwork outputs image detail information.

[0140] 11-4. The image convolution features and image detail information are concatenated through the first fusion sub-network to obtain object state feature information.

[0141] 12. The target object detection model inputs the object state feature information into the second feature extraction network set to obtain the position feature information corresponding to the target detection object in the image to be detected. The second feature extraction network set includes parallel position feature extraction sub-networks and a second fusion sub-network connected to each parallel position feature extraction sub-network.

[0142] 12-1. Input the object state feature information into the second shared convolutional subnetwork to extract the second shared image feature.

[0143] 12-2. The second shared image features are input into the parallel second convolution subnetwork, the third convolution subnetwork and the second pooling subnetwork. The feature extraction network scales of the second convolution subnetwork and the third convolution subnetwork are different. The second shared convolution subnetwork, the second convolution subnetwork, the third convolution subnetwork and the second pooling subnetwork include a downsampling process.

[0144] 12-3. The second convolution subnetwork outputs the first position convolution feature, the third convolution subnetwork outputs the second position convolution feature, and the second pooling subnetwork outputs the position detail information.

[0145] 12-4. The first position convolution feature, the second position convolution feature and the position detail information are concatenated through the second fusion sub-network to obtain the position feature information.

[0146] 13. Output the location area corresponding to the target detection object based on the location feature information.

[0147] 13-1. Determine the contour of the target detection object in the image to be detected based on the position feature information.

[0148] 13-2. Determine the area range of the target detection object according to the outline of the target detection object.

[0149] 13-3. Determine the center point position of the target detection object based on the position feature information.

[0150] 13-4. Draw the location area and center position of the target detection object based on the area range and center point position.

[0151] Take a small target (not less than 80*80) detection scene in a high-resolution (1080p) scene as an example, the small target is a hand. Specifically, the image to be detected is input into the trained target object detection model. The target object detection model extracts features from the image to be detected through the first feature extraction network set to obtain object state feature information, such as the color of the hand and the outline of the hand. The target object detection model inputs the object state feature information into the second feature extraction network set. The second feature extraction network set determines that the target detection object is a hand based on the object state feature information, and then extracts features from the part of the hand to obtain position feature information, such as the position area where the hand is located and the center point position of the hand. Finally, the target object detection model draws the position area corresponding to the hand based on the position feature information. Among them, if there are multiple hands in the image to be detected, the target object detection model can finally draw the position area where each hand in the image to be detected is located.

[0152] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0153] In one embodiment, Fig. 9 As shown, an object detection device 900 is provided, the device comprising:

[0154] The to-be-detected image acquisition module 902 is used to acquire the to-be-detected image.

[0155] The target object detection model detection module 904 is used to input the image to be detected into the target object detection model. The target object detection model extracts features from the image to be detected through a first feature extraction network set to obtain object state feature information. The first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork.

[0156] The target object detection model detection module 904 is also used for the target object detection model to input the object state feature information into the second feature extraction network set to obtain the position feature information corresponding to the target detection object in the image to be detected. The second feature extraction network set includes parallel position feature extraction sub-networks and a second fusion sub-network connected to each parallel position feature extraction sub-network.

[0157] The position area output module 906 is used to output the position area corresponding to the target detection object according to the position feature information.

[0158] In one embodiment, Fig.10 As shown, the target object detection model detection module 904 includes:

[0159] The first shared convolutional sub-network processing unit 904a is used to input the image to be detected into the first shared convolutional sub-network to extract the first shared image features.

[0160] The first shared image feature processing unit 904b is used to input the first shared image feature into the parallel first convolution subnetwork and the first pooling subnetwork, and the first shared convolution subnetwork, the first convolution subnetwork and the first pooling subnetwork include a downsampling process.

[0161] The image output unit 904c is used for the first convolution sub-network to output image convolution features, and the first pooling sub-network to output image detail information.

[0162] The object state feature information splicing unit 904d is used to perform feature splicing on the image convolution features and the image detail information through the first fusion sub-network to obtain the object state feature information.

[0163] In one embodiment, Fig.11 As shown, the target object detection model detection module 904 also includes:

[0164] The second shared convolutional sub-network processing unit 904e is used to input the object state feature information into the second shared convolutional sub-network to extract the second shared image feature.

[0165] The second shared image feature processing unit 904f is used to input the second shared image feature into the parallel second convolution subnetwork, the third convolution subnetwork and the second pooling subnetwork, the feature extraction network scales of the second convolution subnetwork and the third convolution subnetwork are different, and the second shared convolution subnetwork, the second convolution subnetwork, the third convolution subnetwork and the second pooling subnetwork include a downsampling process.

[0166] The position detail information output unit 904g is used for the second convolution sub-network to output the first position convolution feature, the third convolution sub-network to output the second position convolution feature, and the second pooling sub-network to output the position detail information.

[0167] The position feature information splicing unit 904h is used to perform feature splicing on the first position convolution feature, the second position convolution feature and the position detail information through the second fusion sub-network to obtain the position feature information.

[0168] In one embodiment, Fig.12 As shown, the location area output module 906 includes:

[0169] The contour determining unit 906a is used to determine the contour of the target detection object in the image to be detected according to the position feature information.

[0170] The area range determining unit 906b is used to determine the area range of the target detection object according to the outline of the target detection object.

[0171] The center point position determining unit 906c is used to determine the center point position of the target detection object according to the position feature information.

[0172] The position area drawing unit 906d is used to draw the position area and center position of the target detection object according to the area range and the center point position.

[0173] In one embodiment, Fig.13 As shown, the object detection device 900 also includes:

[0174] The training image set acquisition module 1302 is used to acquire a training image set, where the training images in the training image set include the marked area positions corresponding to the training target objects.

[0175] The initial object detection model processing module 1304 is used to input the training images in the training image set into the initial object detection model. The initial object detection model extracts features from the training images through a first feature extraction network set to obtain training object state feature information. The first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork.

[0176] The initial object detection model processing module 1304 is also used for the initial object detection model to input the training object state feature information into the second feature extraction network set to obtain the training position feature information corresponding to the training target object in the training image. The second feature extraction network set includes parallel position feature extraction sub-networks and a second fusion sub-network connected to each parallel position feature extraction sub-network.

[0177] The initial object detection model training module 1306 is used to adjust the model parameters of the initial object detection model according to the training position feature information and the marked area position until the training position feature information output by the initial object detection model meets the convergence condition, thereby obtaining a trained target object detection model.

[0178] In one embodiment, the object detection device 900 is also used to obtain annotation data corresponding to the position of the annotation area corresponding to the training target object in the training image; calculate the initialization value corresponding to the object detection model based on the annotation data; initialize the model parameters of the object detection model according to the initialization value to obtain an initial object detection model.

[0179] In one embodiment, the object detection device 900 is also used to obtain a sample image set, the sample images in the sample image set include corresponding sample object areas; obtain preset screening rules, the preset screening rules include matching relationships between sample objects in different area ranges and sample image processing rules; obtain target sample data processing rules corresponding to each sample image based on the range and matching relationship of the sample object area in the sample image, perform sample image processing on the corresponding sample image according to the target sample data processing rules to obtain training data, the training data including the marked area position corresponding to the training target object.

[0180] In one embodiment, the object detection device 900 is also used to filter the sample image when the sample object area in the sample image is smaller than the first area range corresponding to the first-level sample image processing rule, and / or when the sample object area in the sample image is larger than the fourth area range corresponding to the fourth-level sample image processing rule; when the sample object area in the sample image is located in the second area range corresponding to the second-level sample image processing rule, obtain a preset multiple, enlarge part of the sample image according to the preset multiple to obtain the enlarged sample image, and form the enlarged sample image into a training image set; when the sample object area in the sample image is located in the third area range corresponding to the third-level sample image processing rule, directly form the sample image into a training image set.

[0181] Fig.14 The internal structure diagram of a computer device in one embodiment is shown. The computer device may specifically be Figure 1 The terminal 110 or the server 120 in FIG. Fig.14As shown, the computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program, which, when executed by the processor, enables the processor to implement an object detection method. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to execute an object detection method. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc. It should be noted that if the computer device is a server 120, the computer device does not include a display screen.

[0182] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0183] In one embodiment, the object detection device provided by the present application can be implemented in the form of a computer program. The computer program can be Fig.14 The computer device is run on the computer device shown. The memory of the computer device can store various program modules constituting the object detection device, such as, Fig. 9 The image acquisition module to be detected, the target object detection model detection module and the position area output module shown in the figure. The computer program composed of each program module enables the processor to execute the steps of the object detection method in each embodiment of the present application described in this specification.

[0184] For example, Fig.14 The computer device shown can be Fig. 9The image acquisition module to be detected in the object detection device shown executes acquisition of the image to be detected. The target object detection model detection module executes inputting the image to be detected into the target object detection model, and the target object detection model extracts features from the image to be detected through the first feature extraction network set to obtain object state feature information, and the first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork. The target object detection model detection module also executes the target object detection model to input the object state feature information into the second feature extraction network set to obtain the position feature information corresponding to the target detection object in the image to be detected, and the second feature extraction network set includes parallel position feature extraction subnetworks and a second fusion subnetwork connected to each parallel position feature extraction subnetwork. The position area output module executes outputting the position area corresponding to the target detection object according to the position feature information.

[0185] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above object detection method. The steps of the object detection method here can be the steps in the object detection method of each embodiment above.

[0186] In one embodiment, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the object detection method described above. The steps of the object detection method here can be the steps of the object detection method in each of the above embodiments.

[0187] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0188] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0189] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An object detection method, comprising: Acquire the image to be detected; The image to be detected is input into a target object detection model, and the target object detection model extracts features from the image to be detected through a first feature extraction network set to obtain object state feature information, wherein the first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork; The target object detection model inputs the object state feature information into a second feature extraction network set to obtain position feature information corresponding to the target detection object in the image to be detected, wherein the second feature extraction network set includes parallel position feature extraction subnetworks and a second fusion subnetwork connected to each parallel position feature extraction subnetwork; Determine the contour of the target detection object in the image to be detected according to the position feature information; Determine the area range of the target detection object according to the outline of the target detection object; Determine the center point position of the target detection object according to the position feature information; The location area and the center position of the target detection object are drawn according to the area range and the center point position.

2. The method according to claim 1, characterized in that: The target object detection model extracts features from the image to be detected through a first feature extraction network set to obtain object state feature information, including: Inputting the image to be detected into a first shared convolutional subnetwork to extract a first shared image feature; Inputting the first shared image feature into a first convolutional subnetwork and a first pooling subnetwork in parallel, wherein the first shared convolutional subnetwork, the first convolutional subnetwork, and the first pooling subnetwork include a downsampling process; The first convolution subnetwork outputs image convolution features, and the first pooling subnetwork outputs image detail information; The image convolution features and the image detail information are feature spliced ​​through the first fusion sub-network to obtain the object state feature information.

3. The method according to claim 1, characterized in that The target object detection model inputs the object state feature information into a second feature extraction network set to obtain position feature information corresponding to the target detection object in the image to be detected, including: Inputting the object state feature information into a second shared convolutional subnetwork to extract a second shared image feature; Inputting the second shared image feature into a parallel second convolutional subnetwork, a third convolutional subnetwork, and a second pooling subnetwork, wherein the feature extraction network scales of the second convolutional subnetwork and the third convolutional subnetwork are different, and the second shared convolutional subnetwork, the second convolutional subnetwork, the third convolutional subnetwork, and the second pooling subnetwork include a downsampling process; The second convolution sub-network outputs a first position convolution feature, the third convolution sub-network outputs a second position convolution feature, and the second pooling sub-network outputs position detail information; The first position convolution feature, the second position convolution feature and the position detail information are feature spliced ​​through the second fusion sub-network to obtain the position feature information.

4. The method according to claim 1, characterized in that: The training steps of the target object detection model include: Acquire a training image set, wherein the training images in the training image set include annotated area positions corresponding to the training target objects; Inputting a training image in the training image set into an initial object detection model, wherein the initial object detection model extracts features from the training image through a first feature extraction network set to obtain training object state feature information, wherein the first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each of the parallel object state feature extraction subnetworks; The initial object detection model inputs the training object state feature information into a second feature extraction network set to obtain training position feature information corresponding to the training target object in the training image, wherein the second feature extraction network set includes parallel position feature extraction subnetworks and a second fusion subnetwork connected to each parallel position feature extraction subnetwork; The model parameters of the initial object detection model are adjusted according to the training position feature information and the position of the marked area until the training position feature information output by the initial object detection model meets the convergence condition, thereby obtaining the trained target object detection model.

5. The method according to claim 4, characterized in that The method further comprises: Acquire annotation data corresponding to the position of the annotation area corresponding to the training target object in the training image; Calculate the initialization value corresponding to the object detection model according to the labeled data; The model parameters of the object detection model are initialized according to the initialization values ​​to obtain an initial object detection model.

6. The method according to claim 4, characterized in that The acquiring of a training image set, wherein the training images in the training image set include the marked area positions corresponding to the training target objects, includes: Acquire a sample image set, wherein the sample images in the sample image set include corresponding sample object areas; Acquire preset screening rules, wherein the preset screening rules include matching relationships between sample objects in different area ranges and sample image processing rules; According to the range of the sample object area in the sample image and the matching relationship, the target sample data processing rules corresponding to each sample image are obtained, and the corresponding sample image is processed according to the target sample data processing rules to obtain training data, wherein the training data includes the marked area position corresponding to the training target object.

7. The method according to claim 6, characterized in that The preset screening rules include first-level sample image processing rules, second-level sample image processing rules, third-level sample image processing rules and fourth-level sample image processing rules. According to the range of the sample object area in the sample image and the matching relationship, the target sample data processing rules corresponding to each sample image are obtained, and the corresponding sample image is processed according to the target sample data processing rules to obtain training data, and the training data includes the position of the marked area corresponding to the training target object, including: When the sample object area in the sample image is smaller than the first area range corresponding to the first level sample image processing rule, and / or When the sample object area in the sample image is larger than the fourth area range corresponding to the fourth level sample image processing rule, filtering the sample image; When the sample object area in the sample image is located in the second area range corresponding to the second-level sample image processing rule, a preset magnification is obtained, and part of the sample image is magnified according to the preset magnification to obtain an enlarged sample image, and the enlarged sample image is used to form the training image set; When the sample object area in the sample image is located in the third area range corresponding to the third-level sample image processing rule, the sample image is directly incorporated into the training image set.

8. An object detection device, characterized in that: The device comprises: An image acquisition module to be detected is used to acquire the image to be detected; A target object detection model detection module, used for inputting the image to be detected into the target object detection model, wherein the target object detection model extracts features from the image to be detected through a first feature extraction network set to obtain object state feature information, wherein the first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each parallel object state feature extraction subnetwork; The target object detection model detection module is also used for the target object detection model to input the object state feature information into a second feature extraction network set to obtain position feature information corresponding to the target detection object in the image to be detected, wherein the second feature extraction network set includes parallel position feature extraction subnetworks and a second fusion subnetwork connected to each parallel position feature extraction subnetwork; Location area output module, including: A contour determining unit, used to determine the contour of the target detection object in the image to be detected according to the position feature information; An area range determining unit, configured to determine the area range of the target detection object according to the contour of the target detection object; A center point position determining unit, used to determine the center point position of the target detection object according to the position feature information; A position area drawing unit is used to draw the position area and center position of the target detection object according to the area range and the center point position.

9. The device according to claim 8, characterized in that The target object detection model detection module includes: A first shared convolutional sub-network processing unit, used for inputting the image to be detected into the first shared convolutional sub-network to extract a first shared image feature; A first shared image feature processing unit, configured to input the first shared image feature into a first convolutional subnetwork and a first pooling subnetwork in parallel, wherein the first shared convolutional subnetwork, the first convolutional subnetwork, and the first pooling subnetwork include a downsampling process; An image output unit, configured for the first convolution subnetwork to output image convolution features, and the first pooling subnetwork to output image detail information; The object state feature information splicing unit is used to perform feature splicing on the image convolution feature and the image detail information through the first fusion sub-network to obtain the object state feature information.

10. The device according to claim 8, characterized in that The target object detection model detection module includes: A second shared convolutional sub-network processing unit, used for inputting the object state feature information into the second shared convolutional sub-network to extract a second shared image feature; A second shared image feature processing unit, configured to input the second shared image feature into a parallel second convolutional subnetwork, a third convolutional subnetwork, and a second pooling subnetwork, wherein the feature extraction network scales of the second convolutional subnetwork and the third convolutional subnetwork are different, and the second shared convolutional subnetwork, the second convolutional subnetwork, the third convolutional subnetwork, and the second pooling subnetwork include a downsampling process; A position detail information output unit, configured for the second convolution sub-network to output the first position convolution feature, the third convolution sub-network to output the second position convolution feature, and the second pooling sub-network to output the position detail information; A position feature information splicing unit is used to perform feature splicing on the first position convolution feature, the second position convolution feature and the position detail information through the second fusion sub-network to obtain the position feature information.

11. The device according to claim 8, characterized in that The device also includes: A training image set acquisition module, used to acquire a training image set, wherein the training images in the training image set include the marked area positions corresponding to the training target objects; an initial object detection model processing module, configured to input a training image in the training image set into an initial object detection model, wherein the initial object detection model extracts features from the training image through a first feature extraction network set to obtain training object state feature information, wherein the first feature extraction network set includes parallel object state feature extraction subnetworks and a first fusion subnetwork connected to each of the parallel object state feature extraction subnetworks; The initial object detection model processing module is also used for the initial object detection model to input the training object state feature information into a second feature extraction network set to obtain training position feature information corresponding to the training target object in the training image, wherein the second feature extraction network set includes parallel position feature extraction subnetworks and a second fusion subnetwork connected to each parallel position feature extraction subnetwork; The initial object detection model training module is used to adjust the model parameters of the initial object detection model according to the training position feature information and the position of the marked area until the training position feature information output by the initial object detection model meets the convergence condition, thereby obtaining the trained target object detection model.

12. The device according to claim 11, characterized in that The device also includes: Acquire annotation data corresponding to the position of the annotation area corresponding to the training target object in the training image; Calculate the initialization value corresponding to the object detection model according to the labeled data; The model parameters of the object detection model are initialized according to the initialization values ​​to obtain an initial object detection model.

13. The device according to claim 11, characterized in that The acquiring of a training image set, wherein the training images in the training image set include the marked area positions corresponding to the training target objects, includes: Acquire a sample image set, wherein the sample images in the sample image set include corresponding sample object areas; Acquire preset screening rules, wherein the preset screening rules include matching relationships between sample objects in different area ranges and sample image processing rules; According to the range of the sample object area in the sample image and the matching relationship, the target sample data processing rules corresponding to each sample image are obtained, and the corresponding sample image is processed according to the target sample data processing rules to obtain training data, wherein the training data includes the marked area position corresponding to the training target object.

14. The device according to claim 13, characterized in that The preset screening rules include first-level sample image processing rules, second-level sample image processing rules, third-level sample image processing rules and fourth-level sample image processing rules. According to the range of the sample object area in the sample image and the matching relationship, the target sample data processing rules corresponding to each sample image are obtained, and the corresponding sample image is processed according to the target sample data processing rules to obtain training data, and the training data includes the position of the marked area corresponding to the training target object, including: When the sample object area in the sample image is smaller than the first area range corresponding to the first level sample image processing rule, and / or When the sample object area in the sample image is larger than the fourth area range corresponding to the fourth level sample image processing rule, filtering the sample image; When the sample object area in the sample image is located in the second area range corresponding to the second-level sample image processing rule, a preset magnification is obtained, and part of the sample image is magnified according to the preset magnification to obtain an enlarged sample image, and the enlarged sample image is used to form the training image set; When the sample object area in the sample image is located in the third area range corresponding to the third-level sample image processing rule, the sample image is directly incorporated into the training image set.

15. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

16. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

Images