Image mask generation method, device, electronic device and storage medium

Abstract

The present application discloses an image mask generation method, device, electronic device and storage medium. The image mask generation method comprises: obtaining image data containing mask features, wherein the mask features are mask features of an image including an object to be identified; obtaining center point information and identification frame parameters of the object to be identified on the image data; extracting the mask features of the object to be identified from the image data based on the center point information and identification frame information; and generating an image mask of the object based on the mask features of the object to be identified extracted from the image data. The present invention pools image data of multiple feature dimensions based on the object key point information and identification frame parameters extracted by the backbone network, so that the mask features belonging to the identified object can be identified from the complete image, thereby accurately extracting the mask of the identified object.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to an image mask generation method, device, electronic device and storage medium. Background Art

[0002] Image recognition technology has been widely used in the commercial field. Currently, the commonly used image recognition methods either generate segmented instance segments by using sliding windows to select and assemble the output of the score map, or directly predict bounding boxes without suggestions based on the detector. These methods rely heavily on predefined anchors, and hyperparameter adjustment (such as anchor ratio, anchor step) is crucial for different data sets and box scales. Some image recognition methods use the key point detection idea to obtain the four extreme points of the object and generate a mask, or use the polar coordinate representation method to re-segment the instance and predict the center of mass of the target, and then predict the recognition box based on the distance between the center of mass and the dense contour points. In addition, after obtaining the recognition box, the traditional method also includes the step of eliminating redundant detection boxes at the category level to avoid multiple recognition boxes on the same item. In addition, the current mainstream image recognition algorithms usually use deep backbone networks to process image data. The commonly used backbone networks use huge parameters for accuracy, resulting in low model inference speed and severely limiting the implementation of the model on low-memory devices. Some backbone networks focus on improving the model inference speed, but reduce the accuracy.

[0003] These traditional methods either have complex parameter adjustments or are only applicable to generating recognition frames or masks. In addition, the redundant frame detection technology used in traditional methods will cause serious missed detection when used in some special industrial scenarios, such as multiple tilted objects in parallel or objects with obstructions. However, in industrial scenarios, such as using robots to grasp objects, missed detection of objects is intolerable, and masks and recognition frames are necessary information for subsequent processing. Existing image recognition technology still needs to overcome many problems when applied to industrial scenarios. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to overcome the above problems or at least partially solve the above problems. Specifically, firstly, the present invention can perform the processing of generating a recognition frame and the processing of generating a mask in parallel based on the key point information of the object and the recognition frame parameters. When used in industrial scenarios, the recognition is accurate, no redundant recognition frames are generated, and the recognition frame and the mask are generated at the same time, which is highly practical; secondly, the backbone network proposed in the present invention can process the input image data in multiple stages of two processing processes with multiple resolutions and multiple feature dimensions, and only perform upsampling processing in the second processing process, thereby ensuring the high resolution of the output feature image data. The backbone network of the present invention improves the inference speed while maintaining the accuracy; thirdly, in addition to the data processing flow, the backbone network of the present invention also includes feature transition processing and residual processing, which can ensure a smooth feature transition under high-level features. The present invention can avoid the gradient loss of the deep network and improve the accuracy of the backbone network reasoning; thirdly, the mask generation process of the present invention obtains feature image data of multiple high-level dimensional features, and extracts mask features from each feature dimension by pooling, thereby ensuring the integrity of the generated image mask and preventing the occurrence of mask incompleteness; thirdly, the present invention pools the image data of multiple feature dimensions based on the object key point information extracted by the backbone network and the recognition frame parameters, thereby being able to identify the mask features belonging to the identified object from the complete image and accurately extract the mask of the identified object; thirdly, based on the general image recognition method, the present invention proposes an image recognition method that is particularly suitable for identifying multiple inclined and parallel objects, which can improve the accuracy of identifying multiple objects and avoid missed detections.

[0005] All the solutions disclosed in the claims and the specification of this application have one or more of the above-mentioned innovations, and accordingly, can solve one or more of the above-mentioned technical problems. Specifically, this application provides an image mask generation method, device, electronic device and storage medium.

[0006] The image mask generation method of the embodiment of the present application includes:

[0007] Acquire image data containing mask features, wherein the mask features are mask features of an image including an object to be identified;

[0008] Obtaining the center point information and identification frame parameters of the object to be identified on the image data;

[0009] Extracting mask features of the object to be identified from the image data based on the center point information and the identification frame information;

[0010] An image mask of the object is generated based on the mask features of the object to be identified extracted from the image data.

[0011] In some embodiments, the identification frame parameters include width information of the identification frame and height information of the identification frame.

[0012] In certain embodiments, the center point comprises a true center point.

[0013] In some embodiments, the image data containing mask features includes feature image data obtained from a backbone network and after fusion.

[0014] In some embodiments, the fusion includes a channel-level addition operation and / or a channel-level superposition operation.

[0015] In some embodiments, the backbone network includes a first data processing process and a second data processing process, the data processing process includes one or more processing stages, and the data stage includes one or more processing branches.

[0016] In some embodiments, during the first processing, a later processing stage includes more processing branches than a previous processing stage.

[0017] In certain embodiments, during the second processing, a later processing stage includes fewer processing branches than a previous processing stage.

[0018] In some embodiments, the last processing stage of the first processing process has the same number of processing branches as the first processing stage of the second processing process.

[0019] The image mask generating device according to the embodiment of the present application includes:

[0020] An image data acquisition module, used to acquire image data containing mask features, wherein the mask features are mask features of an image including an object to be identified;

[0021] An information acquisition module, used to acquire the center point information and identification frame parameters of the object to be identified on the image data;

[0022] A mask feature acquisition module, used to extract the mask feature of the object to be identified from the image data based on the center point information and the identification frame information;

[0023] The mask generation module is used to generate an image mask of the object based on the mask features of the object to be identified extracted from the image data.

[0024] In some embodiments, the identification frame parameters include width information of the identification frame and height information of the identification frame.

[0025] In certain embodiments, the center point comprises a true center point.

[0026] In some embodiments, the image data containing mask features includes feature image data obtained from a backbone network and after fusion.

[0027] In some embodiments, the fusion includes a channel-level addition operation and / or a channel-level superposition operation.

[0028] In some embodiments, the backbone network includes a first data processing process and a second data processing process, the data processing process includes one or more processing stages, and the data stage includes one or more processing branches.

[0029] In some embodiments, during the first processing, a later processing stage includes more processing branches than a previous processing stage.

[0030] In certain embodiments, during the second processing, a later processing stage includes fewer processing branches than a previous processing stage.

[0031] In some embodiments, the last processing stage of the first processing process has the same number of processing branches as the first processing stage of the second processing process.

[0032] The electronic device of the embodiment of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the image mask generation method of any of the above embodiments is implemented.

[0033] The computer-readable storage medium of the embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, the image mask generation method of any of the above embodiments is implemented.

[0034] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0036] Figure 1 is a flowchart of an image recognition method in certain embodiments of the present application;

[0037] Figure 2 is a flowchart of an image data processing method of certain embodiments of the present application;

[0038] Figure 3 is a schematic diagram of the structure of a backbone network of certain embodiments of the present application;

[0039] Figure 4 is a flowchart of a mask generation method for a mask branch in certain embodiments of the present application;

[0040] Figure 5 is a schematic flow chart of a mask generation method according to certain embodiments of the present application;

[0041] Figure 6 is a flow chart of an image recognition method for a plurality of tilted and parallel objects according to certain embodiments of the present application;

[0042] FIG7 is a schematic diagram of image recognition results of certain embodiments of the present application and image recognition results of the prior art;

[0043] FIG8 is a schematic diagram of another set of image recognition results of certain embodiments of the present application and image recognition results of the prior art;

[0044] FIG9 is a schematic diagram of an area where identification frames intersect and an area formed after the identification frames are merged;

[0045] Fig.10 is a schematic diagram of the structure of an image recognition device in some embodiments of the present application;

[0046] Fig.11 is a schematic diagram of the structure of an image data processing device in some embodiments of the present application;

[0047] Fig.12 is a structural schematic diagram of an image data processing device including a feature transition module and a residual connection module according to certain embodiments of the present application;

[0048] Fig.13 is a schematic structural diagram of a mask generation device of a mask branch in certain embodiments of the present application;

[0049] Fig.14 is a schematic structural diagram of a mask generating device according to some embodiments of the present application;

[0050] Fig.15 is a schematic structural diagram of an image recognition device for multiple tilted and parallel objects according to certain embodiments of the present application;

[0051] Fig.16 It is a schematic diagram of the structure of an electronic device of certain embodiments of the present application. DETAILED DESCRIPTION

[0052] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0053] Figure 1 FIG. 4 is a flow chart showing an image recognition method according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0054] Step S100, acquiring image data containing an object to be identified;

[0055] Step S110, processing the image data to identify the object to be identified in the image data, and obtaining key point information and identification frame parameters of the object to be identified;

[0056] Step S120, for the identified object, generating an identification frame on the image data based on the key point information and the identification frame parameters; and

[0057] Step S130 : generating a mask of the identified object based on the key point information and the identification frame parameters.

[0058] For step S100, the object to be identified in the present invention can be any object placed in any way. Compared with other existing methods, the present invention has a more obvious detection effect for side-by-side tilted objects in dense scenes. The image data in the present invention can be taken on-site or pre-saved and manually annotated data.

[0059] Compared with the traditional method, the image recognition method proposed in this embodiment does not use a predefined anchor frame and complex parameters and calculations related to the anchor frame. Instead, it obtains the key point information of the object to be identified and the identification frame parameters to efficiently and accurately generate an identification frame for marking the object. In addition, the method of this embodiment generates a single identification frame for a single object, does not generate redundant identification frames, and does not need to use redundant frame detection technology. Therefore, it can be applied to all industrial scenes including scenes such as multiple parallel tilted objects or objects with obstructions, and will not cause missed detection. In addition, this method performs the identification frame generation operation and the mask generation operation in parallel, and has strong practicality in industrial scenes. As a preferred embodiment, the key point can be the midpoint of the object, and the identification frame parameters can include the width and length of the identification frame.

[0060] For step S110, the image data is input into the backbone network for processing to identify the objects in the image data and obtain the key point information of the objects to be identified and the identification frame parameters. The backbone network is used to process the input data. For different task objectives, a backbone network suitable for the task objective can be selected to process the data. For example, some backbone networks are suitable for recognizing graphics, some backbone networks are suitable for recognizing faces, and some backbone networks are suitable for recognizing text. As described above, the focus of this embodiment is to perform object recognition operations and object mask generation operations in parallel based on key point information and identification frame parameters. The backbone network is used to identify objects in the image data and obtain the key point information of the objects to be identified and the identification frame parameters. As long as the backbone network can achieve the above functions, it can be used in the image recognition method of this embodiment. This embodiment does not limit the selection of the backbone network.

[0061] Some of the currently commonly used backbone networks for image recognition methods use huge parameters to achieve high performance, which severely limits the implementation of the model on low-memory devices. On the contrary, some focus on improving the model reasoning speed but reduce the accuracy. To this end, we propose a new backbone network for image recognition, which can significantly improve the processing speed of the network while maintaining accuracy. This new backbone network is one of the key points of the present invention, and it can be used in any image recognition method. The image recognition method of the present invention preferably uses the new backbone network to process the input image data

[0062] Figure 2 FIG. 2 shows a schematic diagram of a process for image data processing using a novel backbone network according to an embodiment of the present invention. Figure 2 As shown, the method includes:

[0063] Step S200, receiving image data to be processed;

[0064] Step S210, using a first data processing process to process the image data to be processed;

[0065] Step S220: Use the second data processing process to process the image data processed by the first data processing process.

[0066] In order to facilitate the explanation of the backbone network of the present invention, Figure 3 The structure of the novel backbone network of the present invention is schematically shown. Figure 3 As shown, the network includes two main parts: a first data processing process and a second data processing process. According to the needs of the actual application scenario, each data processing process can include one or more processing stages, and each processing stage can include one or more parallel processing branches ( Figure 3The processing branch is shown in the form of a "block", and the "block" and "convolution block" mentioned below all refer to the processing branch). As an example, the first data processing process may include three stages, the first stage, the second stage and the third stage. The first stage includes a processing branch, which includes multiple convolution processes of equal resolution. The convolution process uses a convolution kernel size of 3x3 and a step size of 1, and then a 1x1 convolution layer is used for downsampling. The second stage includes two processing branches, one processing branch receives the feature image data output by the processing branch of the first stage and repeats the processing process of the first stage, and the other branch performs convolution processing on the feature image data output by the first stage after downsampling. The third stage includes three processing branches, which include both repeating the operation of the processing branch of the previous stage and convolution processing on the feature image data output by the two processing branches of the second stage after downsampling / upsampling. In addition, the data input to the third stage has also undergone multi-resolution and multi-feature dimensional fusion.

[0067] After the third stage, the second data processing process begins. The second data processing process can also include three stages: the fourth stage, the fifth stage, and the sixth stage. The fourth stage includes three processing branches. Similar to the third stage, these three processing branches include both repeating the operations of the processing branches of the previous stage and convolution processing of the feature image data of the three processing branches of the third stage that have been downsampled / upsampled. Similarly, the input data of the fourth stage has also undergone multi-resolution and multi-feature dimensional fusion. The image data is convoluted at each stage. The more convolution processes are performed, the higher the level of features contained in the output feature image data. When the feature image data enters the second data processing process, it has already gone through three stages of processing. Therefore, the feature image data output from the fourth stage contains quite high-level features. In order to improve the accuracy of image recognition, it is hoped that the feature dimension can be smoothly increased so that the processed feature image data loses as little information as possible. Our backbone network adds a feature transition module from the fourth stage. Before the current processing branch outputs data to a processing branch with more feature dimensions, the output of the current processing branch is adjusted through the feature transition module. That is to say, an additional feature conversion module is extended after the main convolution block, and the feature conversion module doubles the previous feature dimension. In one embodiment, in order to enhance feature diversity, a deformable convolution is used as the convolution layer of the feature conversion block.

[0068] The method of the present invention generates an identification frame based on the key point information of the object, and the resolution of the image data cannot be too low, especially for two objects close to each other. Therefore, in the second processing process, no operation that will reduce the resolution of the image data is performed. In other words, in the first data processing process, some processing stages include processing to increase the resolution of the image data, and some processing stages include processing to reduce the resolution of the image data, while in the second data processing process, any processing stage only includes processing to increase the resolution of the image data to amplify the lower-level features, and does not include processing to reduce the resolution of the image data. The present invention preferably uses upsampling operations to increase the resolution of the image data and uses downsampling operations to reduce the resolution. Therefore, after the fourth and fifth stages, only the feature image data is upsampled, and in the fifth and sixth stages, only the operation of repeating the processing branch of the previous stage and the operation of convolution processing the feature image data output by the upsampled processing branch are included, and the operation of convolution processing the feature image data output by the downsampled processing branch is not included. On the whole, the number of processing branches in each processing stage of the first data processing process of the new backbone network gradually increases, presenting a "descending triangle" structure; the number of processing branches in each processing stage of the second data processing process gradually decreases, presenting an "ascending triangle" structure, and the new backbone network cross-integrates the data output by multiple processing branches in multiple feature dimensions and multiple resolutions. This architecture constructs high-to-low and low-to-high convolutions in parallel, while maintaining high resolution of the entire process, and performing multiple fusions with features of different dimensions.

[0069] The following takes an input image data with a resolution of 512x512 as an example to explain how the backbone network of the present invention processes image data.

[0070] For step S210, the image data with a resolution of 512x512 is input into the processing branch of the first processing stage of the first processing process. After being processed by the processing branch, feature image data with a resolution of 128x128 in 32 feature dimensions is obtained. On the one hand, the feature image data output by the first processing branch is input into the upper processing branch of the second processing stage. On the other hand, the data output by the first processing branch is downsampled by 1x1 convolution and output to the middle processing branch of the second processing stage. It should be noted that the "upper" and "middle" here refer to the Figure 3In the figure, the processing branch is located at the top or in the middle, which does not mean that the upper processing branch processes data before the middle processing branch. In fact, multiple processing branches in each processing stage are executed in parallel without any order. In the entire backbone network, the resolution of image data processed by the upper processing branch is 128x128, the resolution of image data processed by the middle processing branch is 64x64, and the resolution of image data processed by the lower processing branch is 32x32.

[0071] After being processed by the upper processing branch of the second processing stage, feature image data of 32 feature dimensions are obtained, and after being processed by the middle processing branch, feature image data of 64 feature dimensions are obtained. On the one hand, the feature image data output by the upper processing branch is fused with the feature image data output by the middle processing branch after 1x1 convolution upsampling, and then input into the upper branch of the third processing stage; on the second hand, the feature image number output by the upper processing branch after 1x1 convolution downsampling is fused with the feature image data output by the middle processing branch, and then input into the middle processing branch of the third processing stage; on the third hand, the feature image number after 1x1 convolution downsampling of the upper processing branch is fused with the feature image data output by the middle processing branch after 1x1 convolution downsampling, and then input into the lower branch of the third processing stage.

[0072] After being processed by the upper processing branch of the third processing stage, feature image data of 32 feature dimensions are obtained, after being processed by the middle processing branch, feature image data of 64 feature dimensions are obtained, and after being processed by the lower processing branch, feature image data of 128 feature dimensions are obtained. On the one hand, the feature image data output by the upper processing branch is fused with the feature image data output by the middle processing branch after 1x1 convolution upsampling and the feature image data output by the lower processing branch after 1x1 convolution upsampling, and then input into the upper branch of the fourth processing stage; on the second hand, the number of feature images output by the upper processing branch after 1x1 convolution downsampling is fused with the feature image data output by the middle processing branch and the feature image data output by the lower processing branch after 1x1 convolution upsampling, and then input into the middle processing branch of the fourth processing stage; on the third hand, the number of feature images after 1x1 convolution downsampling of the upper processing branch is fused with the feature image data output by the middle processing branch after 1x1 convolution downsampling and the feature image data output by the lower processing branch, and then input into the lower branch of the fourth processing stage.

[0073] For step S220, after being processed by the upper processing branch of the fourth processing stage, feature image data of 32 feature dimensions are obtained, and then after being processed by the feature transition module, feature image data of 64 feature dimensions are obtained, and after being processed by the middle processing branch, feature image data of 64 feature dimensions are obtained, and then after being processed by the feature transition module, feature image data of 128 feature dimensions are obtained, and after being processed by the lower processing branch, feature image data of 128 feature dimensions are obtained. On the one hand, the feature image data output by the feature transition module after the upper processing branch is fused with the feature image data output by the middle processing branch after 1x1 convolution upsampling and the feature image data output by the lower processing branch after 1x1 convolution upsampling, and then input to the upper branch of the fifth processing stage; on the other hand, the feature image data output by the feature transition module after the middle processing branch is fused with the feature image data output by the lower processing branch after 1x1 convolution upsampling, and then input to the middle processing branch of the fifth processing stage.

[0074] After being processed by the upper processing branch of the fifth processing stage, feature image data of 64 feature dimensions are obtained, and then after being processed by the feature transition module, feature image data of 128 feature dimensions are obtained, and after being processed by the middle processing branch, feature image data of 128 feature dimensions are obtained. The feature image data output by the feature transition module after the upper processing branch is fused with the feature image data output by the middle processing branch after 1x1 convolution upsampling, and then input into the upper branch of the sixth processing stage. After being processed by the upper processing branch of the sixth processing stage, feature image data of 256 feature dimensions are obtained.

[0075] Since the backbone network of the present invention may include many processing stages, the more processing stages there are, the deeper the network is, and the more likely it is to lose information during the image data processing process. In one embodiment, in order to prevent the gradient loss of the deep network and strengthen the head and tail features, the new backbone network of the present invention may also include a residual connection module, and one or more processing branches in the first data processing process may input data processed by the residual processing process to one or more processing branches in the second data processing process through the residual processing module. Figure 3 As shown, as an example, two residual connection modules of the present invention are shown above the entire network, one residual connection module connects the processing branch of the first processing stage with the processing branch of the sixth processing stage, and the other residual connection module connects the upper processing branch of the second processing stage with the upper processing branch of the fifth processing stage, so as to break through the information blockage between low-level and high-level features and avoid information loss during the processing process.

[0076] For step S120, based on the recognition results of the backbone network and the data obtained during the backbone network processing, the center point information of the object to be identified is extracted and the length information and width information of the identification box used to mark the object are calculated, and an identification box is generated on the image data to mark the identified object.

[0077] For step S130, the image recognition method of the present invention consists of two parts, generating a recognition frame and predicting an object mask. Both parts require feature image data obtained during data processing using a backbone network. Therefore, in addition to the main process of using a backbone network to generate a recognition frame, the present invention also simultaneously performs a mask branch processing process. In one embodiment, the operation of generating a recognition frame shares feature weights with the operation of mask generation. The mask branch processing process makes full use of the feature image data generated during the backbone network processing of image data. It is one of the key points of the present invention to fuse feature image data with multiple feature dimensions to extract the mask of the object.

[0078] Figure 4 FIG. 4 shows a mask generation method used by a mask generation branch in an image recognition method according to an embodiment of the present invention. Figure 4 As shown, the method includes:

[0079] Step S300, receiving image data to be processed;

[0080] Step S310, inputting the image data to be processed into a data processing process including M processing stages for processing, where M is an integer greater than or equal to 2;

[0081] Step S320, obtaining data output by N processing stages out of the M processing stages, where N is an integer greater than or equal to 2 and less than or equal to M;

[0082] Step S330, fusing the data output from the N processing stages;

[0083] Step S340: pooling the fused data to obtain an image mask.

[0084] For step S310, as mentioned above, the present invention needs to input the image data into the backbone network for processing, and the backbone network can preferably use the novel backbone network of the present invention. The backbone network includes multiple processing stages. For the specific processing process, please refer to the related embodiments of the backbone network mentioned above, which will not be repeated here.

[0085] For step S320, in order to accurately and efficiently extract the mask of the object, this embodiment obtains a plurality of feature image data generated during the backbone network data processing, for example, Figure 3The backbone network shown includes 6 data processing stages, and the mask branch receives feature image data generated by 3 of the stages, specifically the feature image data generated by the 4th, 5th and 6th stages.

[0086] For step S330, an example of the mask branch performing feature image data fusion is shown in Figure 3 In the lower part of , it can be seen that the 4th processing stage inputs feature image data of 128 feature dimensions to the mask branch, the 5th processing stage inputs feature image data of 128 feature dimensions to the mask branch, and the channel-level superposition calculation is used between the 4th and 5th stages to obtain feature image data of 256 feature dimensions; the 6th processing stage inputs feature image data of 256 feature dimensions to the mask branch, and the channel-level addition calculation is used between the 5th and 6th stages to obtain new feature image data of 256 feature dimensions. In other words, the channel-level superposition calculation between data with a dimension lower than the maximum feature dimension of the backbone network is equal to the channel-level addition calculation between data with the maximum feature dimension of the backbone network.

[0087] For step S340, the present invention processes the fused feature image data using a pooling method to obtain the mask of the object. There are many methods for obtaining the mask of the object using a pooling method in the prior art. The focus of this embodiment is to extract and fuse data from multiple data processing stages, and then pool the new feature image data containing mask features to obtain the image mask, rather than the specific pooling method. Any suitable pooling method for generating a mask can be used in this embodiment.

[0088] Although any pooling method can be used, in order to improve the accuracy of mask generation, the present invention develops a mask generation method based on a center point, which is particularly suitable for a method for image recognition based on a center point, which is also one of the key points of the present invention. Figure 5 FIG. 4 shows a method for generating an image mask according to an embodiment of the present invention. Figure 5 As shown, the method includes:

[0089] Step S400, acquiring image data containing mask features, wherein the mask features are mask features of an image including an object to be identified;

[0090] Step S410, obtaining the center point information and identification frame information of the object to be identified on the image data;

[0091] Step S420, extracting mask features of the object to be identified from the image data based on the center point information and the identification frame information;

[0092] Step S430: generating an image mask of the object to be identified based on the mask features of the object to be identified extracted from the image data.

[0093] For step S400, the image data processed by the backbone network is image data containing the object for which the mask is to be generated, other objects, and background images. The method of this embodiment extracts the mask features of the object from as many high-level feature dimensions as possible. Therefore, preferably, Figure 3 As shown, the present invention extracts image data of 256 feature dimensions from multiple processing stages of the second data processing process, and these feature image data contain mask features of the object to be identified.

[0094] For step S410, based on the data obtained during the backbone network processing, the center point information of the object to be identified is extracted and the length and width information of the identification box used to mark the object are calculated. This information can be used to generate the identification box as well as the mask of the object to be identified. The process of the backbone network processing data is not described in detail here. In a preferred embodiment, the center point can be the ground truth center point, referred to as the GT center point.

[0095] For step S420, as mentioned above, the image data obtained is complete image data including the object to be identified, and contains feature image data of 256 feature dimensions. In this step, on the one hand, it is necessary to find the position of the object to be identified based on the center point information and the length information and width information of the identification frame, and on the other hand, it is necessary to find the object mask features from the feature image data; thereby, in the next step, a mask of the object to be identified is generated based on the obtained mask features of the object to be identified, so as to facilitate various operations based on the mask.

[0096] The image recognition method of the present invention is particularly suitable for industrial scenarios such as recognizing multiple objects that are tilted or blocked. How to use the image recognition method of the present invention in such industrial scenarios is also one of the key points of the present invention.

[0097] Figure 6 A method for identifying and marking image data including multiple objects according to an embodiment of the present invention is shown. Figure 6 As shown, the method includes:

[0098] Step S500, acquiring image data containing a group of objects to be identified;

[0099] Step S510, identifying each object in the group of objects to be identified based on the image data;

[0100] Step S520: for each identified object, generate an identification box on the image data to mark the object.

[0101] For step S500, FIGS. 7 and 8 exemplarily show some image data, in which a plurality of obliquely placed objects to be operated are included, and these objects constitute an object group;

[0102] For step S510, each object needs to be identified from the image data. The present invention identifies all objects through the aforementioned image recognition method, which requires inputting the image data into the backbone network for processing. The data processing method of the backbone network is not described in detail here.

[0103] For step S520, in industrial scenarios, all items in the group of objects in the image data are operation objects, which may need to be grabbed or painted, so it is usually necessary to identify all items in the group of objects without omission. Figures 7(a) and 8(a) are the results of identification using existing identification methods, and Figures 7(b) and 8(b) are the results of identification using the identification method of the present invention. The present invention generates an identification frame for each identified object based on the center point information of the object and the length and width information of the identification frame to mark the object, so the center of each generated identification frame is located on the identified object. In addition, in the prior art, when the overlap between two identification frames exceeds a certain threshold, one of the frames is usually determined to be a redundant identification frame, and the identification frame is deleted in the output image, resulting in the identification results shown in Figures 7(a) and 8(a), that is, many missed detections occur. Using the recognition method of the present invention, the recognition results shown in Figures 7(b) and 8(b) can be generated. That is to say, the image recognition method of the present invention can tolerate partial or even complete overlap of the recognition frames. Even if the overlap reaches more than 60%, the present invention will not delete the recognition frame, but will correctly identify each object without missing any objects. The overlap between two recognition frames = the area of ​​the region where the two recognition frames intersect / the area of ​​the region formed by merging the two recognition frames. As shown in Figure 9, the black portion in Figure 9(a) is the area of ​​the region where the two recognition frames intersect, and the black portion in Figure 9(b) is the area of ​​the region formed by merging the two recognition frames.

[0104] In addition, it should be noted that, although each embodiment of the present invention has a specific combination of features, further combinations and cross-combinations of these features between the embodiments are also feasible.

[0105] According to the above embodiments, firstly, the present invention can perform the processing of generating the recognition frame and the processing of generating the mask in parallel based on the key point information of the object and the recognition frame parameters. When used in industrial scenarios, no redundant recognition frames will be generated, and the practicability is strong. Secondly, the backbone network proposed in the present invention can process the input image data in multiple stages of two processing processes, and only perform upsampling processing in the second processing process, thereby ensuring the high resolution of the output feature image data. Thirdly, in addition to the data processing flow, the backbone network of the present invention also includes feature transition processing and residual processing, which can ensure the smooth feature transition under high-level features. The invention can also avoid the gradient loss of the deep network; fourthly, the mask generation process of the invention obtains the feature image data of multiple high-level dimensional features, and extracts the mask features from each feature dimension by pooling, so that the image mask can be completely generated without omission; fifthly, the invention pools the image data of multiple feature dimensions based on the key point information of the object extracted by the backbone network and the recognition frame parameters, so that the mask of the place identification object can be accurately extracted from the complete image; sixthly, based on the general image recognition method, the invention proposes a method specifically for identifying multiple inclined and parallel objects, which can improve the recognition accuracy and avoid missed detection.

[0106] Fig.10 An image recognition device according to another embodiment of the present invention is shown, the device comprising:

[0107] An image data acquisition module 600 is used to acquire image data containing the object to be identified, that is, to implement step S100;

[0108] The image data processing module 610 is used to process the image data to identify the object to be identified in the image data, and obtain key point information and identification frame parameters of the object to be identified, that is, to implement step S110;

[0109] The identification frame generation module 620 generates an identification frame on the image data based on the key point information and the identification frame parameters for the identified object, that is, for implementing step S120; and

[0110] The mask generation module 630 generates a mask of the identified object based on the key point information and the identification frame parameters, that is, is used to implement step S130. Fig.11 A device for processing image data according to another embodiment of the present invention is shown, the device comprising:

[0111] An image data receiving module 700, used to receive image data to be processed, that is, to implement step S200;

[0112] A first data processing module 710, used for processing the image data to be processed using a first data processing process, that is, for implementing step S210;

[0113] A second data processing module 720, used for processing the image data processed by the first data processing process using a second data processing process, that is, for implementing step S220;

[0114] The data processing process includes one or more processing stages. In the first data processing process, some processing stages include processing to increase the resolution of image data, and some processing stages include processing to reduce the resolution of image data; in the second data processing process, any processing stage includes processing to increase the resolution of image data, and does not include processing to reduce the resolution of image data.

[0115] Fig.12 A device for processing image data according to another embodiment of the present invention is shown, the device comprising:

[0116] An image data receiving module 800 receives image data to be processed, i.e., is used to implement step S200;

[0117] A first data processing module 810 processes the image data to be processed using a first data processing process, that is, is used to implement step S210;

[0118] A second data processing module 820, which uses a second data processing process to process the image data processed by the first data processing process, that is, to implement step S220;

[0119] Wherein, the data processing process includes one or more processing stages, and each processing stage includes one or more processing branches;

[0120] The processing device also includes:

[0121] A residual processing module 830, configured to connect one or more processing branches in the first data processing process with one or more processing branches in the second data processing process through a residual processing process;

[0122] The feature transition module 840 is used to perform feature transition processing on the data to be output before one or more processing branches in the second processing process output the data to the next processing stage.

[0123] As for the residual processing module 830, since the backbone network of the present invention may include many processing stages, the more processing stages there are, the deeper the network is, and the more likely it is to lose information during the image data processing process. In one embodiment, in order to prevent the gradient loss of the deep network and strengthen the head and tail features, the new backbone network of the present invention may also include a residual connection module, and one or more processing branches in the first data processing process may input the data processed by the residual processing process to one or more processing branches in the second data processing process through the residual processing module. Figure 3 As shown, as an example, two residual connection modules of the present invention are shown above the entire network, one residual connection module connects the processing branch of the first processing stage with the processing branch of the sixth processing stage, and the other residual connection module connects the upper processing branch of the second processing stage with the upper processing branch of the fifth processing stage, so that the information blockage between low-level and high-level features can be broken through, and information loss during processing can be avoided. The residual processing module 830 is used to implement the above method steps.

[0124] For the feature transition module 840, since the image data is convoluted at each stage, the more convolutions are processed, the higher the level of features contained in the output feature image data. When the feature image data enters the second data processing process, it has been processed at three stages. Therefore, the feature image data output from the fourth stage contains quite high-level features. In order to improve the accuracy of image recognition, it is hoped that the feature dimension can be smoothly increased so that the processed feature image data loses as little information as possible. Our backbone network adds a feature transition module from the fourth stage. Before the current processing branch outputs data to the processing branch with more feature dimensions, the output of the current processing branch is adjusted by the feature transition module. That is to say, an additional feature conversion module is extended after the main convolution block, and the feature conversion module doubles the previous feature dimension. In one embodiment, in order to enhance feature diversity, a deformable convolution is used as the convolution layer of the feature conversion block. The feature transition module 840 is used to implement the above method steps.

[0125] Fig.13 An image mask generating device according to another embodiment of the present invention is shown, the device comprising:

[0126] An image data receiving module 900, used to receive image data to be processed, that is, to implement step S300;

[0127] An image data processing module 910 is used to input the image data to be processed into a data processing process including M processing stages for processing, where M is an integer greater than or equal to 2, i.e., to implement step S310;

[0128] A data acquisition module 920, used to acquire data output by N processing stages out of the M processing stages, where N is an integer greater than or equal to 2 and less than or equal to M, i.e., used to implement step S320;

[0129] A fusion module 930, used to fuse the data output by the N processing stages, that is, to implement step S330;

[0130] The mask generation module 940 is used to perform pooling processing on the fused data to obtain an image mask, that is, to implement step S340. Fig.14 An image mask generating device according to another embodiment of the present invention is shown, the device comprising:

[0131] An image data acquisition module 1000, used to acquire image data including mask features, i.e., to implement step S400, wherein the mask features are mask features of an image including an object to be identified;

[0132] The information acquisition module 1010 is used to acquire the center point information and identification frame parameters of the object to be identified on the image data, that is, to implement step S410;

[0133] The mask feature acquisition module 1020 is used to extract the mask feature of the object to be identified from the image data based on the center point information and the identification frame information, that is, to implement step S420;

[0134] The mask generation module 1030 is used to generate an image mask of the object based on the mask features of the object to be identified extracted from the image data, that is, to implement step S430.

[0135] Fig.15 An image recognition device according to another embodiment of the present invention is shown, the device comprising:

[0136] An image data acquisition module 1100 is used to acquire image data containing a group of objects to be identified, i.e., to implement step S500;

[0137] An image recognition module 1110, configured to recognize each item in the group of items to be recognized based on the image data, i.e., to implement step S510;

[0138] An identification frame generating module 1120 is used to generate an identification frame on the image data for each identified object to mark the object, that is, to implement step S520;

[0139] wherein the group of items includes at least two items; and

[0140] The center of each identification frame is located in the image of the object marked by the identification frame; and

[0141] The plurality of identification frames generated on the image data at least partially overlap.

[0142] Above Figure 10-15 In the device embodiment shown, only the main functions of the modules are described, and all the functions of each module correspond to the corresponding steps in the method embodiment. The working principle of each module can also refer to the description of the corresponding steps in the method embodiment, and will not be repeated here. In addition, although the corresponding relationship between the functions of the functional modules and the methods is defined in the above embodiments, those skilled in the art can understand that the functions of the functional modules are not limited to the above correspondence, that is, a specific functional module can also implement other method steps or part of the method steps. For example, the above embodiments describe that the mask generation module 1030 is used to implement the method of step S430, but according to the needs of the actual situation, the mask generation module 1030 can also be used to implement the method of step S400, S410 or S420 or part of the method.

[0143] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method of any of the above-mentioned embodiments is implemented. It should be pointed out that the computer program stored in the computer-readable storage medium of the embodiment of the present application can be executed by the processor of the electronic device. In addition, the computer-readable storage medium can be a storage medium built into the electronic device, or a storage medium that can be plugged into the electronic device. Therefore, the computer-readable storage medium of the embodiment of the present application has high flexibility and reliability.

[0144] Fig.16 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. The electronic device may be a control system / electronic system configured in an automobile, a mobile terminal (e.g., a smart mobile phone, etc.), a personal computer (PC, such as a desktop computer or a notebook computer, etc.), a tablet computer, and a server, etc. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

[0145] like Fig.16 As shown, the electronic device may include: a processor (processor) 1202 , a communication interface (Communications Interface) 1204 , a memory (memory) 1206 , and a communication bus 1208 .

[0146] in:

[0147] The processor 1202 , the communication interface 1204 , and the memory 1206 communicate with each other via a communication bus 1208 .

[0148] The communication interface 1204 is used to communicate with other devices such as clients or other servers.

[0149] The processor 1202 is used to execute the program 1210, and specifically can execute the relevant steps in the above method embodiment.

[0150] Specifically, the program 1210 may include program codes, which include computer operation instructions.

[0151] The processor 1202 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0152] The memory 1206 is used to store the program 1210. The memory 1206 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0153] The program 1210 may be downloaded and installed from a network through the communication interface 1204, and / or installed from a removable medium. When the program is executed by the processor 1202, the processor 1202 may perform various operations in the above method embodiments.

[0154] In summary, the invention of the present invention includes: an image recognition method, comprising:

[0155] Acquire image data containing an object to be identified;

[0156] Processing the image data to identify an object to be identified in the image data, and obtaining key point information and identification frame parameters of the object to be identified;

[0157] For the identified object, generating an identification frame on the image data based on the key point information and the identification frame parameters; and

[0158] For the identified object, a mask of the identified object is generated based on the key point information and the identification box parameters.

[0159] Optionally, the operation of generating the identification frame and the operation of generating the mask of the identified object are performed in parallel.

[0160] Optionally, the identification frame parameters include the width of the identification frame and the height of the identification frame.

[0161] Optionally, the key point includes a center point of the identified object.

[0162] Optionally, the processing the image data includes inputting the image data into a backbone network for processing.

[0163] Optionally, the backbone network includes a first data processing process and a second data processing process, the data processing process includes one or more processing stages, and the processing stage includes one or more processing branches.

[0164] Optionally, data output by the multiple processing branches are fused.

[0165] Optionally, the data output by the multiple processing branches has multiple resolutions and / or multiple feature dimensions.

[0166] Optionally, in the first processing process, the post-processing stage includes more processing branches than the previous processing stage; and / or, in the second processing process, the post-processing stage includes fewer processing branches than the previous processing stage.

[0167] An image recognition device, comprising:

[0168] An image data acquisition module, used to acquire image data containing an object to be identified;

[0169] An image data processing module, used to process the image data to identify the object to be identified in the image data, and obtain key point information and identification frame parameters of the object to be identified;

[0170] an identification frame generation module, for the identified object, generating an identification frame on the image data based on the key point information and the identification frame parameters; and

[0171] The mask generation module generates a mask of the identified object based on the key point information and the identification frame parameters.

[0172] Optionally, the identification frame generation module and the mask generation module run in parallel.

[0173] Optionally, the identification frame parameters include the width of the identification frame and the height of the identification frame.

[0174] Optionally, the key point includes a center point of the identified object.

[0175] Optionally, the image data processing module is used to input the image data into the backbone network for processing.

[0176] Optionally, the backbone network includes a first data processing process and a second data processing process, the data processing process includes one or more processing stages, and the processing stage includes one or more processing branches.

[0177] Optionally, data output by the multiple processing branches are fused.

[0178] Optionally, the data output by the multiple processing branches has multiple resolutions and / or multiple feature dimensions.

[0179] Optionally, in the first processing process, the post-processing stage includes more processing branches than the previous processing stage; and / or, in the second processing process, the post-processing stage includes fewer processing branches than the previous processing stage.

[0180] A method for processing image data, comprising:

[0181] receiving image data to be processed;

[0182] Processing the image data to be processed using a first data processing process;

[0183] Using a second data processing process to process the image data processed by the first data processing process;

[0184] The data processing process includes one or more processing stages. In the first data processing process, some processing stages include processing to increase the resolution of image data, and some processing stages include processing to reduce the resolution of image data; in the second data processing process, any processing stage includes processing to increase the resolution of image data, and does not include processing to reduce the resolution of image data.

[0185] Optionally, the processing stage includes one or more processing branches.

[0186] Optionally, data output by the multiple processing branches are fused.

[0187] Optionally, the data output by the multiple processing branches has multiple resolutions and / or multiple feature dimensions.

[0188] Optionally, the multiple processing branches belong to the same processing stage.

[0189] Optionally, in the first processing process, the post-processing stage includes more processing branches than the previous processing stage.

[0190] Optionally, in the second processing step, the post-processing stage includes fewer processing branches than the previous processing stage.

[0191] Optionally, the last processing stage of the first processing process and the first processing stage of the second processing process have the same number of processing branches.

[0192] Optionally, the process of reducing the resolution includes reducing the resolution using 1x1 convolution downsampling; and / or the process of increasing the resolution includes increasing the resolution using 1x1 convolution upsampling.

[0193] A device for processing image data, comprising:

[0194] An image data receiving module, used for receiving image data to be processed;

[0195] A first data processing module, used for processing the image data to be processed using a first data processing process;

[0196] A second data processing module, used for processing the image data processed by the first data processing process using a second data processing process;

[0197] The data processing process includes one or more processing stages. In the first data processing process, some processing stages include processing to increase the resolution of image data, and some processing stages include processing to reduce the resolution of image data; in the second data processing process, any processing stage includes processing to increase the resolution of image data, and does not include processing to reduce the resolution of image data.

[0198] Optionally, the processing stage includes one or more processing branches.

[0199] Optionally, data output by the multiple processing branches are fused.

[0200] Optionally, the data output by the multiple processing branches has multiple resolutions and / or multiple feature dimensions.

[0201] Optionally, the multiple processing branches belong to the same processing stage.

[0202] Optionally, in the first processing process, the post-processing stage includes more processing branches than the previous processing stage.

[0203] Optionally, in the second processing step, the post-processing stage includes fewer processing branches than the previous processing stage.

[0204] Optionally, the last processing stage of the first processing process and the first processing stage of the second processing process have the same number of processing branches.

[0205] Optionally, the process of reducing the resolution includes reducing the resolution using a 1x1 convolution downsampling process; and / or the process of increasing the resolution includes increasing the resolution using a 1x1 convolution upsampling process.

[0206] A method for processing image data, comprising:

[0207] receiving image data to be processed;

[0208] Processing the image data to be processed using a first data processing process;

[0209] Using a second data processing process to process the image data processed by the first data processing process;

[0210] Wherein, the data processing process includes one or more processing stages, and each processing stage includes one or more processing branches;

[0211] The method further includes a residual processing process, wherein one or more processing branches in the first data processing process are connected to one or more processing branches in the second data processing process through the residual processing process;

[0212] Before one or more processing branches in the second processing process output data to the next processing stage, the data to be output is processed through a feature transition operation.

[0213] Optionally, the feature transition operation includes performing deformable convolution processing on the data to be output.

[0214] Optionally, the residual processing process includes connecting a processing branch of a first processing stage in a first data processing process with a processing branch of a last processing stage in a second data processing process through the residual processing process.

[0215] Optionally, the data output by multiple processing branches are fused.

[0216] Optionally, the data output by the multiple processing branches has multiple resolutions and / or multiple feature dimensions.

[0217] Optionally, the multiple processing branches belong to the same processing stage.

[0218] Optionally, in the first processing process, the post-processing stage includes more processing branches than the previous processing stage.

[0219] Optionally, in the second processing step, the post-processing stage includes fewer processing branches than the previous processing stage.

[0220] Optionally, the last processing stage of the first processing process and the first processing stage of the second processing process have the same number of processing branches.

[0221] A device for processing image data, comprising:

[0222] An image data receiving module receives image data to be processed;

[0223] A first data processing module, using a first data processing process to process the image data to be processed;

[0224] A second data processing module, using a second data processing process to process the image data processed by the first data processing process;

[0225] Wherein, the data processing process includes one or more processing stages, and each processing stage includes one or more processing branches;

[0226] The processing device also includes:

[0227] A residual processing module, used for connecting one or more processing branches in the first data processing process with one or more processing branches in the second data processing process through a residual processing process;

[0228] The feature transition module is used to perform feature transition processing on the data to be output before one or more processing branches in the second processing process output data to the next processing stage.

[0229] Optionally, the feature transition module is also used to perform deformable convolution processing on the data to be output.

[0230] Optionally, the residual processing module is further used to connect the processing branch of the first processing stage in the first data processing process with the processing branch of the last processing stage in the second data processing process through the residual processing process.

[0231] Optionally, the data output by multiple processing branches are fused.

[0232] Optionally, the data output by the multiple processing branches has multiple resolutions and / or multiple feature dimensions.

[0233] Optionally, the multiple processing branches belong to the same processing stage.

[0234] Optionally, in the first processing process, the post-processing stage includes more processing branches than the previous processing stage.

[0235] Optionally, in the second processing step, the post-processing stage includes fewer processing branches than the previous processing stage.

[0236] Optionally, the last processing stage of the first processing process and the first processing stage of the second processing process have the same number of processing branches.

[0237] A method for generating an image mask, comprising:

[0238] receiving image data to be processed;

[0239] Inputting the image data to be processed into a data processing process including M processing stages for processing, where M is an integer greater than or equal to 2;

[0240] Obtaining data output by N processing stages out of the M processing stages, where N is an integer greater than or equal to 2 and less than or equal to M;

[0241] Fusion of data output from the N processing stages;

[0242] The fused data is pooled to obtain the image mask.

[0243] Optionally, the inputting of the image data to be processed into a data processing process including M processing stages for processing is specifically inputting the image data to be processed into a backbone network for processing, the backbone network including a first processing process and a second processing process, and the processing process including M processing stages.

[0244] Optionally, the N processing stages are processing stages in a second data processing process.

[0245] Optionally, the fusion includes a channel-level addition operation and / or a channel-level superposition operation.

[0246] Optionally, performing pooling processing to obtain an image mask includes calculating the image mask based on central point information of the object to be identified and parameters of the identification box.

[0247] Optionally, the identification frame parameters include width information of the identification frame and height information of the identification frame.

[0248] Optionally, the center point information includes a ground truth center point.

[0249] Optionally, the processing stage includes one or more processing branches, and inputting the image data to be processed into the backbone network for processing also includes fusing the data output by the multiple processing branches.

[0250] Optionally, the data output by the multiple processing branches has multiple resolutions and / or multiple feature dimensions.

[0251] An image mask generating device, comprising:

[0252] An image data receiving module, used for receiving image data to be processed;

[0253] An image data processing module, used for inputting the image data to be processed into a data processing process including M processing stages for processing, where M is an integer greater than or equal to 2;

[0254] A data acquisition module, used to acquire data output by N processing stages out of the M processing stages, where N is an integer greater than or equal to 2 and less than or equal to M;

[0255] A fusion module, used for fusing the data outputted by the N processing stages;

[0256] The mask generation module is used to perform pooling processing on the fused data to obtain an image mask.

[0257] Optionally, the image data processing module is specifically used to input the image data to be processed into a backbone network for processing, the backbone network includes a first processing process and a second processing process, and the processing process includes M processing stages.

[0258] Optionally, the N processing stages are processing stages in a second data processing process.

[0259] Optionally, the fusion includes a channel-level addition operation and / or a channel-level superposition operation.

[0260] Optionally, the mask generation module is further used to calculate the image mask based on the center point information of the object to be identified and the identification frame parameters.

[0261] Optionally, the identification frame parameters include width information of the identification frame and height information of the identification frame.

[0262] Optionally, the center point information includes a ground truth center point.

[0263] Optionally, the processing stage includes one or more processing branches, and inputting the image data to be processed into the backbone network for processing also includes fusing the data output by the multiple processing branches.

[0264] Optionally, the data output by the multiple processing branches has multiple resolutions and / or multiple feature dimensions.

[0265] A method for generating an image mask, comprising:

[0266] Acquire image data containing mask features, wherein the mask features are mask features of an image including an object to be identified;

[0267] Obtaining the center point information and identification frame parameters of the object to be identified on the image data;

[0268] Extracting mask features of the object to be identified from the image data based on the center point information and the identification frame information;

[0269] An image mask of the object is generated based on the mask features of the object to be identified extracted from the image data.

[0270] Optionally, the identification frame parameters include width information of the identification frame and height information of the identification frame.

[0271] Optionally, the center point information includes a ground truth center point.

[0272] Optionally, the image data containing mask features includes feature image data acquired from a backbone network and obtained after fusion.

[0273] Optionally, the fusion includes a channel-level addition operation and / or a channel-level superposition operation.

[0274] Optionally, the backbone network includes a first data processing process and a second data processing process, the data processing process includes one or more processing stages, and the data stage includes one or more processing branches.

[0275] Optionally, in the first processing process, the post-processing stage includes more processing branches than the previous processing stage.

[0276] Optionally, in the second processing step, the post-processing stage includes fewer processing branches than the previous processing stage.

[0277] Optionally, the last processing stage of the first processing process and the first processing stage of the second processing process have the same number of processing branches.

[0278] An image mask generating device, comprising:

[0279] An image data acquisition module, used to acquire image data containing mask features, wherein the mask features are mask features of an image including an object to be identified;

[0280] An information acquisition module, used to acquire the center point information and identification frame parameters of the object to be identified on the image data;

[0281] A mask feature acquisition module, used to extract the mask feature of the object to be identified from the image data based on the center point information and the identification frame information;

[0282] The mask generation module is used to generate an image mask of the object based on the mask features of the object to be identified extracted from the image data.

[0283] Optionally, the identification frame parameters include width information of the identification frame and height information of the identification frame.

[0284] Optionally, the center point information includes a ground truth center point.

[0285] Optionally, the image data containing mask features includes feature image data acquired from a backbone network and obtained after fusion.

[0286] Optionally, the fusion includes a channel-level addition operation and / or a channel-level superposition operation.

[0287] Optionally, the backbone network includes a first data processing process and a second data processing process, the data processing process includes one or more processing stages, and the data stage includes one or more processing branches.

[0288] Optionally, in the first processing process, the post-processing stage includes more processing branches than the previous processing stage.

[0289] Optionally, in the second processing step, the post-processing stage includes fewer processing branches than the previous processing stage.

[0290] Optionally, the last processing stage of the first processing process and the first processing stage of the second processing process have the same number of processing branches.

[0291] An image recognition method, comprising:

[0292] Acquire image data containing a group of objects to be identified;

[0293] identifying each object in the group of objects to be identified based on the image data;

[0294] For each identified object, generating an identification box on the image data to mark the object;

[0295] wherein the group of items includes at least two items; and

[0296] The center of each identification frame is located in the image of the object marked by the identification frame; and

[0297] The plurality of identification frames generated on the image data at least partially overlap.

[0298] Optionally, the degree of overlap of the identification frames is represented by an overlap degree, where the overlap degree of two identification frames = the area of ​​the region where the two identification frames intersect / the area of ​​the region formed after the two identification frames are merged.

[0299] Optionally, the at least partial overlap includes an overlap greater than 60%.

[0300] Optionally, generating the recognition frame includes generating the recognition frame based on key point information of the object and recognition frame parameters.

[0301] Optionally, the identification frame parameters include length information and width information of the identification frame.

[0302] Optionally, the key point of the object includes the center point of the object.

[0303] Optionally, identifying each item in the group of items to be identified based on the image data includes inputting the image data into a backbone network for processing to identify each item in the group of items.

[0304] Optionally, the backbone network includes a first data processing process and a second data processing process, the data processing process includes one or more processing stages, and the data stage includes one or more processing branches.

[0305] Optionally, in the first processing process, the post-processing stage includes more processing branches than the previous processing stage; and / or, in the second processing process, the post-processing stage includes fewer processing branches than the previous processing stage.

[0306] An image recognition device, comprising:

[0307] An image data acquisition module, used to acquire image data containing a group of objects to be identified;

[0308] An image recognition module, configured to recognize each object in the group of objects to be recognized based on the image data;

[0309] An identification frame generation module, for generating an identification frame on the image data for each identified object to mark the object;

[0310] wherein the group of items includes at least two items; and

[0311] The center of each identification frame is located in the image of the object marked by the identification frame; and

[0312] The plurality of identification frames generated on the image data at least partially overlap.

[0313] Optionally, the degree of overlap of the identification frames is represented by an overlap degree, where the overlap degree of two identification frames = the area of ​​the region where the two identification frames intersect / the area of ​​the region formed after the two identification frames are merged.

[0314] Optionally, the at least partial overlap includes an overlap greater than 60%.

[0315] Optionally, the recognition frame generation module is further used to generate a recognition frame based on key point information of the object and recognition frame parameters.

[0316] Optionally, the identification frame parameters include length information and width information of the identification frame.

[0317] Optionally, the key point of the object includes the center point of the object.

[0318] Optionally, the image recognition module is further used to input the image data into a backbone network for processing to identify each item in the item group.

[0319] Optionally, the backbone network includes a first data processing process and a second data processing process, the data processing process includes one or more processing stages, and the data stage includes one or more processing branches.

[0320] Optionally, in the first processing process, the post-processing stage includes more processing branches than the previous processing stage; and / or, in the second processing process, the post-processing stage includes fewer processing branches than the previous processing stage.

[0321] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0322] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0323] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0324] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0325] It should be understood that the various parts of the embodiments of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0326] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0327] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0328] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0329] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above implementation methods within the scope of the present application.

Claims

1. A method for generating an image mask, characterized in that: include: Acquire image data containing mask features, wherein the mask features are mask features of an image including an object to be identified; Obtaining the center point information and identification frame parameters of the object to be identified on the image data; Extracting mask features of the object to be identified from the image data based on the center point information and the identification frame information; generating an image mask of the object based on mask features of the object to be identified extracted from the image data; The extracting the mask feature of the object to be identified from the image data based on the center point information and the identification frame information includes: Based on the center point information and the length and width information of the recognition frame, the position of the object to be recognized is obtained, and the mask features of the object are extracted from the image data containing the mask features; The image data containing mask features includes feature image data obtained from the backbone network and obtained after fusion; The backbone network includes a first data processing process and a second data processing process, wherein the data processing process includes one or more processing stages, and the processing stage includes one or more processing branches; the second data processing process includes processing to increase the resolution of image data, but does not include processing to reduce the resolution of image data.

2. The image mask generation method according to claim 1, characterized in that: The identification frame parameters include width information of the identification frame and height information of the identification frame.

3. The image mask generation method according to claim 1, characterized in that: The center point includes a true center point.

4. The image mask generation method according to claim 1, characterized in that: The fusion includes a channel-level addition operation and / or a channel-level superposition operation.

5. The image mask generation method according to claim 1, characterized in that: In the first data processing process, the post-processing stage includes more processing branches than the pre-processing stage.

6. The image mask generation method according to claim 1, characterized in that: In the second data processing process, the post-processing stage includes fewer processing branches than the pre-processing stage.

7. The image mask generation method according to claim 1, characterized in that: The last processing stage of the first data processing process has the same number of processing branches as the first processing stage of the second data processing process.

8. An image mask generating device, characterized in that: include: An image data acquisition module, used to acquire image data containing mask features, wherein the mask features are mask features of an image including an object to be identified; An information acquisition module, used to acquire the center point information and identification frame parameters of the object to be identified on the image data; A mask feature acquisition module, used to extract the mask feature of the object to be identified from the image data based on the center point information and the identification frame information; A mask generation module, used for generating an image mask of the object based on the mask features of the object to be identified extracted from the image data; The mask feature acquisition module is specifically used for: Based on the center point information and the length and width information of the recognition frame, the position of the object to be recognized is obtained, and the mask features of the object are extracted from the image data containing the mask features; The image data containing mask features includes feature image data obtained from the backbone network and obtained after fusion; The backbone network includes a first data processing process and a second data processing process, wherein the data processing process includes one or more processing stages, and the processing stage includes one or more processing branches; the second data processing process includes processing to increase the resolution of image data, but does not include processing to reduce the resolution of image data.

9. The image mask generating device according to claim 8, characterized in that: The identification frame parameters include width information of the identification frame and height information of the identification frame.

10. The image mask generating device according to claim 8, characterized in that: The center point includes a true center point.

11. The image mask generating device according to claim 8, characterized in that: The fusion includes a channel-level addition operation and / or a channel-level superposition operation.

12. The image mask generating device according to claim 8, characterized in that: In the first data processing process, the post-processing stage includes more processing branches than the pre-processing stage.

13. The image mask generating device according to claim 8, characterized in that: In the second data processing process, the post-processing stage includes fewer processing branches than the pre-processing stage.

14. The image mask generating device according to claim 8, characterized in that: The last processing stage of the first data processing process has the same number of processing branches as the first processing stage of the second data processing process.

15. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image mask generation method according to any one of claims 1 to 7 when executing the computer program.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image mask generating method according to any one of claims 1 to 7 is implemented.

Images