Image recognition method, target object extraction method, device, medium and equipment
By combining the pre-trained recognition model with position and stacking information, the problem of poor positioning accuracy of image positioning technology in stacked object recycling is solved, and high-precision target object extraction and recycling is achieved.
Patent Information
- Application Number
- CN202011626224.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-31
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-12-31
AI Technical Summary
The existing image positioning technology leads to low recovery when processing stacked objects, poor positioning accuracy leads to low recovery.
Using a pre-trained recognition model, by acquiring the image to be identified and positioning and extracting the target object based on the position information and stacking information, the stacking information assists the positioning accuracy, and determines the extraction strategy based on the occlusion situation.
It improves the positioning accuracy and extraction success rate of the target object, ensuring high-precision recycling in the case of stacking.
Smart Images

Figure CN112699888B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of image processing technology, and in particular to an image recognition method, a target object extraction method, a device, a medium and a device. Background Art
[0002] In the process of processing objects that can be stacked, such as tea leaves and flower petals, there is a process of recovering and extracting the distributed objects. In the current technical solution, the position of each object is mainly determined by image positioning of the distributed objects, so as to recover and extract each object.
[0003] However, the current image positioning method has the problem of poor positioning accuracy, resulting in a low recovery rate of the above objects. Summary of the invention
[0004] The present invention provides an image recognition method, device, storage medium and electronic equipment to achieve accurate positioning of a target object.
[0005] In a first aspect, an embodiment of the present invention provides an image recognition method, comprising:
[0006] Acquire an image to be identified, wherein the image to be identified includes at least one target object;
[0007] The image to be identified is input into a pre-trained recognition model, and the position information of each target object in the image to be identified and the stacking information of each target object are determined based on the output result of the recognition model, wherein the recognition model is trained based on a sample image of the target object and the stacking labels and position labels of each target object in the sample image.
[0008] In a second aspect, an embodiment of the present invention further provides a method for extracting a target object, comprising:
[0009] Acquire a first image of at least one target object;
[0010] Inputting the first image into a pre-trained recognition model, and determining position information of each target object in the first image and stacking information of each target object based on an output result of the recognition model;
[0011] Each target object is extracted based on the position information and stacking information of each target object.
[0012] In a third aspect, an embodiment of the present invention further provides an image recognition device, comprising:
[0013] A first image acquisition module, used to acquire an image to be identified, wherein the image to be identified includes at least one target object;
[0014] The first image recognition module is used to input the image to be recognized into a pre-trained recognition model, and determine the position information of each target object in the image to be recognized and the stacking information of each target object based on the output result of the recognition model, wherein the recognition model is trained based on a sample image of the target object and the stacking labels and position labels of each target object in the sample image.
[0015] In a fourth aspect, an embodiment of the present invention further provides a device for extracting a target object, comprising:
[0016] A second image acquisition module, used to acquire a first image of at least one target object;
[0017] a second image recognition module, configured to input the first image into a pre-trained recognition model, and determine position information of each target object in the first image and stacking information of each target object based on an output result of the recognition model;
[0018] The target object extraction module is used to extract each target object based on the position information and stacking information of each target object.
[0019] In a fifth aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an image recognition method or a target object extraction method as provided in any embodiment of the present invention is implemented.
[0020] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an image recognition method or a target object extraction method as provided in any embodiment of the present invention.
[0021] The technical solution provided by the embodiment of the present invention processes the image to be identified through a recognition model with position information and stacking information recognition functions, obtains the position information and stacking information of each target object in the image to be identified, and locates each target object through the stacking information to assist the position information, thereby improving the accuracy of the positioning information. At the same time, the occlusion status of the target object is determined based on the stacking information, thereby facilitating the high-precision extraction of the target object. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A schematic diagram of a flow chart of an image recognition method provided in Embodiment 1 of the present invention;
[0023] Figure 2 is a structural schematic diagram of a recognition model provided by an embodiment of the present invention;
[0024] Figure 3 is a structural schematic diagram of another recognition model provided by an embodiment of the present invention;
[0025] Figure 4 is a flow chart of a method for extracting a target object provided in the second embodiment of the present invention;
[0026] Figure 5 is a structural schematic diagram of an image recognition device provided by Embodiment 3 of the present invention;
[0027] Figure 6 is a schematic diagram of the structure of a target object extraction device provided by a fourth embodiment of the present invention;
[0028] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION
[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0030] Embodiment 1
[0031] Figure 1 This is a flow chart of an image recognition method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case where the position and stacking information of a target object are determined by recognizing an image. The method can be executed by an image recognition device provided in an embodiment of the present invention, and the device can be integrated in an electronic device such as a computer or a mobile phone. The method specifically includes the following steps:
[0032] S110: Acquire an image to be identified, wherein the image to be identified includes at least one target object.
[0033] S120. Input the image to be identified into a pre-trained recognition model, and determine the position information of each target object in the image to be identified and the stacking information of each target object based on the output result of the recognition model, wherein the recognition model is trained based on a sample image of the target object and the stacking labels and position labels of each target object in the sample image.
[0034] Among them, the target object can be an object that has a stacked relationship or can be stacked, including but not limited to plant leaves, petals, books, etc., and an image including at least one target object is collected as the image to be identified. By processing the image to be identified, the positioning information of each target image in the image to be identified is obtained.
[0035] In this embodiment, a recognition model with a positioning function is pre-trained to identify the position information and stacking information of each target object in the image to be identified. Among them, the stacking information is used to characterize whether the target object is blocked by other objects. Optionally, the stacking information includes a top layer and a bottom layer, wherein the stacking information for the top layer indicates that the target object is not blocked, and the stacking information for the bottom layer indicates that the target object is blocked. It should be noted that the target object with the stacking information for the bottom layer may be blocked by one or more blocking objects. The stacking information assists the position information in locating the target object, improves the accuracy of locating the target object, and facilitates determining the correct extraction strategy when extracting the target object to ensure the effectiveness of the target object extraction. Exemplarily, the target object with the stacking information for the top layer can be directly extracted based on the position information, and the target object with the stacking information for the bottom layer can be extracted by moving the blocking object or extracting the blocking object.
[0036] In this embodiment, by training a recognition model with the function of simultaneously recognizing position information and stacking information, the image to be recognized is processed to synchronously obtain the position information and stacking information of each target object. This improves the recognition efficiency of the image to be recognized while improving the accuracy of the positioning information of each target object.
[0037] In some embodiments, the output result of the recognition model is an image to be recognized marked with a detection frame. The detection frame is an external frame of the target object, and the detection frame can be a rectangle, a circle, or other regular polygons. The position information of each target object is determined by setting a detection frame for each target object in the image to be recognized. Correspondingly, the position information of the target object is the position information of the detection frame including the target object. The detection frame position information includes the coordinates of the center point of the detection frame and the size of the detection frame.
[0038] In some embodiments, the position information of the target object includes the center position of the target object. Accordingly, the center position of the target object may be the center position of a detection box including the target object.
[0039] It should be noted that the output result of the recognition model also includes stacking information of the target object. For example, top or 1 indicates that the stacking information is the top layer, and bottom or 2 indicates that the stacking information is the bottom layer.
[0040] On the basis of the above embodiments, the recognition model may be a machine learning model such as a deep learning neural network model. Optionally, the recognition model includes a feature extraction module and a positioning module, wherein the feature extraction module is used to extract the feature map in the image to be identified, and the positioning module determines the position information and stacking information of each target object based on the feature map. Among them, the feature extraction module may include a preset number of convolution blocks, and each convolution block is connected in sequence to extract feature information of different scales of the image to be identified. Among them, each convolution block may include at least one convolution layer, and each convolution layer may be connected to at least one of an activation function layer, a pooling layer, and a normalization layer. Optionally, the convolution block may also be a residual block. In some embodiments, the feature extraction module is a resnet50 network module.
[0041] The positioning module may include a preset number of convolution blocks, each of which is connected in sequence, each of which may include at least one convolution layer, and each of which may be connected to at least one of an activation function layer, a pooling layer, and a normalization layer. The positioning module is used to identify the feature information output by the feature extraction module and determine the location information and stacking information.
[0042] On the basis of the above embodiment, the recognition model is trained based on the sample image of the target object and the stacking label and position label of each target object in the sample image. Specifically, a sample image of the target object is collected, and a stacking label and a position label are set for each target object in the sample image, wherein the position label can be a detection frame through the target object, that is, the minimum bounding box of the target object. The stacking label can be set to the top layer (for example, top or 1) when the target object is not obscured, and set to the bottom layer (for example, bottom or 2) when the target object is obscured.
[0043] Correspondingly, the training method of the recognition model includes: iteratively training the recognition model to be trained based on the sample image and the stacking labels and position labels of each target object in the sample image until a convergence state is reached or a preset recognition accuracy is met: inputting the sample image into the recognition model to be trained to obtain predicted position information and predicted stacking information; determining a first loss function based on the predicted position information and position label, determining a second loss function based on the predicted stacking information and stacking label, and performing weighted processing based on the first loss function and the second loss function to obtain a target loss function. Adjusting the network parameters in the recognition model to be trained based on the target loss function.
[0044] Exemplarily, the predicted position information of the target object is L1(x1, y1, w1, h1), where x1 and y1 are the horizontal and vertical coordinates of the center position of the predicted detection frame of the target object, w1 is the width information of the predicted detection frame of the target object, and h1 is the width information of the predicted detection frame of the target object, where the target object detection frame is a rectangular frame. The position label of the target object is L0(x0, y0, w0, h0), where x0 and y0 are the horizontal and vertical coordinates of the center position of the standard detection frame of the target object, w0 is the width information of the standard detection frame of the target object, and h0 is the width information of the standard detection frame of the target object.
[0045] Optionally, the first loss function determined based on the predicted position information and the position label may be:
[0046] Among them, L1 i is the i-class value in the predicted position information of the target object, L0 i is the i-type value in the location label of the target object, where i is the horizontal coordinate x, the vertical coordinate y, the height information h, and the width information y.
[0047] in, Here m is the difference between any two values, that is, m i =L1 i -L0 i .
[0048] Optionally, the second loss function determined based on the predicted stack information and the stack label may be:
[0049] Among them, y i It can be prediction cascade information, Can be a stacked tag, for example, y i and They can be 1 (i.e. top layer) and 2 (i.e. bottom layer) respectively.
[0050] It should be noted that the above-mentioned first loss function and second loss function are only optional examples. In other embodiments, other forms of loss functions can be used and can be set according to user needs.
[0051] The target loss function is obtained by weighting the first loss function and the second loss function, and the target loss function is reversely input into the recognition model to be trained, and the network parameters in the recognition model are adjusted based on the gradient descent method, wherein the network parameters at least include the network weights. The above training process is iteratively performed until the training conditions are met, that is, until the convergence state is reached or the preset recognition accuracy is met, and it is determined that the recognition model training is completed.
[0052] The technical solution of this embodiment processes the image to be identified through a recognition model with position information and stacking information recognition functions, obtains the position information and stacking information of each target object in the image to be identified, and locates each target object through the stacking information to assist the position information, thereby improving the accuracy of the positioning information. At the same time, the occlusion status of the target object is determined based on the stacking information, thereby facilitating the high-precision extraction of the target object.
[0053] On the basis of the above embodiment, after determining the position information and stacking information of each target object, the occlusion state of the obscured target object is determined based on the position information and stacking information of each target object. For example, the occlusion state can be intersection and overlap, wherein the intersection can be that the occlusion range of the target object is less than a preset ratio, and the overlap can be that the occlusion range of the target object is greater than a preset ratio. Specifically, the occlusion state can be determined based on the occlusion area of the detection frame of the target object with the stacking information as the bottom layer.
[0054] In some embodiments, the detection frames may be divided into two categories according to the overlapping information, and the soft-NMS algorithm may be called to screen and filter the detection frames of the identified target objects, so that the frames of other frames of the same type that have obvious overlap with the selected suggestion frames are suppressed, and the occlusion status of the occluded target objects is determined to avoid the problem of missing target objects with large areas being occluded.
[0055] On the basis of the above technical solution, the recognition model further includes a segmentation module, which is used to segment the contour segmentation area of each target object from the image to be recognized, wherein the contour segmentation area of the target object is an image area within the contour range of the target object, which is obtained based on the contour segmentation of the target object. The segmentation module can be a neural network module, for example, it can include a preset number of convolution blocks, each convolution block is connected in sequence, each convolution block can include at least one convolution layer, and each convolution layer can be connected to at least one of an activation function layer, a pooling layer, and a normalization layer.
[0056] See also Figure 2 and Figure 3 , Figure 2 and Figure 3 They are schematic diagrams of the structure of a recognition model provided by an embodiment of the present invention. Figure 3 The feature extraction module, the positioning module and the segmentation module are connected in sequence. The segmentation module segments the target objects in each detection frame area output by the positioning module to obtain the contour segmentation area of each target object, thereby reducing the interference of the area outside the detection frame area. Figure 3 The positioning module and the segmentation module are respectively connected to the feature extraction module. The segmentation module segments each target object based on the feature information output by the feature extraction module to obtain the contour segmentation area of each target object.
[0057] On the basis of the above-mentioned embodiment, the position information of the target object is updated through the contour segmentation area of each target object. The position information of the target object output by the positioning module is determined based on the detection frame of the target object, wherein the detection frame includes a blank area outside the target object or the contour segmentation area of other target objects. When the target object is subsequently extracted, it is easy to have the extraction position deviation leading to the extraction failure. In this embodiment, the position information is updated through the contour segmentation area of the target object output by the segmentation module, wherein the updated position information can be located within the contour segmentation area of the target object, or the center point position of the contour segmentation area of the target object, so as to improve the accuracy of the position information and further improve the success rate of subsequent target object extraction.
[0058] On the basis of the above embodiment, the output result of the segmentation module also includes the pixel prediction value of each pixel point in the contour segmentation area of the target object. By determining the pixel value of the pixel point in the contour segmentation area, the predicted contour segmentation area is distinguished from the background area and displayed, so as to facilitate intuitive determination of the distribution position of each target object. Furthermore, by comparing the pixel prediction value of the contour segmentation area with the true value of the pixel in the image to be identified, the recognition model is further trained to improve the training accuracy of the recognition model.
[0059] Accordingly, the recognition model to be trained is iteratively trained based on the sample image and the stacking labels and position labels of each target object in the sample image until it reaches a convergence state or meets the preset recognition accuracy: the sample image is input into the recognition model to be trained to obtain predicted position information and predicted stacking information; a first loss function is determined based on the predicted position information and the position label, a second loss function is determined based on the predicted stacking information and the stacking label, a third loss function is determined based on the pixel prediction value of the contour segmentation area of the target object and the pixel true value of the target object, and a target loss function is obtained by weighting the first loss function, the second loss function and the third loss function. The network parameters in the recognition model to be trained are adjusted based on the target loss function.
[0060] Exemplarily, the third loss function determined based on the pixel prediction value of the contour segmentation area of the target object and the pixel true value of the target object can be:
[0061] loss_fenge=-y true log(y pred )-(1-y true )log(1-y pred ), where y true is the true value of the pixel in the contour segmentation area of the target object, y predis the pixel prediction value in the contour segmentation area of the target object. It should be noted that the contour segmentation area of the target object is within the detection frame area of the target object, that is, the contour segmentation area of the target object is the foreground area in the detection frame area, and the pixel prediction value and the pixel true value are respectively the foreground pixel prediction value and the foreground (i.e., target object) pixel true value in the detection frame area, so as to exclude the interference of background pixels.
[0062] Among them, the weight values of the weighted processing of the first loss function, the second loss function and the third loss function can be preset and can be set separately according to user needs.
[0063] The technical solution of this embodiment updates the position information of the target object by setting the target segmentation area output by the segmentation module to further improve the positioning accuracy of the target object.
[0064] Embodiment 2
[0065] Figure 4 : is a flow chart of a method for extracting a target object provided by the second embodiment of the present invention. This embodiment is applicable to the case where the target object is extracted by determining the location information of the target object in the image. The method can be executed by the target object extraction device provided by the embodiment of the present invention, and the device can be integrated in an electronic device such as a computer or a mobile phone. The method specifically includes the following steps:
[0066] S210: Acquire a first image of at least one target object.
[0067] S220: Input the first image into a pre-trained recognition model, and determine the position information of each target object in the first image and the stacking information of each target object based on an output result of the recognition model.
[0068] S230: Extract each target object based on the position information and stacking information of each target object.
[0069] In this embodiment, the target object may be a sheet material. When the sheet material is being transported on the conveyor belt, a first image of the sheet material distributed on the conveyor belt is collected. Exemplarily, an image acquisition device may be provided above the conveyor belt to acquire images of the sheet material distributed on the conveyor belt according to a preset time interval. The image acquisition device may be, for example, a camera, which may be a depth camera, and accordingly, the image to be identified that is acquired is an image including depth information.
[0070] The position information and stacking information of each target object (i.e., sheet material) are determined through the recognition model, and the stacking information is used to assist the position information in locating each target object, thereby improving the positioning accuracy of the target object and facilitating the accurate extraction of each target object during the extraction process, thereby improving the success rate of target object extraction.
[0071] The stacking information includes the top layer and the bottom layer. The target object with the stacking information as the top layer can be extracted first, and then the target object with the stacking information as the bottom layer can be extracted to ensure that each template target object is not obscured during extraction, avoiding the problem of extraction failure of the obscured target object due to the obstructing object.
[0072] In some embodiments, each target object is extracted based on the position information and stacking information of each target object, including: determining the extraction order of each target object according to the stacking information of each target object; determining the extraction position of each target object based on the position information of each target object; and extracting each target object in sequence based on the extraction order and the extraction position of each target object.
[0073] In this embodiment, the position information of each target object can be sorted based on the extraction order of each target object, so as to control the extraction device to reach each position information in sequence based on the sorting of the above position information, and control the extraction device to perform the extraction operation of the target object after reaching each position information.
[0074] In some embodiments, the extraction order of each target object is determined according to the stacking information of each target object, including: traversing the target object distribution area corresponding to the first image according to a preset direction (for example, from top to bottom and from left to right), for any traversal position, when there is a target object within the extraction range of the traversal position, the extraction order of the target object with the stacking information as the top layer is determined as the previous extraction order, and the extraction order of the target object with the stacking information as the bottom layer is determined as the next extraction order, and then the extraction order of the target object at the next traversal position is determined. Among them, the extraction order of the target object at the next traversal position is after the extraction order of the target object at the previous traversal position. It should be noted that if there is only a target object with the stacking information as the top layer at any traversal position, the extraction order of the target object is after the extraction order of the target object at the previous traversal position. If there are more than two target objects at any traversal position, the extraction order is determined in sequence based on the stacking order.
[0075] In some embodiments, the stacking relationship of the target object is not limited to two layers, but can be more than two layers. For distribution situations where there are more than two layers of stacking relationships, the stacking relationship between the obscured target objects is not obvious, and positioning is difficult, which further leads to a poor success rate in target object extraction.
[0076] In view of the above problem, according to the stacking information of each target object, determining the extraction order of each target object can include: determining the target object with the stacking information at the top layer as the first batch of target objects; determining the target object with the stacking information at the bottom layer as the second batch of target objects. Wherein, the stacking information of two or more layers of the blocked target objects are all at the bottom layer.
[0077] In this embodiment, by dividing the distributed target objects into two batches according to the stacking information, for the first batch of target objects whose stacking information is the top layer, the extraction order of the first batch of target objects can be determined based on the position information of each target object, for example, based on a preset direction (for example, from top to bottom and from left to right) and traversed and sorted in sequence. For the second batch of target objects whose stacking information is the bottom layer, in some embodiments, the extraction order can also be determined based on the position information of each target object, for example, based on a preset direction (for example, from top to bottom and from left to right) and traversed and sorted in sequence.
[0078] In some embodiments, for the second batch of target objects whose stacking information is the bottom layer, the stacking information of the second batch of target objects may be re-identified to facilitate accurate extraction of each second batch of target objects when stacking still exists in the second batch of target objects.
[0079] Correspondingly, each of the target objects is extracted in sequence based on the extraction order and the extraction position of each of the target objects, including: extracting each of the first batch of target objects in sequence based on the position information of the first batch of target objects; acquiring a second image of the second batch of target objects, and determining the position information and stacking information of each of the second batch of target objects in the second image based on the recognition model; determining the extraction order and the extraction position of each of the second batch of target objects based on the position information and stacking information of each of the second batch of target objects, and extracting the second batch of target objects based on the extraction order and the extraction position of each of the second batch of target objects.
[0080] By re-capturing the second image of the second batch of target objects and determining the overlapping information in the second batch of target objects, the extraction order and extraction position of the second batch of target objects are determined based on the overlapping information and position information in the second batch of target objects, and the second batch of target objects are extracted with a high success rate, which is suitable for the situation of extracting multi-layered target objects.
[0081] Based on the above embodiment, the recognition model is also used to determine the contour segmentation area of each target object from the first image. Accordingly, before extracting each target object based on the position information and stacking information of each target object, the method also includes: updating the position information of each target object based on the contour segmentation area of each target object.
[0082] In this embodiment, the extraction device of the target object can be a suction head, and the top area of the suction head is small. The accuracy of the position information of the target object is an important factor affecting the successful extraction of the target object. The positioning module of the recognition model outputs the position information as the external detection frame information of the target object, wherein the external detection frame area of the target object includes a blank area or an overlapping area of other target objects. When the suction head falls on the blank area of the external detection frame of the target object or the overlapping area of other target objects, the target object cannot be extracted, resulting in the failure of the extraction of the target object. Alternatively, when the suction head falls on the edge of the target object, there is also a risk of extraction failure.
[0083] In this embodiment, the contour segmentation area of the target object output by the segmentation module is used to update the position information, wherein the updated position information may be within the contour segmentation area of the target object, or the center point position of the contour segmentation area of the target object, so as to improve the accuracy of the position information and further improve the success rate of subsequent target object extraction.
[0084] The technical solution provided in this embodiment determines the position information and stacking information of each target object through image recognition, uses the stacking information to assist the position information, improves the positioning accuracy of the target object, and uses the stacking information to assist in determining the extraction order of the target objects, thereby avoiding the failure of extraction of the target objects when they are obscured, and improving the success rate of target object extraction.
[0085] On the basis of the above embodiments, a preferred example is further provided, which is specifically that a distribution image of sheet materials (configured as an image to be recognized) is collected by a camera above a sheet material conveyor belt, the distribution image of sheet materials is input into a pre-trained recognition model, and a position detection frame (configured as position information), stacking information, and contour segmentation area (configured as the contour segmentation area of the target object) of each sheet material in the distribution image of sheet materials are output, the position information of each sheet material is updated through the contour segmentation area and position detection frame of each sheet material, and each sheet material is sequentially extracted from the conveyor belt based on the stacking information and position information of each sheet material. The extraction of each sheet material can be to first extract the sheet material with the stacking information as the top layer (top), and then extract the sheet material with the stacking information as the bottom layer (bottom), so as to improve the recovery rate of the sheet material.
[0086] Embodiment 3
[0087] Figure 5 : is a schematic diagram of the structure of an image recognition device provided by Embodiment 3 of the present invention, the device comprising:
[0088] A first image acquisition module 310 is used to acquire an image to be identified, wherein the image to be identified includes at least one target object;
[0089] The first image recognition module 320 is used to input the image to be recognized into a pre-trained recognition model, and determine the position information of each target object in the image to be recognized and the stacking information of each target object based on the output result of the recognition model, wherein the recognition model is trained based on a sample image of the target object and the stacking labels and position labels of each target object in the sample image.
[0090] Optionally, the recognition model includes a feature extraction module and a positioning module, wherein the feature extraction module is used to extract a feature map in the image to be recognized, and the positioning module determines the position information and stacking information of each target object based on the feature map.
[0091] Optionally, the position information includes the center position of the target object, or the position information includes the detection box position information of the target object; the stacking information includes a top layer and a bottom layer.
[0092] Optionally, the device further comprises:
[0093] The model training module is used to iteratively train the recognition model to be trained based on the sample image and the stacked labels and position labels of each target object in the sample image until a convergence state is reached or a preset recognition accuracy is met:
[0094] Inputting the sample image into a recognition model to be trained to obtain predicted position information and predicted stacking information;
[0095] A first loss function is determined based on the predicted position information and the position label, a second loss function is determined based on the predicted stack information and the stack label, and a target loss function is obtained by weighting the first loss function and the second loss function.
[0096] The network parameters in the recognition model to be trained are adjusted based on the target loss function.
[0097] Optionally, the recognition model further includes a segmentation module, and the segmentation module is used to segment the contour segmentation area of each target object from the image to be recognized.
[0098] Optionally, the device further comprises:
[0099] The position information updating module is used to update the position information of the corresponding target object according to the position information of the contour segmentation area of the target object.
[0100] Optionally, the output result of the segmentation module also includes a pixel prediction value of each pixel point in the contour segmentation area of the target object.
[0101] Optionally, the model training module also includes:
[0102] Before performing weighted processing based on the first loss function and the second loss function to obtain a target loss function, a third loss function is determined based on pixel prediction values of a contour segmentation area of the target object and true pixel values of the target object.
[0103] Accordingly, the model training module is specifically used for:
[0104] A target loss function is obtained by performing weighted processing based on the first loss function, the second loss function and the third loss function.
[0105] The above-mentioned product can execute the method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0106] Embodiment 4
[0107] Figure 6 : is a schematic diagram of a target object extraction device provided by a fourth embodiment of the present invention, the device comprising:
[0108] A second image acquisition module 410, configured to acquire a first image of at least one target object;
[0109] A second image recognition module 420, configured to input the first image into a pre-trained recognition model, and determine position information of each target object in the first image and stacking information of each target object based on an output result of the recognition model;
[0110] The target object extraction module 430 is used to extract each target object based on the position information and stacking information of each target object.
[0111] Optionally, the target object extraction module 430 includes:
[0112] An extraction order determination unit, used to determine the extraction order of each target object according to the stacking information of each target object;
[0113] An extraction position determination module, used to determine the extraction position of each target object based on the position information of each target object;
[0114] The target object extraction unit is used to extract each target object in sequence based on the extraction order and the extraction position of each target object.
[0115] Optionally, the extraction order determination unit is used to:
[0116] Determine the target object with the stacking information at the top layer as the first batch of target objects;
[0117] The target objects with the stacking information as the bottom layer are determined as the second batch of target objects.
[0118] Optionally, the target object extraction unit is used to:
[0119] Sequentially extracting each of the first batch of target objects based on the position information of the first batch of target objects;
[0120] Acquire a second image of the second batch of target objects, and determine position information and stacking information of each of the second batch of target objects in the second image based on the recognition model;
[0121] The extraction order and the extraction position of each second batch target object are determined based on the position information and the stacking information of each second batch target object, and the second batch target objects are extracted based on the extraction order and the extraction position of each second batch target object.
[0122] Optionally, the recognition model is further used to determine the contour segmentation area of each target object from the first image;
[0123] Accordingly, the device further comprises:
[0124] The position updating module is used to update the position information of each target object based on the contour segmentation area of each target object before extracting each target object based on the position information and stacking information of each target object.
[0125] The above-mentioned product can execute the method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0126] Embodiment 5
[0127] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. Figure 7 A block diagram of an electronic device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 7 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. The device 12 is typically an electronic device that undertakes the image classification function.
[0128] like Figure 7 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors 16, a storage device 28, and a bus 18 connecting various system components (including the storage device 28 and the processor 16).
[0129] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
[0130] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.
[0131] The storage device 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a read-only optical disk (Compact Disc-Read Only Memory, CD-ROM), a digital video disk (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The storage device 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.
[0132] A program 36 having a set (at least one) of program modules 26 may be stored, for example, in a storage device 28. Such program modules 26 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof may include an implementation of a network environment. The program modules 26 generally perform the functions and / or methods of the embodiments described herein.
[0133] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, cameras, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed through an input / output (I / O) interface 22. In addition, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 through a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.
[0134] The processor 16 executes various functional applications and data processing by running the programs stored in the storage device 28, such as implementing the image recognition method or the target object extraction method provided in the above-mentioned embodiment of the present invention.
[0135] Embodiment 6
[0136] Embodiment 6 of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the image recognition method or the target object extraction method provided in the embodiment of the present invention is implemented.
[0137] Of course, the computer-readable storage medium provided by an embodiment of the present invention stores a computer program which is not limited to the method operation described above, but can also execute the image recognition method or target object extraction method provided by any embodiment of the present invention.
[0138] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0139] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable source code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0140] Source code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0141] Computer source code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The source code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0142] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. An image recognition method, characterized in that: include: Acquire an image to be identified, wherein the image to be identified includes at least one target object, and the target object is an object that has a stacked relationship or can be stacked; Input the image to be identified into a pre-trained recognition model, and determine the position information of each target object in the image to be identified and the stacking information of each target object based on the output result of the recognition model, wherein the recognition model is trained based on a sample image of the target object and the stacking labels and position labels of each target object in the sample image, and the recognition model includes a feature extraction module and a positioning module, the feature extraction module is used to extract a feature map in the image to be identified, and the positioning module determines the position information and stacking information of each target object based on the feature map; The training method of the recognition model includes: iteratively training the recognition model to be trained based on the sample image and the stacked labels and position labels of each target object in the sample image until a convergence state is reached or a preset recognition accuracy is met: Inputting the sample image into a recognition model to be trained to obtain predicted position information and predicted stacking information; Determine a first loss function based on the predicted position information and the position label, determine a second loss function based on the predicted stack information and the stack label, and determine a target loss function based on the first loss function and the second loss function; Adjusting network parameters in the recognition model to be trained based on the target loss function; Wherein, the recognition model further includes a segmentation module, and the segmentation module is used to determine the contour segmentation area of each target object from the image to be recognized; The method further comprises: Update the position information of the corresponding target object according to the position information of the contour segmentation area of the target object; Wherein, the output result of the segmentation module also includes the pixel prediction value of each pixel point in the contour segmentation area; The determining a target loss function based on the first loss function and the second loss function includes: Determine a third loss function based on the pixel prediction value within the contour segmentation area of the target object and the pixel true value of the target object; A target loss function is determined based on the first loss function, the second loss function and the third loss function.
2. The method according to claim 1, characterized in that The position information includes the center position of the target object, or the position information includes the detection box position information of the target object; the stacking information includes the top layer and the bottom layer.
3. A method for extracting a target object, characterized in that: The target object is a sheet material, and the method comprises the following steps: Acquire a first image of at least one target object; Input the first image into a pre-trained recognition model, the recognition model is trained based on a sample image of the target object and the stacking labels and position labels of each target object in the sample image, the recognition model includes a feature extraction module and a positioning module, the feature extraction module is used to extract a feature map in the first image, the positioning module determines the position information and stacking information of each target object based on the feature map, and determines the position information of each target object in the first image and the stacking information of each target object based on an output result of the recognition model; Extracting each target object based on the position information and stacking information of each target object, determining the extraction order of each target object according to the stacking information of each target object, determining the extraction position of each target object based on the position information of each target object, and extracting each target object in sequence based on the extraction order and the extraction position of each target object; The training method of the recognition model includes: iteratively training the recognition model to be trained based on the sample image and the stacking labels and position labels of each target object in the sample image until a convergence state is reached or a preset recognition accuracy is met: inputting the sample image into the recognition model to be trained to obtain predicted position information and predicted stacking information; determining a first loss function based on the predicted position information and the position label, determining a second loss function based on the predicted stacking information and the stacking label, and determining a target loss function based on the first loss function and the second loss function; adjusting the network parameters in the recognition model to be trained based on the target loss function; wherein the recognition model also includes a segmentation module, which is used to determine the contour segmentation area of each target object from the image to be recognized; the method also includes: updating the position information of the corresponding target object according to the position information of the contour segmentation area of the target object; wherein the output result of the segmentation module also includes the pixel prediction value of each pixel point in the contour segmentation area; the determination of the target loss function based on the first loss function and the second loss function includes: determining a third loss function based on the pixel prediction value in the contour segmentation area of the target object and the pixel true value of the target object; determining the target loss function based on the first loss function, the second loss function and the third loss function.
4. The method according to claim 3, characterized in that The step of determining the extraction order of each target object according to the stacking information of each target object includes: Determine the target object with the stacking information at the top layer as the first batch of target objects; The target objects with the stacking information as the bottom layer are determined as the second batch of target objects.
5. The method according to claim 4, characterized in that The extracting each of the target objects in sequence based on the extraction order and the extraction position of each of the target objects comprises: Sequentially extracting each of the first batch of target objects based on the position information of the first batch of target objects; Acquire a second image of the second batch of target objects, and determine position information and stacking information of each of the second batch of target objects in the second image based on the recognition model; The extraction order and the extraction position of each second batch target object are determined based on the position information and the stacking information of each second batch target object, and the second batch target objects are extracted based on the extraction order and the extraction position of each second batch target object.
6. An image recognition device, characterized in that: include: A first image acquisition module, used for acquiring an image to be identified, wherein the image to be identified includes at least one target object, and the target object is an object having a stacked relationship or capable of being stacked; The first image recognition module is used to input the image to be recognized into a pre-trained recognition model, and determine the position information of each target object in the image to be recognized and the stacking information of each target object based on the output result of the recognition model, wherein the recognition model is trained based on a sample image of the target object and the stacking labels and position labels of each target object in the sample image, and the recognition model includes a feature extraction module and a positioning module, the feature extraction module is used to extract a feature map in the image to be recognized, and the positioning module determines the position information and stacking information of each target object based on the feature map; wherein the training method of the recognition model comprises: iteratively training the recognition model to be trained based on the sample image and the stacking labels and position labels of each target object in the sample image until a convergence state is reached or a preset recognition accuracy is met: inputting the sample image into the recognition model to be trained to obtain predicted position information and predicted stacking information; based on the predicted position information The method comprises the following steps: determining a first loss function based on the predicted stacking information and the stacking label, determining a second loss function based on the predicted stacking information and the stacking label, and determining a target loss function based on the first loss function and the second loss function; adjusting the network parameters in the recognition model to be trained based on the target loss function; wherein the recognition model also includes a segmentation module, and the segmentation module is used to determine the contour segmentation area of each target object from the image to be recognized; the method also includes: updating the position information of the corresponding target object according to the position information of the contour segmentation area of the target object; wherein the output result of the segmentation module also includes the pixel prediction value of each pixel point in the contour segmentation area; determining the target loss function based on the first loss function and the second loss function includes: determining a third loss function based on the pixel prediction value in the contour segmentation area of the target object and the pixel true value of the target object; determining the target loss function based on the first loss function, the second loss function and the third loss function.
7. A device for extracting a target object, characterized in that: The target object is a sheet material, and the device includes the following modules: A second image acquisition module, used to acquire a first image of at least one target object; The second image recognition module is used to input the first image into a pre-trained recognition model, the recognition model is trained based on a sample image of the target object and the stacking labels and position labels of each target object in the sample image, the recognition model includes a feature extraction module and a positioning module, the feature extraction module is used to extract a feature map in the first image, the positioning module determines the position information and stacking information of each target object based on the feature map, and determines the position information of each target object in the first image and the stacking information of each target object based on the output result of the recognition model; wherein the training method of the recognition model comprises: iteratively training the recognition model to be trained based on the sample image and the stacking labels and position labels of each target object in the sample image until a convergence state is reached or a preset recognition accuracy is met: inputting the sample image into the recognition model to be trained to obtain predicted position information and predicted stacking information; based on the predicted position information and The method comprises the following steps: determining a first loss function based on a position label, determining a second loss function based on predicted stacking information and stacking labels, and determining a target loss function based on the first loss function and the second loss function; adjusting network parameters in the recognition model to be trained based on the target loss function; wherein the recognition model also includes a segmentation module, and the segmentation module is used to determine the contour segmentation area of each target object from the image to be recognized; the method further comprises: updating the position information of the corresponding target object according to the position information of the contour segmentation area of the target object; wherein the output result of the segmentation module also includes the pixel prediction value of each pixel point in the contour segmentation area; determining the target loss function based on the first loss function and the second loss function comprises: determining a third loss function based on the pixel prediction value in the contour segmentation area of the target object and the pixel true value of the target object; determining the target loss function based on the first loss function, the second loss function and the third loss function; A target object extraction module is used to extract each target object based on the position information and stacking information of each target object, determine the extraction order of each target object according to the stacking information of each target object, determine the extraction position of each target object based on the position information of each target object, and extract each target object in turn based on the extraction order and extraction position of each target object.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the image recognition method described in any one of claims 1-2 or the target object extraction method described in any one of claims 3-5 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image recognition method described in any one of claims 1 to 2 or the target object extraction method described in any one of claims 3 to 5 is implemented.
Citation Information
Patent Citations
2D-3D display mode switching method and corresponding display equipment
CN102316336A
Multi-floor indoor positioning method based on Softmax regression multi-classification recognizer
CN110401977A
Image tampering detection method and device, computer equipment and storage medium
CN111080628A
Panoramic segmentation method with bidirectional connection and shielding processing
CN111242954A
Fruit image contour identification method
CN111325764A
