Neural network model for multi-target identification, article detection method and equipment

Abstract

The invention discloses a neural network model for multi-target recognition, and an article detection method and equipment, and the method comprises a backbone network which is used for extracting the feature information of a plurality of input images; the training module is used for performing classification construction on the feature information extracted by the backbone network to form a plurality of feature training models, and training the feature training models until a loss function is converged; wherein the loss function comprises a plurality of feature loss functions, and each feature loss function corresponds to one feature training model. The neural network model constructed by the invention can be used for multi-target recognition, and can complete recognition tasks of multiple pieces of target feature information at the same time by sharing one backbone network, so that the calculation amount of a target recognition algorithm is reduced.

Description

Technical Field

[0001] The present invention relates to the field of refrigeration devices, and in particular to a neural network model, an object detection method and a device for multi-target recognition. Background Art

[0002] At present, the object recognition methods used by smart refrigerators mainly include: manual or voice input of food types by users, radio frequency identification (RFID) scanning methods, barcode or QR code scanning methods, and image recognition technology methods. The first three methods require manual cooperation from users, which is cumbersome and less practical. Image recognition technology does not require user participation, which greatly facilitates users, but it is difficult to identify. The food stored in the refrigerator is mainly fruits and vegetables, and there are many types of fruits and vegetables, which are randomly placed and easily blocked by each other. This brings great challenges to the identification of information about items stored in smart refrigerators.

[0003] The deep learning neural network model learns the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in interpreting data such as text, images, and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images, and sounds.

[0004] The overall solution for full-space refrigeration recognition in smart refrigerators requires modules such as human segmentation, human positioning, food classification, food positioning, drawer positioning, and key point (such as drawer vertex) regression. Each module requires training a neural network model, and the computational complexity of the deep learning algorithm for so many models is enormous. Summary of the invention

[0005] The purpose of the present invention is to provide a neural network model, an object detection method and a device for multi-target recognition, so as to realize the recognition of multi-target feature information using the same model to reduce the amount of calculation.

[0006] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides a neural network model for multi-target recognition, comprising:

[0007] Backbone network, used to extract feature information of multiple input images;

[0008] A training module, classifying the feature information extracted by the backbone network to construct a plurality of feature training models, and training the feature training models until the loss function converges;

[0009] The loss function includes multiple feature loss functions, and each feature loss function corresponds to a feature training model.

[0010] As a further improvement of an embodiment of the present invention, the characteristic information at least includes food information and drawer information, the food information includes food type information and food location information, and the drawer information includes drawer vertex information and drawer location information.

[0011] As a further improvement of one embodiment of the present invention, the feature training model includes at least an ingredient training model and a drawer training model, the ingredient training model includes an ingredient type training model and an ingredient positioning training model, and the drawer training model includes a drawer vertex training model and a drawer positioning training model.

[0012] As a further improvement of an embodiment of the present invention, the feature information at least further includes human body information, and the human body information includes human body segmentation information and human body positioning information.

[0013] As a further improvement of an embodiment of the present invention, the feature training model also includes at least a human body training model, and the human body training model includes a human body segmentation training model and a human body positioning training model.

[0014] As a further improvement of an embodiment of the present invention, the backbone network includes a detection frame generation module, which is used to generate a corresponding detection frame for each feature information of the plurality of input images;

[0015] The backbone network also includes a coordinate construction module, which is used to establish a unified coordinate system for the multiple input images.

[0016] As a further improvement of an embodiment of the present invention, the loss function at least includes a loss function of a food training model, a loss function of a drawer training model, and a loss function of a human training model, and the expression of the loss function is:

[0017] L=L center +λ size L size +L off +L ketp +L keyp_off +λ seg L seg

[0018] Among them, L center is the loss function of the center point of the detection box, L size is the loss function of the detection box size, L off is the loss function of the detection box size deviation, L keyp is the loss function of the drawer vertex position, L keyp_off is the loss function of the drawer vertex position offset, L seg is the loss function of human body segmentation, λ sizeis the preset coefficient of the loss function of the detection frame size, λ seg is the preset coefficient of the human body segmentation loss function.

[0019] As a further improvement of an embodiment of the present invention, the loss function of the food training model includes a loss function of the center point position of the food detection frame, a loss function of the size of the food detection frame, and a loss function of the size deviation of the food detection frame;

[0020] The loss function of the drawer training model includes a loss function of the drawer detection frame center point position, a loss function of the drawer detection frame size, a loss function of the drawer detection frame size deviation, a loss function of the drawer vertex position, and a loss function of the drawer vertex position offset;

[0021] The loss function of the human body training model includes a loss function of the center point position of the human body detection frame, a loss function of the size of the human body detection frame, a loss function of the size deviation of the human body detection frame, and a loss function of human body segmentation.

[0022] To achieve the above-mentioned object of the invention, an embodiment of the present invention provides an object detection method, comprising:

[0023] When the equipment door is opened, multiple real-time detection images are continuously acquired;

[0024] Inputting the real-time detection image into the neural network model for multi-target recognition as described above for recognition;

[0025] The recognition result is output, and the access status of the food is determined according to the recognized human hand feature and the movement trajectory of the food that partially overlaps with the human hand feature.

[0026] To achieve the above-mentioned purpose, an embodiment of the present invention provides a refrigeration device, including: a device body, at least one camera, a memory and a processor, wherein:

[0027] The device body defines a storage space, and a plurality of drawers are arranged in the storage space for placing a plurality of food materials;

[0028] The camera is arranged at the top of the storage space and shoots in the direction of the device door;

[0029] When the door of the device is opened, the camera starts shooting; when the door of the device is closed, the camera stops shooting;

[0030] The memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of the object detection method described above are implemented.

[0031] Compared with the prior art, the neural network model constructed by the present invention can be used for multi-target recognition. By sharing a backbone network, the recognition task of multiple target feature information can be completed simultaneously, thus reducing the computational complexity of the target recognition algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a structural schematic diagram of a neural network model for multi-target recognition in one embodiment of the present invention;

[0033] Figure 2 It is a schematic flow chart of an object detection method in one embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in combination with the specific implementation methods of this application and the corresponding drawings. Obviously, the described implementation methods are only part of the implementation methods of this application, not all of the implementation methods. Based on the implementation methods in this application, all other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0035] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0036] For ease of explanation, this document uses terms that represent relative positions in space for description, such as "above", "below", "back", "front", etc., to describe the relationship of one unit or feature shown in the drawings relative to another unit or feature. Terms of relative positions in space can include different orientations of the device in use or operation other than the orientation shown in the figure. For example, if the device in the figure is turned over, the unit described as being "below" or "above" other units or features will be located "below" or "above" the other units or features. Therefore, the exemplary term "below" can encompass both the spatial orientations of below and above.

[0037] like Figure 1 As shown, this embodiment provides a neural network model for multi-target recognition, including a backbone network 100 and a training module 200.

[0038] The backbone network 100 is used to extract feature information of multiple input images. Here, the backbone network 100 adopts a single-layer feature input form to reduce the computational complexity and time consumption of multi-layer network feature extraction.

[0039] In one embodiment of the present invention, the characteristic information at least includes food information and drawer information, the food information includes food type information and food location information, and the drawer information includes drawer vertex information and drawer location information.

[0040] In another embodiment of the present invention, the characteristic information includes not only the food information and the drawer information, but also at least human body information, and the human body information includes human body segmentation information and human body positioning information.

[0041] Exemplarily, the input image is an image captured when the camera is installed on the top of the refrigerator's refrigerator compartment and facing the direction of the refrigerator door opening. After the user opens the refrigerator door, the input image captured by the camera includes at least information about the ingredients stored in the refrigerator compartment or the ingredients that the user is about to access, information about the drawers arranged in the refrigerator compartment, and information about the user's body facing the refrigerator compartment.

[0042] Specifically, the backbone network 100 includes a detection frame generation module, which is used to generate a corresponding detection frame for each feature information in the multiple input images obtained.

[0043] Of course, each detection box generated is a rectangular box, and is the smallest rectangular box that can completely surround each corresponding feature information.

[0044] Furthermore, the backbone network 100 also includes a coordinate construction module, which is used to establish a unified coordinate system for the acquired multiple input images.

[0045] For multiple input images captured by the camera, the shooting angles are fixed. For example, a vertex of the input image can be used as the origin, and a coordinate system can be established with two sides connecting the vertex as the x-axis and y-axis. Of course, for the coordinate system established by multiple input images, the origin of the coordinate system and the extension directions of the x-axis and y-axis are consistent. Based on this, the coordinate information of the detection box corresponding to each feature information can be obtained.

[0046] The training module 200 classifies the feature information extracted by the backbone network 100 to form multiple feature training models, and trains the feature training models until the loss function converges.

[0047] The loss function includes multiple feature loss functions, and each feature loss function corresponds to a feature training model.

[0048] In one embodiment of the present invention, the feature training model at least includes an ingredient training model 210a and a drawer training model 210b.

[0049] Specifically, the food training model 210a includes a food type training model 211a and a food location training model 212a, and the drawer training model 210b includes a drawer vertex training model 211b and a drawer location training model 212b.

[0050] In another embodiment of the present invention, the feature training model includes at least a human body training model 210c in addition to the food training model 210a and the drawer training model 210b.

[0051] The human body training model 210c includes a human body segmentation training model 211c and a human body positioning training model 212c.

[0052] Specifically, the loss function at least includes the loss function of the food training model 210a, the loss function of the drawer training model 210b and the loss function of the human body training model 210c, and the expression of the loss function is:

[0053] L=L center +λ size L size +L off +L ketp +L keyp_off +λ seg L seg

[0054] Among them, L center is the loss function of the center point of the detection box, L size is the loss function of the detection box size, L off is the loss function of the detection box size deviation, L keyp is the loss function of the drawer vertex position, L keyp_off is the loss function of the drawer vertex position offset, L seg is the loss function of human body segmentation, λ size is the preset coefficient of the loss function of the detection frame size, λ seg is the preset coefficient of the human body segmentation loss function.

[0055] Of course, size and λ seg Specific adjustments can be made based on actual conditions.

[0056] The formula of the above loss function can be respectively corresponded to the food training model 210a, the drawer training model 210b and the human body training model 210c for splitting.

[0057] Specifically, for the detection of food information, this embodiment uses the Anchor Free detection algorithm. In the Anchor Free algorithm, the detection frame corresponding to the food information is directly converted into a two-dimensional Gaussian distribution, and its maximum value marks the center point of the food detection frame. The loss function of the food training model 210a consists of three parts, including a loss function for the position of the center point of the food detection frame, a loss function for the size of the food detection frame, and a loss function for the size deviation of the food detection frame. In other words, the specific formula for constructing the loss function of the food training model 210a is:

[0058] L(ingredients) = L center (Ingredients)+λ size L size (Ingredients)+L off (food)

[0059] The recognition of the drawer information adopts the method of heat map key point prediction. The heat map generated by the feature map can be directly used in the backbone network 100, without the need to discretize the border detection using the default drawer border. The drawer key point detection is performed independently of the target detection task, and the key point learning is completed by regressing its offset.

[0060] Therefore, in addition to target detection of the drawer itself, detection of the drawer key points is also required for the detection of drawer information. In this embodiment, a vertex of the drawer is selected as the drawer key point. The loss function of the drawer training model 210b includes the loss function of the center point position of the drawer detection frame, the loss function of the drawer detection frame size, the loss function of the drawer detection frame size deviation, the loss function of the drawer vertex position, and the loss function of the drawer vertex position offset. In other words, the specific formula for constructing the loss function of the drawer training model 210b is:

[0061] L(Drawer)=L center (Drawer)+λ size L size (Drawer)+L off (Drawer)+L ketp (Drawer)+L keyp_off (drawer)

[0062] The semantic segmentation of the human body adopts the full convolution method. In the semantic segmentation architecture, the model classifies the image at the pixel level. The human feature map generated in the backbone network 100 is upsampled to the size of the human segmentation map, and then normalized by the Softmax function to obtain the probability of each class and generate the corresponding binary image. The loss function of the human body training model 210c includes the loss function of the center point position of the human body detection frame, the loss function of the size of the human body detection frame, the loss function of the size deviation of the human body detection frame, and the loss function of the human body segmentation. In other words, the specific formula for constructing the loss function of the human body training model 210c is:

[0063] L(human body)=L center (Human body)+λ size L size (Human body)+L off (Human body)+λ seg L seg (human body)

[0064] In this embodiment, the three visual tasks of food information detection, drawer information detection and human information detection share a backbone network 100 for training, the three detection training models are combined into one, and a total loss function is used to train and optimize multiple recognition targets.

[0065] Of course, the loss function model constructed in the present invention is not limited to the detection of food information, drawer information and human body information, but can also be a combination of other target recognition tasks. According to actual needs, the target task to be identified can be selected to construct the corresponding loss function.

[0066] like Figure 2 As shown, this embodiment provides an object detection method, including:

[0067] S1: When the device door is opened, multiple real-time detection images are continuously acquired.

[0068] S2: Inputting the real-time detection image into the neural network model for multi-target recognition as described above for recognition.

[0069] S3: Output the recognition result, and determine the access status of the food according to the recognized human hand feature and the movement trajectory of the food that partially overlaps with the human hand feature.

[0070] Specifically, after the equipment door is opened, the real-time detection image obtained includes, of course, the information of food stored in the equipment room, the information of drawers arranged in the equipment room, and the body information of the user standing facing the equipment room after the door is opened. If the user takes food from the equipment room or the user places food towards the equipment room, there will be some overlap between the user's hand features and the features of the food taken by his hand.

[0071] After the acquired real-time detection image is input into the above-mentioned neural network model for multi-target recognition, the recognition result can be directly output, and the food information and human hand information in the real-time detection image can be identified. According to the continuous real-time detection image, the human hand features and the movement trajectory of the food overlapping with the human hand features can be determined. According to whether the food moves toward the indoor direction of the equipment room or toward the outdoor direction of the equipment room over time, it can be determined whether the food is stored in the equipment room or taken out of the equipment room. At the same time, the type of food stored and accessed by the user, as well as the location where the user stores the food or the location where the user takes the food can also be directly output.

[0072] This embodiment also provides a refrigeration device, including: a device body, at least one camera, a memory, and a processor.

[0073] The device body defines a storage space, and a plurality of drawers are arranged in the storage space for placing a variety of food materials.

[0074] The camera is arranged at the top of the storage space and shoots in the direction of the device door.

[0075] When the device door is opened, the camera starts shooting; when the device door is closed, the camera stops shooting.

[0076] The memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of the object detection method described above are implemented.

[0077] In summary, the neural network model constructed by the present invention can be used for multi-target recognition. By sharing a backbone network, the recognition task of multiple target feature information can be completed simultaneously, thus reducing the computational complexity of the target recognition algorithm.

[0078] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0079] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A neural network model for multi-target recognition, characterized in that: include: Backbone network, used to extract feature information of multiple input images; A training module, classifying the feature information extracted by the backbone network to construct a plurality of feature training models, and training the feature training models until the loss function converges; The loss function includes multiple feature loss functions, and each feature loss function corresponds to a feature training model.

2. The neural network model for multi-target recognition according to claim 1, characterized in that: The characteristic information at least includes food information and drawer information, the food information includes food type information and food location information, and the drawer information includes drawer vertex information and drawer location information.

3. The neural network model for multi-target recognition according to claim 1, characterized in that: The feature training model at least includes a food training model and a drawer training model, the food training model includes a food type training model and a food positioning training model, and the drawer training model includes a drawer vertex training model and a drawer positioning training model.

4. The neural network model for multi-target recognition according to claim 2, characterized in that: The feature information at least further includes human body information, and the human body information includes human body segmentation information and human body positioning information.

5. The neural network model for multi-target recognition according to claim 3, characterized in that: The feature training model also includes at least a human body training model, and the human body training model includes a human body segmentation training model and a human body positioning training model.

6. The neural network model for multi-target recognition according to claim 5, characterized in that: The backbone network includes a detection frame generation module, which is used to generate a corresponding detection frame for each feature information of the plurality of input images; The backbone network also includes a coordinate construction module, which is used to establish a unified coordinate system for the multiple input images.

7. The neural network model for multi-target recognition according to claim 6, characterized in that: The loss function at least includes the loss function of the food training model, the loss function of the drawer training model and the loss function of the human body training model, and the expression of the loss function is: L=L center +λ size L size +L off +L ketp +L keyp_off +λ seg L seg Among them, L center is the loss function of the center point of the detection box, L size is the loss function of the detection box size, L off is the loss function of the detection box size deviation, L keyp is the loss function of the drawer vertex position, L keyp_off is the loss function of the drawer vertex position offset, L seg is the loss function of human body segmentation, λ size is the preset coefficient of the loss function of the detection frame size, λ seg is the preset coefficient of the human body segmentation loss function.

8. The neural network model for multi-target recognition according to claim 7, characterized in that: The loss function of the food training model includes a loss function of the center point position of the food detection frame, a loss function of the size of the food detection frame, and a loss function of the size deviation of the food detection frame; The loss function of the drawer training model includes a loss function of the drawer detection frame center point position, a loss function of the drawer detection frame size, a loss function of the drawer detection frame size deviation, a loss function of the drawer vertex position, and a loss function of the drawer vertex position offset; The loss function of the human body training model includes a loss function of the center point position of the human body detection frame, a loss function of the size of the human body detection frame, a loss function of the size deviation of the human body detection frame, and a loss function of human body segmentation.

9. A method for detecting an object, characterized in that: include: When the equipment door is opened, multiple real-time detection images are continuously acquired; Inputting the real-time detection image into the neural network model for multi-target recognition as described in any one of claims 1 to 8 for recognition; The recognition result is output, and the access status of the food is determined according to the recognized human hand feature and the movement trajectory of the food that partially overlaps with the human hand feature.

10. A refrigeration device comprising: The device comprises at least one camera, a memory and a processor, wherein: The device body defines a storage space, and a plurality of drawers are arranged in the storage space for placing a plurality of food materials; The camera is arranged at the top of the storage space and shoots in the direction of the device door; When the door of the device is opened, the camera starts shooting; when the door of the device is closed, the camera stops shooting; The memory stores a computer program that can be run on the processor, and the processor implements the steps of the object detection method according to claim 9 when executing the program.

Images