Object identification method, electronic device, readable storage medium and product

Abstract

The disclosed embodiments disclose an object identification method, an electronic device, a readable storage medium and a product, the method comprising obtaining the starting frame and the ending frame of the target video when the user updates the display of the items; obtaining the coordinate information of the target object in the starting frame and the ending frame respectively according to the preset target detection model; determining in sequence whether the target objects at the same position in the starting frame and the ending frame match according to the coordinate information of the target object in the starting frame and the ending frame; if there is a mismatch in the target object at the reference position, confirming that the target object at the reference position changes before and after the user updates the display of the items. According to the disclosed embodiments, the display of items in the container can be dynamically obtained by collecting the target video of each user taking the items, and the real-time monitoring of the display of items can be realized, so that the operation and maintenance personnel can confirm the quantity and / or type of items taken by the user, and avoid the situation where malicious users take items multiple times.

Description

Technical Field

[0001] The present disclosure relates to the field of terminal control technology, and in particular to an object identification method, an electronic device, a readable storage medium and a product. Background Art

[0002] In daily life, we often encounter scenarios where merchants or businesses display items to users so that users can understand the types or functions of the items and then make a purchase. In these scenarios, the companies or merchants need to monitor the displayed items to determine whether the items have been taken or replaced. Summary of the invention

[0003] In order to solve the problems in the related art, the embodiments of the present disclosure provide an object identification method, an electronic device, a readable storage medium and a product.

[0004] In a first aspect, an object identification method is provided in an embodiment of the present disclosure.

[0005] Specifically, the object identification method includes:

[0006] Acquire a start frame and an end frame of a target video when a user updates an item display, wherein the start frame includes display information of the shelf when the user has not started to update the item display, and the end frame includes display information of the shelf when the user has completed updating the item display;

[0007] Obtaining coordinate information of the target object in the start frame and the end frame respectively according to a preset target detection model; the target object at least includes items displayed on the shelf;

[0008] Determining in sequence whether target objects at the same position in the starting frame and the ending frame match according to coordinate information of the target objects in the starting frame and the ending frame;

[0009] If there is a mismatch between the target object at the reference position in the start frame and the end frame, it is confirmed that the target object at the reference position changes before and after the user updates the item display.

[0010] Optionally, the acquiring the coordinate information of the target object in the start frame and the end frame respectively according to a preset target detection model includes:

[0011] Acquire multiple bounding boxes of each target object in the start frame and the end frame respectively according to a preset target detection model, where different bounding boxes have different confidence levels;

[0012] Perform non-maximum suppression processing on multiple bounding boxes of each target object to obtain at least one remaining bounding box;

[0013] Determine, among the remaining at least one bounding box, a bounding box having the largest confidence and exceeding a predetermined bounding threshold as a bounding box corresponding to the target object;

[0014] According to the bounding box of each target object, coordinate information of each target object is obtained.

[0015] Optionally, the acquiring the coordinate information of the target object in the start frame and the end frame respectively according to a preset target detection model includes:

[0016] Obtaining coordinate information of a plurality of objects and a plurality of shelves in the start frame and the end frame respectively according to a preset target detection model;

[0017] According to the coordinate information of the plurality of objects and the plurality of shelves, obtaining the distance between two adjacent objects on the same shelf;

[0018] If the distance between two adjacent items on the same shelf is greater than or equal to a preset distance threshold, it is determined that there is an empty aisle between the two adjacent items.

[0019] Optionally, obtaining the start frame and the end frame of the target video when the user updates the item display includes:

[0020] The video frames in the target video are identified sequentially, and a video frame in which the number of objects does not change and the shelf is not blocked is obtained as the starting frame;

[0021] A video frame in the target video whose similarity with the start frame meets a preset condition after the start frame is obtained as the end frame.

[0022] Optionally, obtaining a video frame in the target video that is after the start frame and whose similarity with the start frame meets a preset condition as the end frame includes:

[0023] Eliminate the video frames in the target video that are blocked by the frames after the starting frame;

[0024] The video frame with the highest similarity to the start frame is obtained from the remaining multiple video frames as the end frame.

[0025] Optionally, obtaining, from the remaining multiple video frames, a video frame having the highest similarity to the start frame as the end frame includes:

[0026] A video frame having the smallest Hamming distance with the start frame is obtained from the remaining multiple video frames as the end frame.

[0027] Optionally, determining in sequence whether target objects at the same position in the start frame and the end frame match according to coordinate information of the target objects in the start frame and the end frame includes:

[0028] Determine the region of interest of each of the target objects in the start frame and the end frame according to the coordinate information of the target objects in the start frame and the end frame;

[0029] Identify feature points of each target object in the region of interest of each target object in the start frame and the end frame;

[0030] Obtaining the matching percentage of feature points of the target object at the same position in the starting frame and the ending frame;

[0031] If the matching percentage of the reference position is greater than or equal to a preset percentage threshold, confirming that the target object at the reference position in the starting frame and the ending frame matches;

[0032] If the matching percentage of the reference position is less than the preset percentage threshold, it is determined that the target object at the reference position in the start frame and the end frame does not match.

[0033] Optionally, the method further includes:

[0034] Acquire a plurality of training images including shelf display information, wherein the shelf display information included in the plurality of training images is not completely the same;

[0035] Label the bounding box of the target object in each training image respectively, and obtain multiple labeled training images;

[0036] Performing image enhancement on the labeled multiple training images to obtain multiple enhanced training images;

[0037] The target detection model is obtained by training a model through a target detection framework according to the enhanced multiple training images.

[0038] Optionally, the method further includes:

[0039] Obtaining coordinate information of the user's limbs according to a preset target detection model;

[0040] If there are multiple target objects at the same position in the start frame and the end frame that do not match, the reference position where the target object changes before and after updating the display of items is determined from the multiple same positions according to the coordinate information of the user's limb.

[0041] In a second aspect, an article display device is provided in an embodiment of the present disclosure.

[0042] Specifically, the article display device comprises:

[0043] A video acquisition module is configured to acquire a start frame and an end frame of a target video when a user updates an item display, wherein the start frame includes display information of the shelf when the user has not started to update the item display, and the end frame includes display information of the shelf when the user has completed updating the item display;

[0044] A coordinate acquisition module is configured to acquire coordinate information of a target object in the start frame and the end frame respectively according to a preset target detection model; the target object at least includes items displayed on a shelf;

[0045] A determination module is configured to determine, in sequence, whether target objects at the same position in the start frame and the end frame match according to coordinate information of the target objects in the start frame and the end frame;

[0046] The confirmation module is configured to confirm that the target object at the reference position changes before and after the user updates the item display if there is a mismatch between the target object at the reference position in the start frame and the end frame.

[0047] Optionally, the coordinate acquisition module is also configured to respectively acquire multiple bounding boxes of each target object in the starting frame and the ending frame according to a preset target detection model, and different bounding boxes have different confidence levels; perform non-maximum suppression processing on the multiple bounding boxes of each target object to acquire at least one remaining bounding box; determine, among the at least one remaining bounding box, a bounding box with the largest confidence level and exceeding a predetermined boundary threshold as the bounding box of the corresponding target object; and acquire the coordinate information of each target object based on the bounding box of each target object.

[0048] Optionally, the coordinate acquisition module is also configured to respectively acquire the coordinate information of multiple objects and multiple shelves in the starting frame and the ending frame according to a preset target detection model; acquire the distance between two adjacent objects on the same shelf according to the coordinate information of the multiple objects and the multiple shelves; if the distance between two adjacent objects on the same shelf is greater than or equal to a preset distance threshold, it is determined that there is an empty aisle between the two adjacent objects.

[0049] Optionally, the video acquisition module is also configured to identify the video frames in the target video in sequence, and acquire a video frame in which the number of objects no longer changes and the partitions are not blocked as the starting frame; and acquire a video frame in the target video whose similarity with the starting frame after the starting frame meets a preset condition as the ending frame.

[0050] Optionally, the video acquisition module is further configured to remove video frames in the target video that are blocked by a frame after the start frame; and acquire a video frame with the highest similarity to the start frame from the remaining multiple video frames as the end frame.

[0051] Optionally, the video acquisition module is further configured to acquire, from the remaining multiple video frames, a video frame having the smallest Hamming distance with the start frame as the end frame.

[0052] Optionally, the determination module is configured to determine the area of ​​interest of each target object in the starting frame and the ending frame based on the coordinate information of the target object in the starting frame and the ending frame; identify the feature points of each target object in the area of ​​interest of each target object in the starting frame and the ending frame; obtain the matching percentage of the feature points of the target objects at the same position in the starting frame and the ending frame; if the matching percentage of the reference position is greater than or equal to a preset percentage threshold, confirm that the target object at the reference position in the starting frame and the ending frame matches; if the matching percentage of the reference position is less than the preset percentage threshold, confirm that the target object at the reference position in the starting frame and the ending frame does not match.

[0053] Optionally, the device further comprises:

[0054] An image acquisition module is configured to acquire a plurality of training images including shelf display information, wherein the shelf display information included in the plurality of training images is not completely the same;

[0055] An image annotation module is configured to respectively annotate a bounding box of a target object in each training image, and obtain a plurality of annotated training images;

[0056] An image enhancement module is configured to perform image enhancement on the labeled multiple training images to obtain multiple enhanced training images;

[0057] The model training module is configured to train the model through the target detection framework according to the enhanced multiple training images to obtain the target detection model.

[0058] Optionally, the coordinate acquisition module is also configured to acquire coordinate information of the user's limbs according to a preset target detection model; the determination module is also configured to determine the reference position where the target object changes before and after updating the item display according to the coordinate information of the user's limbs in the multiple same positions if there are multiple target objects in the same position that do not match in the starting frame and the ending frame.

[0059] In a third aspect, an embodiment of the present disclosure provides a computer program product, which includes computer instructions, and when the computer instructions are executed by a processor, the method described in any embodiment of the first aspect is implemented.

[0060] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer instructions, and wherein the one or more computer instructions are executed by the processor to implement a method as described in any embodiment of the first aspect.

[0061] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the method described in any embodiment of the first aspect is implemented.

[0062] According to an embodiment of the present disclosure, an object recognition method is provided. The method can obtain the starting frame and the ending frame of the target video when the user updates the display of the object, and then obtain the coordinate information of the target object in the starting frame and the ending frame respectively according to the preset target detection model; and according to the coordinate information of the target object in the starting frame and the ending frame, determine in sequence whether the target object at the same position in the starting frame and the ending frame matches; if there is a mismatch between the target object at the reference position in the starting frame and the ending frame, it is confirmed that the target object at the reference position changes before and after the user updates the display of the object, that is, the target object at the reference position in the starting frame may be taken away or changed. In this way, the display of the items in the container can be dynamically obtained through the collected target video of each user taking the item, and the real-time monitoring of the display of the item can be realized, so that the operation and maintenance personnel can confirm the number and / or type of items taken by the user, and avoid the situation where malicious users take them multiple times.

[0063] At the same time, since the number and type of items taken by each user can be obtained, real-time monitoring of item inventory can be achieved, so that operation and maintenance personnel can replenish inventory in time when they confirm that the inventory is small. In addition, operation and maintenance personnel can also count the types of items most popular with users or customers, which is convenient for adjusting the planning of item display later.

[0064] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0066] Figure 1 A structural diagram of an article display device according to an embodiment of the present disclosure is shown;

[0067] Figure 2 A schematic structural block diagram of a mainboard according to an embodiment of the present disclosure is shown;

[0068] Figure 3 A schematic structural block diagram of a control panel according to an embodiment of the present disclosure is shown;

[0069] Figure 4 A flowchart showing an object identification method according to an embodiment of the present disclosure;

[0070] Figure 5 A flowchart showing an object identification method according to an embodiment of the present disclosure;

[0071] Figure 6 A flowchart showing an object identification method according to an embodiment of the present disclosure;

[0072] Figure 7 A flowchart showing an object identification method according to an embodiment of the present disclosure;

[0073] Figure 8 A flowchart showing an object identification method according to an embodiment of the present disclosure;

[0074] Fig. 9 A schematic diagram showing the structure of an article display device according to an embodiment of the present disclosure is shown;

[0075] Fig.10 A schematic diagram showing the structure of an article display device according to an embodiment of the present disclosure is shown;

[0076] Fig.11 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0077] Fig.12 A schematic diagram showing the structure of a computer system according to a method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0078] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0079] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0080] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0081] The item display scenario described in the embodiment of the present disclosure is to place multiple items to be displayed in an item display cabinet in sequence, so that users can view, use or purchase them. For example, the item display cabinet can be an unmanned vending machine in a public place, a refrigerator in a supermarket, a snack rack and a beverage rack in a company's tea room, or a sample display cabinet in a factory exhibition area. It should be noted that the embodiment of the present application does not impose any restrictions on the specific type of the item display device.

[0082] Taking offline vending machines as an example, there are multiple items for sale displayed in the vending machines, and users can purchase them according to their needs. The method of this application can be used to process the collected videos of users purchasing items, and the types of items purchased by users can be known, which is convenient for suppliers to count the popularity of items.

[0083] Or, taking the snack rack and drink rack in the company's tea room as an example, the company places the snacks and drinks provided on the snack rack and drink rack, and employees can take them as needed. The method of this application can process the video of employees taking snacks or drinks, and obtain the popularity of each snack or drink, so that the company can optimize the types of snacks and drinks.

[0084] Alternatively, taking the sample display cabinet in the factory exhibition area as an example, the factory can place the manufactured products in the sample display cabinet, and customers can view and try each product to determine the performance of the product. The method of this application can process the video of customers trying out the products, determine the most popular products among customers, and adjust the factory's subsequent production plan.

[0085] Alternatively, taking a refrigerator in a supermarket as an example, the supermarket places various items that need to be refrigerated, such as yogurt or ice cream, in the refrigerator for sale, so that consumers can purchase them according to their needs. The method of the present application can be used to process the video of the user picking up the items, so as to know the types of items purchased by the user, which is convenient for the supermarket to adjust the subsequent supply plan.

[0086] For example, Figure 1 A schematic structural block diagram of an article display device 100 according to an embodiment of the present disclosure is shown. Figure 1 As shown, the article display device 100 includes a plurality of shelves 10a, each shelf 10a is provided with a plurality of aisles, each aisle can be provided with a plurality of articles 10c, and the articles 10c displayed in different aisles can be the same or different.

[0087] Furthermore, the article display device 100 may also be provided with a cabinet 10d and a cabinet door 10e, wherein the plurality of shelves 10a are provided in the cabinet 10d, and the cabinet door 10e is used to close the cabinet 10d, so as to isolate the shelves 10a for displaying articles from the outside world, and prevent the displayed articles from being accidentally touched and dropped by external objects. The fixed side of the cabinet door 10e is rotatably connected to the edge of the opening end of the cabinet 10d, so that the cabinet door 10e can be opened by rotation.

[0088] Optionally, the article display device 100 may be provided with a camera 10f, which may be provided at any position on the side of the cabinet door 10e close to the partition 10a. Preferably, in order to ensure that the camera 10f can capture the complete images of multiple partitions 10a, the camera 10f may be provided on the side where the handle of the cabinet door 10e is located, and the handle is the force-applying part when the user opens the cabinet door 10e. As the cabinet door 10e is opened, the camera 10f captures more partitions 10a, and when the cabinet door 10e is opened to the maximum, the camera 10f can capture the images of all the partitions 10a.

[0089] Specifically, the article display device 100 may also have a temperature control function, that is, the article display device 100 may include a compressor, a condenser, a throttling element and an evaporator, wherein the compressor, the condenser, the throttling element and the evaporator are connected through pipes filled with refrigerant to form a closed pipeline, constituting a refrigeration system or heating system that can circulate refrigerant.

[0090] Among them, the compressor refers to a driven fluid machinery used to increase the low-pressure refrigerant to a high-pressure refrigerant. The compressor can inhale low-temperature and low-pressure gaseous refrigerant, drive the piston to compress the refrigerant through the operation of the motor, and then discharge the high-temperature and high-pressure gaseous refrigerant to provide power for the refrigeration cycle. The compressor can include reciprocating compressors, screw compressors, rotary compressors, scroll compressors and centrifugal compressors, etc. The embodiments of the present application do not impose any restrictions on the specific types of compressors.

[0091] The condenser refers to a heat exchanger used to exchange heat between the refrigerant in the condenser and the air outside the condenser to achieve heat release. Specifically, the condenser may include a long pipe for accommodating the refrigerant, which may usually be made of a metal material with strong thermal conductivity such as copper, and the pipe may usually be coiled into a spiral shape. In addition, in order to improve the heat exchange efficiency of the condenser, a heat sink with excellent thermal conductivity may be provided on the pipe to increase the heat dissipation area, thereby accelerating the speed of heat exchange and improving the heat exchange efficiency. It is also possible to speed up the flow of air around the condenser by providing a blower or fan matching the condenser, thereby accelerating the speed of heat exchange and improving the heat exchange efficiency.

[0092] The throttling element is used to throttle the liquid refrigerant at room temperature and high pressure through the throttling element to become a low-temperature and low-pressure gas refrigerant, wherein the throttling element can also be called a throttling element or a regulating valve, and the throttling element can include an expansion valve, a capillary tube, etc. In addition, the throttling element can also control the flow of the refrigerant flowing through the throttling element to avoid the flow of the refrigerant flowing through the throttling element being too large or too small. Among them, if the flow of the refrigerant flowing through the throttling element is too large, the refrigerant flowing out of the throttling element will still include liquid refrigerant, and the liquid refrigerant will enter the compressor to produce liquid hammer, causing damage to the compressor; if the flow of the refrigerant flowing through the throttling element is too small, it will cause too little refrigerant to enter the compressor, reducing the working efficiency of the compressor.

[0093] The evaporator refers to a heat exchanger used to exchange heat between the refrigerant in the evaporator and the air outside the condenser to achieve heat absorption. Specifically, the evaporator may include a long pipe for accommodating the refrigerant, which may usually be made of a metal material with strong thermal conductivity such as copper, and the pipe may usually be coiled into a spiral shape. In addition, in order to improve the heat exchange efficiency of the condenser, a heat sink with excellent thermal conductivity may be provided on the pipe to increase the heat dissipation area, thereby accelerating the speed of heat exchange and improving the heat exchange efficiency. It is also possible to provide a blower or fan matching the evaporator to speed up the flow of air around the evaporator, thereby accelerating the speed of heat exchange and improving the heat exchange efficiency.

[0094] Refrigerant can also be called refrigerant, refrigerant or snow seed, which refers to the medium substance that completes energy conversion in the refrigeration system or heating system. Refrigerant is usually a substance that is easy to undergo reversible phase change. Through reversible phase change, the refrigerant can transfer heat. Specifically, when the gaseous refrigerant is under pressure, it releases heat and becomes liquid. When the high-pressure liquid is decompressed and becomes gas, it absorbs heat. Refrigerants can include ammonia, air, water, salt water, Freon, etc., among which Freon can include monochlorotrifluoromethane, dichlorodifluoromethane, trifluoromethane, tetrafluoroethane, trichlorodifluoroethane, etc.

[0095] When the article display device 100 has a refrigeration function, the low-temperature and low-pressure vapor refrigerant flows from the evaporator into the compressor, and the compressor compresses the low-temperature and low-pressure vapor refrigerant and allows the high-temperature and high-pressure gaseous refrigerant to flow into the condenser; the high-temperature and high-pressure gaseous refrigerant exchanges heat with the air outside the condenser through the condenser, so that the high-temperature and high-pressure gaseous refrigerant is cooled into a normal-temperature and high-pressure liquid refrigerant in the condenser, and then the normal-temperature and high-pressure liquid refrigerant flows into the throttling element, and the throttling element throttles the normal-temperature and high-pressure liquid refrigerant, so that the refrigerant flowing out of the throttling element is converted into a low-temperature and low-pressure liquid refrigerant; the low-temperature and low-pressure liquid refrigerant flows into the evaporator, and the low-temperature and low-pressure liquid refrigerant exchanges heat with the air outside the evaporator through the evaporator, and the low-temperature and low-pressure liquid refrigerant evaporates and gasifies into a low-temperature and low-pressure gaseous refrigerant to absorb heat. The air outside the evaporator can be introduced into the storage area of ​​the article display device 100, and the air outside the condenser can be introduced into the outside of the article display device 100, thereby transferring the heat in the storage area of ​​the article display device 100 (a plurality of shelves 10a can be arranged in the storage area) to the outside of the article display device 100, and cooling the storage area of ​​the article display device 100.

[0096] When the article display device 100 has a heating function, the low-temperature and low-pressure vapor refrigerant flows from the condenser into the compressor, and the compressor compresses the low-temperature and low-pressure vapor refrigerant and allows the high-temperature and high-pressure gaseous refrigerant to flow into the evaporator; the high-temperature and high-pressure gaseous refrigerant exchanges heat with the air outside the evaporator through the evaporator, so that the high-temperature and high-pressure gaseous refrigerant is cooled into a normal-temperature and high-pressure liquid refrigerant in the evaporator, and then the normal-temperature and high-pressure liquid refrigerant flows into the throttling element, and the throttling element throttles the normal-temperature and high-pressure liquid refrigerant, so that the refrigerant flowing out of the throttling element is converted into a low-temperature and low-pressure liquid refrigerant; the low-temperature and low-pressure liquid refrigerant flows into the condenser, and the low-temperature and low-pressure liquid refrigerant exchanges heat with the air outside the condenser through the condenser, and the low-temperature and low-pressure liquid refrigerant evaporates and gasifies into a low-temperature and low-pressure gaseous refrigerant to absorb heat. The air outside the evaporator can be introduced into the storage area of ​​the article display device 100, and the air outside the condenser can be introduced into the outside of the article display device 100, thereby transferring the heat outside the article display device 100 to the storage area of ​​the article display device 100, and heating the storage area of ​​the article display device 100.

[0097] In one embodiment of the present application, a control board and a power management module are disposed in the cabinet of the object display device 100 , and a main board is disposed in the cabinet door of the object display device 100 .

[0098] In one embodiment of the present application, Figure 2 A schematic structural block diagram of a mainboard according to an embodiment of the present disclosure is shown. Figure 2As shown, the mainboard 200 includes a processor 201 , a random access memory 202 , a flash memory 203 , a wireless local area network Bluetooth module 204 , a gyroscope 205 , a pressure sensor 206 , a microphone 207 , a speaker 208 , a camera 209 and a cellular communication module 210 .

[0099] The processor may include one or more processing units, for example, the processor may include one or more of an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a memory, a video codec, a digital signal processor, a baseband processor, and / or a neural network processor. Different processing units may be independent devices or integrated into one or more processors.

[0100] Among them, the image signal processor ISP is used to process the data fed back by the camera. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the image signal processor ISP for processing and converts it into an image visible to the naked eye. The image signal processor ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The image signal processor ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the image signal processor ISP can be set in the camera.

[0101] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, digital signal processors can be used to perform Fourier transform on frequency point energy.

[0102] The video codec is used to compress or decompress digital video. The object display device 100 can support one or more video codecs. In this way, the object display device 100 can play or record videos in multiple encoding formats.

[0103] The neural network computing processor can quickly process input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn. The neural network computing processor can realize applications such as intelligent cognition of the object display device 100, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0104] In some embodiments, the processor may include one or more interfaces. The interface may include an integrated circuit interface, an integrated circuit built-in audio interface, a pulse code modulation interface, a universal asynchronous receiver and transmitter interface, a mobile industry processor interface, a general purpose input and output interface, a subscriber identity module interface, and / or a universal serial bus interface, etc.

[0105] The random access memory 202 can be used to store computer executable program codes, which include instructions and data. The processor 201 executes various functional applications and data processing of the object display device 100 by running the instructions stored in the random access memory 202. The random access memory 202 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data (such as audio data, image data, etc.) created during the use of the object display device 100, etc.

[0106] The flash memory 203 can be used to expand the storage capacity of the article display device 100. The flash memory 203 can communicate with the processor 201 through a flash memory interface to implement a data storage function, such as storing music, video and other files in the flash memory.

[0107] The processor 201 , the random access memory 202 and the flash memory 203 can form a minimum system to provide a system operating environment.

[0108] The wireless LAN Bluetooth module 204 can provide wireless communication solutions including wireless LAN, Bluetooth, global navigation satellite system, frequency modulation, short-range wireless communication technology, infrared technology, etc. applied to the item display device 100. The wireless LAN Bluetooth module 204 can be one or more devices integrating at least one communication processing module. The wireless LAN Bluetooth module 204 receives electromagnetic waves via an antenna, modulates the frequency of the electromagnetic wave signal and performs filtering processing, and sends the processed signal to the processor 201. The wireless LAN Bluetooth module 204 can also receive a signal to be sent from the processor 201, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna. In one embodiment of the present application, communication with the user's terminal can be performed through the wireless LAN Bluetooth module.

[0109] The cellular communication module 210 can provide a solution for wireless communications including 2G / 3G / 4G / 5G applied to the article display device 100. The cellular communication module 210 may include at least one filter, a switch, a power amplifier, a low noise amplifier, etc. The cellular communication module 210 can receive electromagnetic waves by an antenna, filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The cellular communication module 210 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna. In some embodiments, at least some of the functional modules of the cellular communication module 210 can be set in the processor 201. In some embodiments, at least some of the functional modules of the cellular communication module 210 can be set in the same device as at least some of the modules of the processor 201. In one embodiment of the present application, the cellular communication module 210 can communicate with the cloud server of the maintenance service provider of the article display device 100.

[0110] Through the wireless LAN Bluetooth module 204 and the cellular communication module 210, the article display device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications, general packet radio service, code division multiple access, wideband code division multiple access, time division code division multiple access, long term evolution, etc.

[0111] The gyroscope 205 may be used to determine the real-time posture of the door of the object display apparatus 100 .

[0112] The pressure sensor 206 is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 206 can be set on the display screen. There are many types of pressure sensors 206, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 206, the capacitance between the electrodes changes, and the intensity of the pressure is determined according to the change in capacitance. When a touch operation acts on the display screen, the touch operation intensity is detected by the pressure sensor 206, and the touch position can also be calculated according to the detection signal of the pressure sensor 206. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the beverage selection application icon, an instruction to view the specific information of the beverage is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the beverage selection application icon, an instruction to purchase a beverage is executed.

[0113] Microphone 207, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 207 with their mouth to input the sound signal into the microphone 207. The object display device 100 can be provided with at least one microphone 207. In other embodiments, the object display device 100 can be provided with two microphones 207, which can not only collect sound signals but also realize a noise reduction function. In other embodiments, the object display device 100 can also be provided with three, four or more microphones 207 to collect sound signals, reduce noise, identify the source of sound, realize a directional recording function, etc. In one embodiment of the present application, the sound of the object display device 100 when it is running can be collected by the microphone 207.

[0114] The speaker 208, also called a "loudspeaker", is used to convert an audio electrical signal into a sound signal. The object display device 100 can play music or a prompt voice through the speaker 208.

[0115] The camera 209 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device or a complementary metal oxide semiconductor phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the image signal processor to convert it into a digital image signal. The image signal processor outputs the digital image signal to the digital signal processor for processing. The digital signal processor converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the item display device 100 may include one or more cameras 209. In one embodiment of the present application, the camera 209 may have the function of heating itself to ensure that its lens does not fog up.

[0116] In one embodiment of the present application, Figure 3 A schematic structural block diagram of a control panel according to an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the control board 300 includes a power input interface 301, a power output interface 302, a metering chip 303, a micro control unit chip 304, a real-time clock chip, a light switch interface 305, a temperature control switch interface 306, an evaporating fan interface 307, a compressor interface 308, a condensing fan interface 309, a temperature sensor interface 310, a communication interface 311 and a power interface 312.

[0117] Among them, the metering chip 303 is a power sensor, and voltage data, current data, real-time power data and average power data can be obtained through the metering chip 303. The time of the microcontroller chip can be maintained through the real-time clock chip. The control signal of the light switch of the object display device 100 can be received through the light switch interface 305. The control signal of the temperature control switch of the object display device 100 can be received through the temperature control switch interface 306. The evaporation fan control signal can be sent to the evaporation fan of the object display device 100 through the evaporation fan interface 307 to control the operation of the evaporation fan. The compressor control signal can be sent to the compressor of the object display device 100 through the compressor interface 308 to control the operation of the compressor. The condensing fan control signal can be sent to the condensing fan of the object display device 100 through the condensing fan interface 309 to control the operation of the condensing fan. The temperature sensor data collected by one or more temperature sensors can be received through the temperature sensor interface 310 to determine the temperature value of one or more positions of the object display device 100.

[0118] In one embodiment of the present application, the power management module includes an AC to DC conversion module, a charging management module and a battery. The power management module is used to power the mainboard and the control board and manage the charging and discharging of the battery. The power management module can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module can also be set in the processor.

[0119] In one embodiment of the present application, the article display device 100 further includes a display screen. The article display device 100 implements a display function through a graphics processor, a display screen, and an application processor. The graphics processor is a microprocessor for image processing, which is connected to the display screen and the application processor. The graphics processor is used to perform mathematical and geometric calculations for graphics rendering. The processor may include one or more graphics processors that execute program instructions to generate or change display information.

[0120] The display screen is used to display images, videos, etc. The display screen includes a display panel. The display panel can be a liquid crystal display, an organic light emitting diode, an active matrix organic light emitting diode or an active matrix organic light emitting diode, a flexible light emitting diode, a quantum dot light emitting diode, etc. In some embodiments, the object display device 100 may include one or more display screens.

[0121] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the object display device 100. In other embodiments of the present application, the object display device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0122] Figure 4 A flowchart of an object identification method according to an embodiment of the present disclosure is shown. Figure 4 As shown, the object identification method includes the following steps 401 to 405:

[0123] In step 401, the start frame and the end frame of the target video when the user updates the item display are obtained, wherein the start frame includes the display information of the shelf when the user has not started to update the item display, and the end frame includes the display information of the shelf when the user has completed updating the item display.

[0124] In step 402, coordinate information of the target object in the start frame and the end frame is obtained respectively according to a preset target detection model; the target object at least includes items displayed on the shelf.

[0125] In step 403, based on the coordinate information of the target object in the start frame and the end frame, it is determined in turn whether the target object at the same position in the start frame and the end frame matches.

[0126] In step 404, if there is a mismatch between the target object at the reference position in the start frame and the end frame, it is confirmed that the target object at the reference position changes before and after the user updates the item display.

[0127] For example, the embodiments of the present disclosure are described using an unmanned vending machine as an example. A camera can be set on the door handle on the inside of the cabinet door of the unmanned vending machine. When the user opens the cabinet door, the camera turns on and starts to collect image information in the direction of the shelf to form a target video of the user.

[0128] When the user has finished shopping or is in the process of recording the target video, the start frame and end frame of the target video can be obtained, that is, the video frame when the user has not yet taken the required items is obtained as the start frame, and the start frame is the image information of the shelf when the user has not yet taken the required items, including the display information of the shelf when the user has not started to update the display of items; and the video frame when the user has taken the items is obtained as the end frame, and the end frame is the image information when the user has taken the items, including the display information of the shelf when the user has completed the update of the display of items. By comparing the difference between the start frame and the end frame, the type and / or quantity of the items purchased by the user (the number of items taken from different aisles) can be determined, and dynamic monitoring of the display of items can be realized. At the same time, the recorded inventory of the item can be updated according to the obtained sales record, achieving the purpose of real-time monitoring of the inventory of items.

[0129] Specifically, a target detection model can be pre-set, and the target detection model can identify the coordinate information of all target objects in the image information of the shelf. After the unmanned vending machine obtains the start frame and the end frame of the target video, the start frame and the end frame can be respectively input into the preset target detection model, and the target detection model can output the coordinate information of all target objects included in the start frame and the end frame, for example, the coordinate information of the items placed in each aisle on the start frame and the end frame. In practical applications, the target detection model can also be used to identify the coordinate information of the shelf partitions included in the start frame and the end frame, the coordinate information of the empty aisles, and the coordinate information of the user's limbs when purchasing items, for example, the limbs can be the user's hands. According to the coordinate information of the partitions, the coordinate information of the empty aisles, and the coordinate information of the user's limbs, the coordinate information of the items placed in each aisle can be assisted in locating, so that the coordinate information of the items recognized by the target detection model is more accurate.

[0130] According to the coordinate information of all target objects included in the start frame and the end frame, determine in turn whether the target objects at the same position in the start frame and the end frame match, that is, determine whether the items placed on the same aisle of the same shelf in the start frame and the end frame match. Taking the reference position as an example, the reference position is any position in the start frame and the end frame, that is, any shelf and any aisle on the shelf. If the target object at the reference position matches, it means that the item at the reference position has not been sold; if it does not match, that is, the user may place different items at the reference position before and after updating the item display, or there is an item placed at the reference position in the start frame, and the reference position becomes an empty aisle in the end frame, indicating that the item at the reference position has changed before or after the user's shopping, that is, the previous items on the aisle have been sold. At this time, the unmanned vending machine can update the inventory information of the item based on the item on the shelf and the aisle in the start frame.

[0131] It should be noted that the embodiments of the present disclosure are only described using unmanned vending machines as an example. It can be understood that the method of the present application can also be implemented on other item display devices. It is only necessary to set the camera image at a suitable position to monitor the shelf array information.

[0132] For example, before identifying the start frame and the end frame of the target video, it is necessary to obtain the target detection model through training in advance. Specifically, multiple training images can be obtained in advance, and the training images may include the array information of the shelf and the limb information of the user when updating the display of the items, and the array information or the limb information of the user included in the multiple training images are not completely the same, that is, the multiple training images can reflect the image information of the shelf at multiple angles and multiple times. Then, the bounding box of the target object in each training image is marked respectively, and multiple training images after marking are obtained, that is, the boundary of the target object in each training image is marked with a pre-selected box, for example, the bounding box of the shelf, the bounding box of the item, the bounding box of the empty cargo aisle, the bounding box of the user's limb, etc. are marked respectively in each training image. Then, the image enhancement is performed on the multiple training images after marking to obtain multiple enhanced training images, and finally, the model can be trained through the target detection framework according to the multiple enhanced training images to obtain the target detection model.

[0133] For example, multiple video frames from multiple target videos can be intercepted as multiple training images, and the multiple training images can be annotated. Then, random rotation, random cropping and mosaic data enhancement methods can be used to change the saturation and brightness of the images in the annotated data set, or the hue of the images can be changed to enhance the training images, and the training images can be scaled proportionally to adapt to the input size of the network. Using the above-mentioned enhanced training images for model training can ensure that the model can still effectively identify the target under the conditions of illumination changes, partial occlusion, geometric changes, scale changes, and perspective changes. Specifically, in order to facilitate training, the multiple training images obtained can be divided into training sets and test sets, and the ratio of the number of images occupied by the training set and the test set is 7:3.

[0134] Next, the K-means algorithm is used to cluster the multiple training images included in the training set, and multiple, for example, nine different bounding box sizes that meet the scale characteristics of the target object to be detected are determined for each target object (because different training images may be images of shelves taken at different angles and distances, the bounding box sizes of the target object on different training images are different. Here, it means that based on the multiple training images, multiple bounding boxes from images taken at different angles and distances that meet the scale characteristics of the same target object are determined). Optionally, the training set can be trained with the target detection framework (YOLOV5). In order to achieve good training results, the network structure can be modified on the basis of the original YOLOV5, and the last CSPNet (Cross Stage PartialNetworks) structure of the backbone is replaced with a transformer structure. The upper limit of the model's accuracy can increase with the growth of the training set; at the same time, adding label smoothing to the model can prevent the model from predicting labels too confidently during training and improve the problem of poor generalization ability; focal loss can also be introduced to solve the problem of difficult sample convergence difficulties.

[0135] After the model to be trained is constructed, the model can be trained using the target detection framework using multiple training images in the training set until the model no longer converges, and the target detection model can be obtained. After the training is completed, the target detection model can be tested using multiple training images included in the aforementioned test set to determine its prediction accuracy.

[0136] Specifically, the output format of the target detection model can be in json format, as shown below:

[0137]

[0138]

[0139]

[0140] Among them, detail: is a two-dimensional list. The first dimension of the list indicates the number of shelves the item is on, and the second dimension indicates the horizontal position of the shelf. The content inside is a key-value pair.

[0141] sku_code: The SKU code of the item.

[0142] sku_name: The SKU name of the item.

[0143] score: the score of model inference.

[0144] location: The coordinate position of the object in the image, where the upper left corner of the image is the origin, the horizontal axis is the x-axis, and the vertical axis is the y-axis. Top_x is the x-axis coordinate of the detected upper left corner of the object, top_y is the y-axis coordinate of the detected upper left corner of the object, width is the width of the detected object, and height is the height of the detected object.

[0145] rows: is a two-dimensional list. The first dimension is the number of rows (intervals). The second dimension contains five values, including:

[0146] The first value is the position of the upper left corner of the identified shelf mapped to the x-axis.

[0147] The second value is the position of the identified upper left corner of the shelf mapped to the y-axis.

[0148] The third value is the position of the lower right corner of the identified shelf mapped to the x-axis.

[0149] The fourth value is the position of the lower right corner vertex of the identified shelf mapped to the y-axis.

[0150] The fifth value is the score of the model row inference.

[0151] hands: is a two-dimensional list. The first dimension is the number of hands (user limbs). The second dimension contains five values, including:

[0152] The first value is the position of the upper left vertex of the identified limb mapped to the x-axis.

[0153] The second value is the position of the upper left corner of the identified limb mapped to the y-axis.

[0154] The third value is the position of the lower right vertex of the identified limb mapped to the x-axis.

[0155] The fourth value is the position of the lower right vertex of the identified limb mapped to the y-axis.

[0156] The fifth value is the score of the model row inference.

[0157] In one embodiment, Figure 5 As shown, the step 402, i.e., the step of respectively acquiring the coordinate information of the target object in the start frame and the end frame according to the preset target detection model, can be implemented through steps 4021 to 4024:

[0158] In step 4021, multiple bounding boxes of each target object in the start frame and the end frame are respectively obtained according to a preset target detection model, and different bounding boxes have different confidence levels.

[0159] In step 4022, non-maximum suppression processing is performed on multiple bounding boxes of each target object, and at least one remaining bounding box is selected.

[0160] In step 4023, a bounding box with the maximum confidence and exceeding a predetermined bounding threshold is determined among the at least one remaining bounding box as the bounding box corresponding to the target object.

[0161] In step 4024, coordinate information of each target object is obtained according to the bounding box of each target object.

[0162] For example, since multiple bounding boxes are set for each target object when training the target detection model, the target detection model can output multiple bounding boxes for the same target object. At the same time, according to the output format of the above target detection model, the target detection model can predict the confidence of the bounding box (determined by the four coordinate information of the two vertices on the diagonal of the bounding box) of each target object.

[0163] That is, after the acquired start frame and end frame are input into the target detection model, the target detection model can output multiple bounding boxes of each target object in the start frame and the end frame, and different bounding boxes have different confidences. At this time, the multiple bounding boxes of each target object acquired can be subjected to non-maximum suppression processing, that is, the bounding boxes with the maximum confidence and the minimum confidence are removed, and then the bounding box with the maximum confidence and exceeding the predetermined boundary threshold is determined in at least one remaining bounding box as the bounding box of the corresponding target object, and then the coordinate information of each target object is obtained according to the bounding box of the target object finally determined.

[0164] In one embodiment, Figure 6 As shown, the step 402, i.e., the step of respectively acquiring the coordinate information of the target object in the start frame and the end frame according to the preset target detection model, can be implemented through steps 4025 to 4027:

[0165] In step 4025, coordinate information of multiple objects and multiple shelves in the start frame and the end frame are respectively obtained according to a preset target detection model.

[0166] In step 4026, the distance between two adjacent objects on the same shelf is obtained based on the coordinate information of the multiple objects and the multiple shelves.

[0167] In step 4027, if the distance between two adjacent items on the same shelf is greater than or equal to a preset distance threshold, it is determined that there is an empty aisle between the two adjacent items.

[0168] For example, in actual applications, by training the target detection model, the target detection model can be made to directly output the coordinate information of the empty aisle. At the same time, the target detection model can also be used to identify the coordinate information of multiple items and multiple shelves included in the start frame and the end frame, and the distance between two adjacent items on the same shelf can be obtained according to the coordinate information of the multiple items and multiple shelves. If the distance between two adjacent items on the same shelf is greater than or equal to the preset distance threshold, it is determined that there is an empty aisle between the two adjacent items. At this time, the number of empty aisles between them can be determined according to the distance between the two adjacent items. Specifically, a size threshold of an empty aisle is set in advance, and then the multiple relationship between the distance between the two adjacent items and the size threshold is determined. The multiple is rounded to the number of empty aisles, and then the boundary box of the empty aisle can be determined according to the distance between the two adjacent items and the size threshold. If the distance between two adjacent items on the same shelf is less than the preset distance threshold, it means that there is no empty aisle between the two adjacent items.

[0169] In one embodiment, Figure 7 As shown, the step 401, i.e., the step of obtaining the start frame and the end frame of the target video when the user updates the item display, can be implemented by steps 4011 and 4012:

[0170] In step 4011, the video frames in the target video when the user updates the item display are identified in sequence, and a video frame in which the number of items does not change and the shelves are not blocked is obtained as the starting frame.

[0171] In step 4012, a video frame in the target video when the user updates the item display and whose similarity with the start frame after the start frame meets a preset condition is obtained as the end frame.

[0172] The preset condition may be less than or equal to the first preset similarity threshold, and greater than or equal to the second preset similarity threshold. If the similarity between a certain video frame and the starting frame is greater than the first preset similarity threshold, it can be explained that the video frame is still a video frame before the user makes a purchase, and it can be excluded; if the similarity between a certain video frame and the starting frame is less than the second preset similarity threshold, it can be explained that the user's body may block the display of the cabinet in the video frame, that is, the video frame is a video frame in which the user is shopping, and it can also be excluded.

[0173] For example, the target detection model can be used to identify each frame of the target video, starting from the first video frame, until it is recognized that the number of objects in the image does not change and the lateral position of the user's limbs does not overlap with the partition, then the starting frame is confirmed. Then, starting from the starting frame, video frames with a similarity less than or equal to the first preset similarity threshold and greater than or equal to the second preset similarity threshold are obtained as the ending frame.

[0174] Optionally, in order to improve efficiency, after obtaining the start frame, the target detection model can be used to continue recognition, and after all the video frames blocked by the shelf when the user takes the item are recognized, the video frames of this part and the video frames before this part except the start frame are all excluded, and then the video frame with the highest similarity to the start frame is obtained from the remaining multiple video frames as the end frame. Since there is no video frame before shopping in the remaining multiple video frames, its similarity with the start frame must be less than the first preset similarity threshold. At this time, it is only necessary to obtain the video frame with the highest similarity to the start frame from the remaining multiple video frames as the end frame.

[0175] Specifically, the perceptual hash algorithm can be used to sequentially calculate the fingerprint data of the remaining multiple video frames and the starting frame, and then the Hamming distances between the remaining multiple video frames and the starting frame can be sequentially calculated based on the fingerprint data of the multiple video frames and the starting frame, and the video frame with the smallest Hamming distance can be used as the ending frame. Specifically, the perceptual hash algorithm can be used to find the two most similar video frames, that is, the two video frames with the closest door opening angles, so that the interference when obtaining the video frames is minimized.

[0176] In one embodiment, Figure 8 As shown, the step 403, i.e., the step of determining in sequence whether the target objects at the same position in the start frame and the end frame match each other according to the coordinate information of the target objects in the start frame and the end frame, can be implemented by steps 4031 to 4035:

[0177] In step 4031, the region of interest of each target object in the start frame and the end frame is determined according to the coordinate information of the target object in the start frame and the end frame.

[0178] In step 4032, feature points of each target object are identified in the region of interest of each target object in the start frame and the end frame.

[0179] In step 4033, the matching percentage of the feature points of the target object at the same position in the starting frame and the ending frame is obtained.

[0180] In step 4034, if the matching percentage of the reference position is greater than or equal to a preset percentage threshold, it is confirmed that the target object at the reference position in the start frame and the end frame matches.

[0181] In step 4035, if the matching percentage of the reference position is less than the preset percentage threshold, it is determined that the target object at the reference position in the start frame and the end frame does not match.

[0182] For example, the region of interest of the target object may be an image region within a bounding box of the target object, or other image regions including the image region within the bounding box. Specifically, taking any target object as an example, after the coordinate information of the target object in the start frame and the end frame is identified by the target detection model, the bounding box of the target object in the start frame and the end frame may be determined according to the coordinate information of the target object in the start frame and the end frame, and then the image regions included in the bounding boxes of the target object in the start frame and the end frame are respectively used as the region of interest of the target object in the start frame and the end frame.

[0183] After obtaining the region of interest of each target object in the start frame and the end frame, the SIFT operator can be used to calculate the region of interest of each target object in the start frame and the end frame to generate the feature points of each target object. Then, the matching percentage of the key points of the target object at the same position in the start frame and the end frame is determined in turn. The matching percentage is the ratio of the number of successfully matched feature points at the same position in the start frame and the end frame to the total number of feature points at the position in the end frame or the start frame; or the matching percentage is the ratio of the number of successfully matched feature points at the same position in the start frame and the end frame to the total number of feature points identified in the end frame or the start frame. Specifically, the Sift algorithm is used to maintain invariance to scale scaling and brightness changes, and has a certain degree of stability, uniqueness, multiplicity and scalability to changes in viewing angles and noise, ensuring that the key points will not change due to factors such as illumination and noise in actual applications.

[0184] If the matching percentage of the key points of the target object at the reference position is greater than or equal to the preset percentage threshold, it is confirmed that the target object at the reference position has not changed, that is, the object in the target video is not the object at the reference position. If the matching percentage of the key points of the target object at the reference position is less than the preset percentage threshold, it is confirmed that the target object at the reference position has changed, that is, the object in the target video is the object at the reference position.

[0185] In actual applications, due to possible errors in image recognition, there may be multiple target objects at the same position in the start frame and the end frame that do not match, making it difficult to determine the reference position of the actual object update. In this case, the coordinate information of the user's limbs can be used to assist in determining the reference position of the actual object update.

[0186] Optionally, the coordinate information of the user's limbs can be first obtained according to a preset target detection model. If there are multiple target objects in the same position in the starting frame and the ending frame that do not match, the reference position where the target object changes before and after the display of the items is updated can be determined in the multiple same positions based on the coordinate information of the user's limbs. Specifically, the target detection model can be used to identify the coordinate information of the user's limbs in each video frame in the target video, such as the position of the shelf of the shelf on which the user's hand is located, and the aisle position of the shelf on which the hand is located, and then each video frame is compared with the starting frame to record the position of the items blocked by the user's limbs on each layer. Since the coordinate information of the turning point of the user's limb in its motion trajectory is the same or close to the coordinate information of the reference position where the object is actually updated, that is, the position of the object blocked by the user's limb at the turning point is most likely the position where the user updates the object display, the movement trajectory of the user's limb can be determined through the coordinate information of the user's limb in each video frame, and then the coordinate information of the turning point is compared with the coordinate information of multiple identical positions, and the position among the multiple identical positions that is the same or close to the coordinate information of the turning point is used as the reference position where the object is actually updated.

[0187] The object identification method provided in the present application can dynamically obtain the display status of objects in the container through the collected target video of each user taking objects, and realize real-time monitoring of the display of objects, so as to facilitate operation and maintenance personnel to confirm the quantity and / or type of objects taken by users, and avoid malicious users taking objects multiple times.

[0188] At the same time, since the number and type of items taken by each user can be obtained, real-time monitoring of item inventory can be achieved, so that operation and maintenance personnel can replenish inventory in time when they confirm that the inventory is small. In addition, operation and maintenance personnel can also count the types of items most popular with users or customers, which is convenient for adjusting the planning of item display later.

[0189] Taking unmanned vending machines as an example, the target video of each user shopping collected by the unmanned vending machine can dynamically obtain the display of items in the cabinet, and realize real-time monitoring of the display of items, so that operation and maintenance personnel can confirm the quantity and type of items taken by users, and avoid malicious users taking items multiple times.

[0190] At the same time, since the number and type of items taken by the user during each sale can be obtained, real-time monitoring of the inventory of items can be achieved, so that the operation and maintenance personnel can replenish the inventory in time when it is confirmed that the inventory is small, avoiding the situation where the vending machine has no goods to sell, thereby improving the practicality of the vending machine and providing a better user experience.

[0191] In addition, combined with the above method, the sales data of a certain item can be obtained by dynamically monitoring the sales of items, and then the taste preferences of consumers in the area can be inferred, which can in turn inform the factory of the key production types and quantities in the area, saving production capacity and avoiding waste.

[0192] In one embodiment, the target detection model may be obtained through pre-training.

[0193] Before step 401, the object identification method may further include the following steps:

[0194] Receive the updated weight parameters sent by the edge server, and update the private detection model according to the updated weight parameters;

[0195] The updated private detection model is trained by taking the historical image information of the shelf as input and the coordinate information of the target object in the historical image information as output;

[0196] When the trained private detection model does not converge, a gradient update vector is obtained according to the trained private detection model, and the gradient update vector is sent to the edge server;

[0197] When the trained private detection model converges, the target detection model is obtained according to the private detection model.

[0198] The historical image information of the shelf and the coordinate information of the target object in the historical image information can be realized by reading the historical image information and the coordinate information corresponding to the historical image information pre-stored in the item display device. The historical image information and the coordinate information corresponding to the historical image information can be collected by the item display device itself, or can be collected by other devices or systems for the item display device and transmitted to the item display device.

[0199] The private detection model may be a neural network (NN) model, a convolutional neural network (CNN) model, or a long short-term memory network (LSTM) model, etc.

[0200] The edge server is used to aggregate the gradient update vectors, and update the weight parameters of the shared detection model of the edge server according to the aggregated gradient update vectors to obtain updated weight parameters. The edge server can be a cloud server or a server provided by an item identification service provider. It should be noted that an edge server can correspond to one or more item identification devices. For example, the item identification service provider can divide the area under its jurisdiction into multiple blocks, and multiple item identification devices in each area correspond to one edge server.

[0201] The shared detection model may be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.

[0202] In the technical solution of this embodiment, the updated weight parameters sent by the edge server and received by the item display device are obtained by aggregating the gradient update vectors sent by multiple item display devices, and updating the weight parameters of the shared detection model of the edge server according to the aggregated gradient update vectors. Therefore, the updated private detection model on the item display device can reflect the shared rules between the historical image information of the shelf and the coordinate information of the target object in the historical image information learned by the shared detection model of the edge server in the previous round of training. By taking the historical image information of the shelf as input and the coordinate information of the target object in the historical image information as output, the updated private detection model is trained. This allows the updated private detection model to learn the common rules and also to learn the data collected by the item display device itself in a personalized manner, so that the trained private detection model can learn the private rules between the image information of the shelf of the item display device itself and the coordinate information of the target object in the image information. When the trained private detection model does not converge, it means that the trained private detection model still needs to be trained. By obtaining the gradient update vector according to the trained private detection model and sending the gradient update vector, the edge server can continue to obtain the corresponding update weight parameter based on the gradient update vectors uploaded by multiple item display devices, thereby continuing to train the private detection model of each item display device. When the trained private detection model converges, it can be considered that the converged private detection model can obtain relatively accurate coordinate information of the target object in the image information based on the image information of the shelf, and the converged private detection model can be stored as the preset target detection model. In this technical solution, on the one hand, the target detection model can be a model that learns both common rules and private rules, and the coordinate information of the target object obtained based on it is more accurate; on the other hand, since the process of continuing to train the private detection model of each object display device is executed by the object display device, compared with only the server training the detection model, less processing resources are required on the server side and the training speed is faster.

[0203] In one embodiment, before receiving the updated weight parameter sent by the edge server, the object identification method may further include the following steps:

[0204] Receive a private data upload instruction; in response to the private data upload instruction, send historical image information of the shelf within the target time interval and coordinate information of the target object in the historical image information within the target time interval; receive initial weight parameters sent by the edge server; update the initial detection model according to the initial weight parameters to obtain a private detection model.

[0205] The target time interval may be pre-set, for example, 6:00 pm to 6:30 pm on the day before the private data upload instruction is received may be pre-set as the target time interval. The target time interval may also be determined according to a target time interval setting instruction sent by other devices or systems.

[0206] The initial detection model may be a neural network model, a convolutional neural network model, a long short-term memory network model, etc. The initial detection model may be understood as an untrained detection model.

[0207] In this embodiment, by receiving a private data upload instruction and responding to the private data upload instruction, sending the historical image information of the shelf within the target time interval and the coordinate information of the target object in the historical image information within the target time interval, the edge server can perform preliminary training on its initial detection model based on the less information within the target time interval (i.e., the historical image information of the shelf and the coordinate information of the target object in the historical image information) to obtain a shared detection model. At this time, the shared detection model can be understood as a model that has been preliminarily learned about the common laws of the image information in multiple object display devices and the coordinate information of the target object in the image information. Afterwards, the edge server sends the initial weight parameter obtained based on the shared detection model to the object array device, so that the object display device updates the initial detection model according to the initial weight parameter to obtain a private detection model. At this time, the private detection model can be understood as a model that has learned the same law as the shared detection model on the edge server, that is, at this time, the private detection model is a model that has been preliminarily learned about the common laws of the image information in multiple object display devices and the coordinate information of the target object in the image information, thereby facilitating the subsequent multiple rounds of training of the private detection model without the need to train based on the initial detection model, thereby reducing the difficulty of training.

[0208] Fig. 9 The structural block diagram of the article display device according to the embodiment of the present disclosure is shown. The device can be implemented as part or all of the electronic device through software, hardware or a combination of both.

[0209] like Fig. 9 As shown, the article display device 900 includes a video acquisition module 901 , a coordinate acquisition module 902 , a determination module 903 and a confirmation module 904 .

[0210] Among them, the video acquisition module 901 is configured to obtain the start frame and end frame of the target video when the user updates the item display, the start frame includes the display information of the shelf when the user has not started to update the item display, and the end frame includes the display information of the shelf when the user has completed updating the item display.

[0211] The coordinate acquisition module 902 is configured to respectively acquire the coordinate information of the target object in the start frame and the end frame according to a preset target detection model; the target object at least includes items displayed on the shelf.

[0212] The determination module 903 is configured to determine in sequence whether the target objects at the same position in the start frame and the end frame match according to the coordinate information of the target objects in the start frame and the end frame.

[0213] The confirmation module 904 is configured to confirm that the target object at the reference position changes before and after the user updates the item display if there is a mismatch between the target object at the reference position in the start frame and the end frame.

[0214] Optionally, the coordinate acquisition module 902 is also configured to respectively acquire multiple bounding boxes of each target object in the starting frame and the ending frame according to a preset target detection model, and different bounding boxes have different confidence levels; perform non-maximum suppression processing on the multiple bounding boxes of each target object to acquire at least one remaining bounding box; determine, among the at least one remaining bounding box, a bounding box with the largest confidence level and exceeding a predetermined boundary threshold as the bounding box of the corresponding target object; and acquire the coordinate information of each target object based on the bounding box of each target object.

[0215] Optionally, the coordinate acquisition module 902 is also configured to respectively acquire the coordinate information of multiple objects and multiple shelves in the starting frame and the ending frame according to a preset target detection model; acquire the distance between two adjacent objects on the same shelf according to the coordinate information of the multiple objects and the multiple shelves; if the distance between two adjacent objects on the same shelf is greater than or equal to a preset distance threshold, it is determined that there is an empty aisle between the two adjacent objects.

[0216] Optionally, the video acquisition module 901 is also configured to identify the video frames in the target video in sequence, and obtain a video frame in which the number of objects no longer changes and the partitions are not blocked as the starting frame; and obtain a video frame in the target video whose similarity with the starting frame after the starting frame meets a preset condition as the ending frame.

[0217] Optionally, the video acquisition module 901 is further configured to remove video frames in the target video that are blocked after the start frame; and acquire the video frame with the highest similarity to the start frame from the remaining multiple video frames as the end frame.

[0218] Optionally, the video acquisition module 901 is further configured to acquire, from the remaining multiple video frames, a video frame having the smallest Hamming distance with the start frame as the end frame.

[0219] Optionally, the determination module 903 is configured to determine the area of ​​interest of each target object in the starting frame and the ending frame based on the coordinate information of the target object in the starting frame and the ending frame; identify the feature points of each target object in the area of ​​interest of each target object in the starting frame and the ending frame; obtain the matching percentage of the feature points of the target objects at the same position in the starting frame and the ending frame; if the matching percentage of the reference position is greater than or equal to a preset percentage threshold, confirm that the target object at the reference position in the starting frame and the ending frame matches; if the matching percentage of the reference position is less than the preset percentage threshold, confirm that the target object at the reference position in the starting frame and the ending frame does not match.

[0220] Optional, such as Fig.10 As shown, the device 900 also includes an image acquisition module 905 , an image annotation module 906 , an image enhancement module 907 and a model training module 908 .

[0221] The image acquisition module 905 is configured to acquire a plurality of training images including shelf display information, and the shelf display information included in the plurality of training images is not completely the same.

[0222] The image annotation module 906 is configured to respectively annotate the bounding box of the target object in each training image, and obtain a plurality of annotated training images.

[0223] The image enhancement module 907 is configured to perform image enhancement on the labeled multiple training images to obtain multiple enhanced training images.

[0224] The model training module 908 is configured to train the model through the target detection framework according to the enhanced multiple training images to obtain the target detection model.

[0225] Optionally, the coordinate acquisition module 902 is also configured to acquire coordinate information of the user's limbs according to a preset target detection model; the determination module 903 is also configured to determine the reference position where the target object changes before and after updating the item display according to the coordinate information of the user's limbs in the multiple same positions if there are multiple target objects in the same position that do not match in the starting frame and the ending frame.

[0226] The item display device provided in the present application can dynamically obtain the item display status of the container through the collected target video of each user taking items, and realize real-time monitoring of the item display, so as to facilitate operation and maintenance personnel to confirm the quantity and / or type of items taken by users, and avoid malicious users taking items multiple times.

[0227] At the same time, since the number and type of items taken by each user can be obtained, real-time monitoring of item inventory can be achieved, so that operation and maintenance personnel can replenish inventory in time when they confirm that the inventory is small. In addition, operation and maintenance personnel can also count the types of items most popular with users or customers, which is convenient for adjusting the planning of item display later.

[0228] The present disclosure also discloses an electronic device, Fig.11 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0229] like Fig.11 As shown, the electronic device 1100 includes a memory 1101 and a processor 1102, wherein the memory 1101 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 1102 to implement the method according to an embodiment of the present disclosure.

[0230] Specifically, the object identification method includes:

[0231] Acquire a start frame and an end frame of a target video when a user updates an item display, wherein the start frame includes display information of the shelf when the user has not started to update the item display, and the end frame includes display information of the shelf when the user has completed updating the item display;

[0232] Obtaining coordinate information of the target object in the start frame and the end frame respectively according to a preset target detection model; the target object at least includes items displayed on the shelf;

[0233] Determining in sequence whether target objects at the same position in the starting frame and the ending frame match according to coordinate information of the target objects in the starting frame and the ending frame;

[0234] If there is a mismatch between the target object at the reference position in the start frame and the end frame, it is confirmed that the target object at the reference position changes before and after the user updates the item display.

[0235] Optionally, the acquiring the coordinate information of the target object in the start frame and the end frame respectively according to a preset target detection model includes:

[0236] Acquire multiple bounding boxes of each target object in the start frame and the end frame respectively according to a preset target detection model, where different bounding boxes have different confidence levels;

[0237] Perform non-maximum suppression processing on multiple bounding boxes of each target object to obtain at least one remaining bounding box;

[0238] Determine, among the remaining at least one bounding box, a bounding box having the largest confidence and exceeding a predetermined bounding threshold as a bounding box corresponding to the target object;

[0239] According to the bounding box of each target object, coordinate information of each target object is obtained.

[0240] Optionally, the acquiring the coordinate information of the target object in the start frame and the end frame respectively according to a preset target detection model includes:

[0241] Obtaining coordinate information of a plurality of objects and a plurality of shelves in the start frame and the end frame respectively according to a preset target detection model;

[0242] According to the coordinate information of the plurality of objects and the plurality of shelves, obtaining the distance between two adjacent objects on the same shelf;

[0243] If the distance between two adjacent items on the same shelf is greater than or equal to a preset distance threshold, it is determined that there is an empty aisle between the two adjacent items.

[0244] Optionally, obtaining the start frame and the end frame of the target video when the user updates the item display includes:

[0245] The video frames in the target video are identified sequentially, and a video frame in which the number of objects does not change and the shelf is not blocked is obtained as the starting frame;

[0246] A video frame in the target video whose similarity with the start frame meets a preset condition after the start frame is obtained as the end frame.

[0247] Optionally, obtaining a video frame in the target video that is after the start frame and whose similarity with the start frame meets a preset condition as the end frame includes:

[0248] Eliminate the video frames in the target video that are blocked by the frames after the starting frame;

[0249] The video frame with the highest similarity to the start frame is obtained from the remaining multiple video frames as the end frame.

[0250] Optionally, obtaining, from the remaining multiple video frames, a video frame having the highest similarity to the start frame as the end frame includes:

[0251] A video frame having the smallest Hamming distance with the start frame is obtained from the remaining multiple video frames as the end frame.

[0252] Optionally, determining in sequence whether target objects at the same position in the start frame and the end frame match according to coordinate information of the target objects in the start frame and the end frame includes:

[0253] Determine the region of interest of each of the target objects in the start frame and the end frame according to the coordinate information of the target objects in the start frame and the end frame;

[0254] Identify feature points of each target object in the region of interest of each target object in the start frame and the end frame;

[0255] Obtaining the matching percentage of feature points of the target object at the same position in the starting frame and the ending frame;

[0256] If the matching percentage of the reference position is greater than or equal to a preset percentage threshold, confirming that the target object at the reference position in the starting frame and the ending frame matches;

[0257] If the matching percentage of the reference position is less than the preset percentage threshold, it is determined that the target object at the reference position in the start frame and the end frame does not match.

[0258] Optionally, the method further includes:

[0259] Acquire a plurality of training images including shelf display information, wherein the shelf display information included in the plurality of training images is not completely the same;

[0260] Label the bounding box of the target object in each training image respectively, and obtain multiple labeled training images;

[0261] Performing image enhancement on the labeled multiple training images to obtain multiple enhanced training images;

[0262] The target detection model is obtained by training a model through a target detection framework according to the enhanced multiple training images.

[0263] Optionally, the method further includes:

[0264] Obtaining coordinate information of the user's limbs according to a preset target detection model;

[0265] If there are multiple target objects at the same position in the start frame and the end frame that do not match, the reference position where the target object changes before and after updating the display of items is determined from the multiple same positions according to the coordinate information of the user's limb.

[0266] Fig.12 A schematic diagram showing the structure of a computer system suitable for implementing the method according to an embodiment of the present disclosure is shown.

[0267] like Fig.12 As shown, the computer system 1200 includes a processing unit 1201, which can execute various methods in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage part 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the system 1200 are also stored. The processing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0268] The following components are connected to the I / O interface 1205: an input part 1206 including a keyboard, a mouse, etc.; an output part 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 1208 including a hard disk, etc.; and a communication part 1209 including a network interface card such as a LAN card, a modem, etc. The communication part 1209 performs a communication process via a network such as the Internet. The drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed, so that the computer program read therefrom is installed into the storage part 1208 as needed. Among them, the processing unit 1201 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0269] In particular, according to an embodiment of the present disclosure, the method described above can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the above method. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1209, and / or installed from the removable medium 1211.

[0270] In addition, the present disclosure also provides a computer program product, in which a computer program is stored. When the computer program is executed by a processor, the processor can at least implement the method provided in the aforementioned embodiment.

[0271] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0272] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or programmable hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0273] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or computer system in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.

[0274] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A method for object identification, characterized in that: include: Acquire a start frame and an end frame of a target video when a user updates an item display, wherein the start frame includes display information of the shelf when the user has not started to update the item display, and the end frame includes display information of the shelf when the user has completed updating the item display; Obtaining coordinate information of the target object in the start frame and the end frame respectively according to a preset target detection model; The target objects at least include items displayed on the shelves; Determining in sequence whether target objects at the same position in the starting frame and the ending frame match according to coordinate information of the target objects in the starting frame and the ending frame; If there is a mismatch between the target object at the reference position in the start frame and the end frame, it is confirmed that the target object at the reference position changes before and after the user updates the item display; The step of respectively acquiring the coordinate information of the target object in the start frame and the end frame according to the preset target detection model includes: Acquire multiple bounding boxes of each target object in the start frame and the end frame respectively according to a preset target detection model, where different bounding boxes have different confidence levels; Perform non-maximum suppression processing on multiple bounding boxes of each target object to obtain at least one remaining bounding box; Determine, among the remaining at least one bounding box, a bounding box having the largest confidence and exceeding a predetermined bounding threshold as a bounding box corresponding to the target object; According to the bounding box of each target object, coordinate information of each target object is obtained.

2. The method according to claim 1, characterized in that The step of respectively acquiring the coordinate information of the target object in the start frame and the end frame according to the preset target detection model includes: Obtaining coordinate information of a plurality of objects and a plurality of shelves in the start frame and the end frame respectively according to a preset target detection model; According to the coordinate information of the plurality of objects and the plurality of shelves, obtaining the distance between two adjacent objects on the same shelf; If the distance between two adjacent items on the same shelf is greater than or equal to a preset distance threshold, it is determined that there is an empty aisle between the two adjacent items.

3. The method according to any one of claims 1 to 2, characterized in that: The step of obtaining the start frame and the end frame of the target video when the user updates the item display includes: The video frames in the target video are identified sequentially, and a video frame in which the number of objects does not change and the shelf is not blocked is obtained as the starting frame; A video frame in the target video whose similarity with the start frame meets a preset condition after the start frame is obtained as the end frame.

4. The method according to claim 3, characterized in that The step of obtaining a video frame in the target video whose similarity with the start frame after the start frame satisfies a preset condition as the end frame comprises: Eliminate the video frames in the target video that are blocked by the frames after the starting frame; The video frame with the highest similarity to the start frame is obtained from the remaining multiple video frames as the end frame.

5. The method according to claim 4, characterized in that The step of obtaining the video frame having the highest similarity to the start frame from the remaining multiple video frames as the end frame comprises: A video frame having the smallest Hamming distance with the start frame is obtained from the remaining multiple video frames as the end frame.

6. The method according to any one of claims 1 to 2, characterized in that: The step of determining, in sequence according to the coordinate information of the target objects in the start frame and the end frame, whether the target objects at the same position in the start frame and the end frame match each other comprises: Determine the region of interest of each of the target objects in the start frame and the end frame according to the coordinate information of the target objects in the start frame and the end frame; Identify feature points of each target object in the region of interest of each target object in the start frame and the end frame; Obtaining the matching percentage of feature points of the target object at the same position in the starting frame and the ending frame; If the matching percentage of the reference position is greater than or equal to a preset percentage threshold, confirming that the target object at the reference position in the starting frame and the ending frame matches; If the matching percentage of the reference position is less than the preset percentage threshold, it is determined that the target object at the reference position in the start frame and the end frame does not match.

7. The method according to any one of claims 1 to 2, characterized in that: The method further comprises: Acquire a plurality of training images including shelf display information, wherein the shelf display information included in the plurality of training images is not completely the same; Label the bounding box of the target object in each training image respectively, and obtain multiple labeled training images; Performing image enhancement on the labeled multiple training images to obtain multiple enhanced training images; The target detection model is obtained by training a model through a target detection framework according to the enhanced multiple training images.

8. The method according to claim 6, characterized in that The method further comprises: Obtaining coordinate information of the user's limbs according to a preset target detection model; If there are multiple target objects at the same position in the start frame and the end frame that do not match, the reference position where the target object changes before and after updating the display of items is determined from the multiple same positions according to the coordinate information of the user's limb.

9. An electronic device, characterized in that: It comprises a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps described in any one of claims 1 to 8.

10. A readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method steps described in any one of claims 1 to 8 are implemented.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images