Display cabinet control method, apparatus, medium, and display cabinet

Abstract

The display cabinet control method, device, medium and display cabinet disclosed by the embodiments of the present disclosure comprise: acquiring a cabinet door image collected by a cabinet door image collection device and multiple cabinet body images collected by a cabinet body image collection device; acquiring a taking area image corresponding to a hand in the cabinet door image, and performing object identification on the taking area image to obtain a first object identification result; acquiring a target movement track of a target object, and intercepting a target object area image in a third target cabinet body image according to the target movement track; performing object identification according to the target object area image to obtain a second object identification result; and matching the first object identification result and the second object identification result to obtain a matched object identification result. This technical solution can accurately determine the object removed from or moved into the display cabinet according to the matched object identification result, thereby improving the reliability of identifying the object removed from or moved into the display cabinet.

Description

Technical Field

[0001] This disclosure relates to the field of control technology, specifically to a display cabinet control method, equipment, medium, and display cabinet. Background Technology

[0002] In recent years, to facilitate user access to product information, merchants and businesses have often placed items in display cases, simultaneously storing and showcasing goods. When users need to remove or add items, they can open the display case themselves. In this scenario, the display case can capture images of the items inside and upload the data. Other devices or systems, such as servers or cloud platforms, can then use these images to identify users and items moving in or out of the display case, enabling statistical analysis. Summary of the Invention

[0003] This disclosure provides a display cabinet control method, device, medium, and display cabinet to solve the problem in related technologies where it is impossible to accurately determine the items moved out or into the display cabinet based on image recognition results.

[0004] Firstly, this disclosure provides a display cabinet control method.

[0005] Specifically, display case control methods include:

[0006] The system acquires cabinet door images captured by a cabinet door image acquisition device and multiple cabinet body images captured by a cabinet body image acquisition device. The cabinet door image acquisition device is located at the cabinet door of the display cabinet, and the cabinet body image acquisition device is located at the cabinet body of the display cabinet.

[0007] Obtain the image of the area corresponding to the hand in the cabinet door image, and perform item recognition on the image of the area to obtain the first item recognition result;

[0008] The target movement trajectory of the target item is obtained, and the target item area image is cropped in the third target cabinet image according to the target movement trajectory. The target item is an item whose position in the first target cabinet image is different from its position in the second target cabinet image. The first target cabinet image, the second target cabinet image, and the third target cabinet image are all multiple cabinet images. The acquisition time of the third target cabinet image is between the acquisition time of the first target cabinet image and the acquisition time of the second target cabinet image.

[0009] Item recognition is performed based on the target item region image to obtain a second item recognition result;

[0010] The matching results are obtained by matching the first item recognition result and the second item recognition result.

[0011] In a second aspect, embodiments of this disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method as described in the first aspect or any implementation thereof.

[0012] Thirdly, this disclosure provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the method as described in the first aspect or any implementation thereof.

[0013] Fourthly, this disclosure provides a computer program product including computer instructions that, when executed by a processor, implement the method as described in the first aspect or any implementation thereof.

[0014] Fifthly, this disclosure provides a display cabinet, which includes a cabinet body, cabinet doors, a first cabinet body image acquisition device, a second cabinet body image acquisition device, a cabinet door image acquisition device, and a processing device.

[0015] The cabinet door is rotatably connected to the cabinet body and is used to open or close the cabinet's entrance / exit for items.

[0016] The cabinet includes an internal cavity, which is connected to the outside of the cabinet through an item inlet / outlet. The internal cavity is used to store items.

[0017] Both the first cabinet image acquisition device and the second cabinet image acquisition device are connected to the top surface of the cabinet's internal cavity. The first cabinet image acquisition device and the second cabinet image acquisition device are used to acquire images of the item's entrance and exit from different directions.

[0018] The cabinet door image acquisition device is connected to the side of the cabinet door closest to the cabinet body, and the position of the cabinet door image acquisition device matches the position of the cabinet door handle, while the door handle is connected to the side of the cabinet door furthest from the cabinet body.

[0019] The processing device is communicatively connected to the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device. The processing device is used to execute the method described in the first aspect or any implementation thereof.

[0020] The technical solutions provided in this disclosure may have the following beneficial effects:

[0021] The above technical solution acquires cabinet door images captured by a cabinet door image acquisition device and multiple cabinet body images captured by a cabinet body image acquisition device. Since the cabinet door image acquisition device is located at the cabinet door of the display case, and the cabinet body image acquisition device is located at the cabinet body, the cabinet door images and cabinet body images are acquired from at least two significantly different locations. The solution also acquires an image of the area corresponding to the hand in the cabinet door image, and performs item recognition on the area image to obtain a first item recognition result. The second item recognition result is based on the image acquired from the location of the cabinet door, indicating that the item is being moved by the hand (i.e., being moved into the display case). The identification results of target items (either displayed in a cabinet or removed from a display case); obtaining the target movement trajectory of the target item; and extracting the target item area image from the third target cabinet image based on the target movement trajectory. The target item is defined as an item whose position in the first target cabinet image differs from its position in the second target cabinet image. Both the first and second target cabinet images are multiple cabinet images, and the third target cabinet image is also a multiple cabinet image, acquired between the acquisition times of the first and second target cabinet images. Therefore, the target item is extracted from the third target cabinet image. The target item in the region image is in a moving state (i.e., being moved into or out of the display case); item recognition is performed based on the target item region image to obtain a second item recognition result, which is the recognition result of the target item being moved into or out of the display case based on the image acquired from the location of the display case; matching is performed based on the first item recognition result and the second item recognition result to obtain a matched item recognition result, wherein the images on which the first item recognition result and the second item recognition result are based during item recognition are acquired from at least two significantly different locations, i.e. The image acquired from one location may contain an occluded object. Since the difference between at least two different locations is significant, the probability of the target object being occluded in images acquired from other locations within those two different locations is relatively small. Therefore, the accuracy of at least one of the first and second item recognition results will not be affected by the occlusion of the target object (i.e., the object being moved out or into the display case) in the acquired image. This ensures that the obtained matching item recognition result can accurately determine the object being moved out or into the display case, thus improving the reliability of identifying the object being moved out or into the display case.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

[0024] Figure 1 A schematic structural block diagram of a display cabinet according to an embodiment of the present disclosure is shown.

[0025] Figure 2 A schematic structural block diagram of a motherboard according to an embodiment of the present disclosure is shown.

[0026] Figure 3 A schematic structural block diagram of a control panel according to an embodiment of the present disclosure is shown.

[0027] Figure 4 A schematic structural block diagram of a power management module according to an embodiment of the present disclosure is shown.

[0028] Figure 5 A flowchart illustrating a display case control method according to an embodiment of the present disclosure is shown.

[0029] Figure 6 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown.

[0030] Figure 7 A schematic top view of a display case according to one embodiment of the present disclosure is shown.

[0031] Figure 8 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0032] Figure 9 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0033] Figure 10 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0034] Figure 11 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0035] Figure 12 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0036] Figure 13 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0037] Figure 14 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0038] Figure 15 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0039] Figure 16 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0040] Figure 17 A flowchart illustrating the overall process of a display case control method according to an embodiment of the present disclosure is shown.

[0041] Figure 18 A schematic structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0042] Figure 19 This is a schematic diagram of the structure of a computer system suitable for implementing a display case control method according to an embodiment of the present disclosure.

[0043] Figure 20 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown.

[0044] Figure 21 A schematic top view of a display case according to one embodiment of the present disclosure is shown. Detailed Implementation

[0045] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement them. Furthermore, for clarity, portions unrelated to the description of the exemplary embodiments have been omitted from the drawings.

[0046] In this disclosure, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, figures, steps, behaviors, components, parts or combinations thereof disclosed in this specification, and do not preclude the possibility of the presence or addition of one or more other features, figures, steps, behaviors, components, parts or combinations thereof.

[0047] It should also be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] As mentioned above, with the development of technology and the improvement of people's living standards, merchants or enterprises no longer simply place goods on shelves. Instead, to facilitate users' understanding of product information, they can place items in display cases, thus simultaneously storing and displaying the goods. When users need to remove items from the display case or put items into it, they can open the display case themselves and perform the corresponding operations.

[0049] In recent years, the number of display cases put into operation has gradually increased. During the use of display cases, merchants or enterprises generally need to monitor the items in the display cases to identify the users who move items out of or into the display cases, as well as the items that are moved out of or into the display cases. Based on the above information, statistics can be compiled to determine the quantity and type of remaining items in the display cases. In addition, further settlement can be carried out based on the statistical results.

[0050] In one embodiment, after the display case is unlocked, at least one image can be captured by at least one camera mounted on the display case body, and image recognition can be performed on the image to determine the item that was moved out or into the display case based on the image recognition result. However, in this solution, since the camera is only mounted on the display case body, the camera's acquisition direction is relatively singular, resulting in a high possibility of obstruction between the camera and the item being moved out or into the display case. Therefore, the image recognition result may not be able to accurately determine the item moved out or into the display case, thus reducing the reliability of the image recognition result.

[0051] Therefore, how to ensure that the images can accurately identify the items being moved out or into the display case is an increasingly urgent problem that needs to be solved.

[0052] In view of the above-mentioned shortcomings, a display cabinet control method is proposed in one embodiment of this disclosure.

[0053] The display cabinet control method provided in this application embodiment can be applied to display cabinets, which can have a temperature control function. The temperature control function can be a cooling function, such as a refrigerated display cabinet, a frozen display cabinet, a refrigerator, a wine cabinet, a cosmetic preservation cabinet, etc.; the temperature control function can also be a heating function, such as a warming cabinet, a heated display cabinet, a hot beverage cabinet, etc. This application embodiment does not limit the specific type of display cabinet.

[0054] For example, Figure 1 A schematic structural block diagram of a display cabinet according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the display case 100 may include a compressor 11, a condenser 12, a throttling element 13, and an evaporator 14. The compressor 11, condenser 12, throttling element 13, and evaporator 14 are connected by pipes filled with refrigerant to form a closed pipeline, which constitutes a refrigeration system or heating system capable of circulating refrigerant.

[0055] In one embodiment of this application, the display cabinet includes a cabinet body and a cabinet door, wherein a control board and a power management module may be installed in the cabinet body, and a main board may be installed in the cabinet door.

[0056] In one embodiment of this application, Figure 2 A schematic structural block diagram of a motherboard according to an embodiment of the present disclosure is shown, such as... Figure 2 As shown, the motherboard 200 includes a processor 201, random access memory 202, flash memory 203, wireless LAN Bluetooth module 204, gyroscope 205, pressure sensor 206, microphone 207, speaker 208, camera 209, and cellular communication module 210.

[0057] In one embodiment of this application, Figure 3 A schematic structural block diagram of a control panel according to an embodiment of the present disclosure is shown, such as... Figure 3 As shown, the control board 300 includes a power input interface 301, a power output interface 302, a metering chip 303, a microcontroller chip 304, a real-time clock chip, a light switch interface 305, a temperature control switch interface 306, an evaporator fan interface 307, a compressor interface 308, a condenser fan interface 309, a temperature sensor interface 310, a communication interface 311, and a power interface 312.

[0058] In one embodiment of this application, Figure 4 A schematic structural block diagram of a power management module according to an embodiment of the present disclosure is shown, such as... Figure 4 As shown, the power management module 400 includes an AC-to-DC conversion module 401, a charging management module 402, and a battery 403. The power management module 400 supplies power to the motherboard and control board and manages the charging and discharging of the battery. The power management module 400 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 400 may also be located within the processor.

[0059] In one embodiment of this application, the display case further includes a display screen. The display case implements its display function through a graphics processor, a display screen, and an application processor. The graphics processor is a microprocessor for image processing, connected to the display screen and the application processor. The graphics processor is used to perform mathematical and geometric calculations and for graphics rendering. The processor may include one or more graphics processors that execute program instructions to generate or modify display information.

[0060] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the display cabinet. In other embodiments of this application, the display cabinet may include more or fewer components than illustrated, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, by combining different components, the display cabinet in the embodiments of this application may be any of the following: a retail cabinet, a heated cabinet, a refrigerated cabinet, a freezer, a combination cabinet, or a display case.

[0061] Figure 5 A flowchart illustrating a display case control method according to an embodiment of the present disclosure is shown, such as... Figure 5 As shown, the display case control method includes the following steps S101-S104:

[0062] In step S101, the cabinet door image acquired by the cabinet door image acquisition device and multiple cabinet body images acquired by the cabinet body image acquisition device are obtained.

[0063] The cabinet door image acquisition device is installed at the cabinet door of the display cabinet, and the cabinet body image acquisition device is installed at the cabinet body of the display cabinet.

[0064] In one embodiment of this disclosure, acquiring cabinet door images can be achieved by receiving cabinet door images sent by a cabinet door image acquisition device, reading cabinet door images pre-stored in the display cabinet, or receiving cabinet door images sent by other devices or systems. Cabinet door images can include one or more static images, or one or more dynamic images or videos. Cabinet door images can be understood as images including all or part of the display cabinet, or images including all or part of the items in the display cabinet. Based on the cabinet door images, items moved out of or into the display cabinet can be identified for statistical purposes. Based on the statistical results, settlement operations can be performed on the corresponding accounts. Alternatively, based on the statistical results, at least one of the following can be determined: the quantity, type, and location of items stored in the display cabinet after the items are moved out or into the display cabinet.

[0065] In one embodiment of this disclosure, the cabinet door image acquisition device can be installed on the top or bottom of the cabinet door, or on the side wall of the cabinet door near the cabinet body, or on the side wall of the cabinet door away from the cabinet body, or on the door handle of the cabinet door. This application does not limit the specific installation location of the cabinet door image acquisition device.

[0066] In one embodiment of this disclosure, acquiring multiple cabinet images can be achieved by receiving multiple cabinet images sent by a cabinet image acquisition device, reading multiple cabinet images pre-stored in the display cabinet, or receiving multiple cabinet images sent by other devices or systems. The multiple cabinet images can be multiple static images, or multiple dynamic images or videos. A cabinet image can be understood as an image including all or part of the display cabinet, or an image including all or part of the items in the display cabinet. Based on the cabinet images, items moved out of or into the display cabinet can be identified for statistical purposes. Based on the statistical results, settlement operations can be performed on the corresponding accounts. Alternatively, based on the statistical results, at least one of the following can be determined: the quantity, type, and location of items stored in the display cabinet after the items are moved out or into the display cabinet.

[0067] In one embodiment of this disclosure, the cabinet image acquisition device can be connected to the top, side, or bottom surface of the inner wall of the cabinet; alternatively, the cabinet image acquisition device can also be connected to the top, side, or bottom surface of the outer wall of the cabinet. This disclosure does not impose specific limitations on the exact location of the cabinet image acquisition device.

[0068] For example, Figure 6 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown. Figure 7 A schematic top view of a display case according to one embodiment of the present disclosure is shown, such as... Figure 6 as well as Figure 7 As shown, the display case includes a cabinet body 501, a cabinet door 502, a cabinet door image acquisition device 503, and a cabinet body image acquisition device 504. The cabinet door image acquisition device 503 is connected to the side wall of the cabinet door 502 near the cabinet body 501, and the cabinet body image acquisition device 504 is connected to the top surface of the inner wall of the cabinet body 501.

[0069] The cabinet 501 includes a display area 511 and an item entrance / exit 521. The display area 511 is connected to the outside of the cabinet 501 through the item entrance / exit 521. The display area 511 is used to accommodate items stored in the display cabinet. Items in the display area 511 can be moved out of the display cabinet through the item entrance / exit 521, or items can be moved into the display area 511 through the item entrance / exit 521.

[0070] The cabinet doors can be rotatably connected to the cabinet body, or they can be slidably connected to the cabinet body. Alternatively, the cabinet doors can be connected to the cabinet body by folding. It should be noted that this application does not specifically limit the way the cabinet doors are connected to the cabinet body. For ease of understanding, this is just a general description. Figure 6 as well as Figure 7 The following explanation uses the rotatable connection between cabinet door 502 and cabinet body 501 as an example. Cabinet door 502 is used to open or close the item access entrance 521.

[0071] The cabinet door image acquisition device 503 can be used to acquire images of at least one of the following: all display areas 511, some display areas 511, all item entrances / exits 521, and some item entrances / exits 521, in order to obtain cabinet door images. It should be noted that the display cabinet may include only one cabinet door image acquisition device 503, or it may include multiple cabinet door image acquisition devices 503 that acquire cabinet door images from different directions. This disclosure does not impose a specific limitation on the number of cabinet door image acquisition devices 503.

[0072] The cabinet image acquisition device 504 can be used to acquire images of at least one of the following: all display areas 511, some display areas 511, all item entrances / exits 521, and some item entrances / exits 521, in order to obtain cabinet images. It should be noted that the display cabinet may include only one cabinet image acquisition device 504, or it may include multiple cabinet image acquisition devices 504 that acquire cabinet images from different directions. This disclosure does not impose a specific limitation on the number of cabinet image acquisition devices 504.

[0073] In step S102, the image of the area to be picked up corresponding to the hand in the cabinet door image is obtained, and the object is identified in the image of the area to obtain the first object identification result.

[0074] In one embodiment of this disclosure, the image of the retrieval area corresponding to the hand can be understood as the image of the area where the item being retrieved by the hand is located in the cabinet door image. The retrieval area image can also be understood as the image of the area where the item is located at a distance less than or equal to a hand distance threshold. The item in the retrieval area image can be understood as an item that may be being moved by the hand, or an item that is being moved by the hand.

[0075] In one embodiment of this disclosure, obtaining the image of the retrieval area corresponding to the hand in the cabinet door image can be understood as performing image recognition on the cabinet door image to obtain the hand position in the cabinet door image, and then cropping the retrieval area image from the cabinet door image based on the hand position. Alternatively, it can be understood as sending the cabinet door image, receiving retrieval area indication information to indicate the retrieval area, and then cropping the retrieval area image from the cabinet door image based on the retrieval area indication information.

[0076] In one embodiment of this disclosure, object recognition of a retrieval area image can be understood as performing object recognition on the retrieval area image according to a pre-acquired algorithm, or it can be achieved by sending the retrieval area image and receiving a first object recognition result obtained by other devices or systems, such as a cloud server, through image recognition of the retrieval area image.

[0077] In one embodiment of this disclosure, the first item recognition result can be understood as item information used to indicate items in the image of the retrieval area. The item information may include at least one of the following: item quantity, item type, and item price.

[0078] In step S103, the target movement trajectory of the target item is obtained, and the target item area image is cropped from the third target cabinet image according to the target movement trajectory.

[0079] The target item is an item whose position in the first target cabinet image is different from its position in the second target cabinet image. The first target cabinet image, the second target cabinet image, and the third target cabinet image are all multiple cabinet images. The acquisition time of the third target cabinet image is between the acquisition time of the first target cabinet image and the acquisition time of the second target cabinet image.

[0080] In one embodiment of this disclosure, the target movement trajectory can be understood as indicating at least two positions of the target item in the display case, and the at least two positions can be understood as the starting position and the ending position of the target item's movement.

[0081] In one embodiment of this disclosure, obtaining the target movement trajectory of a target item can be understood as performing image recognition on two images from multiple cabinet images, namely a first target cabinet image and a second target cabinet image, to obtain the position of each item. The image recognition results are then compared to determine that the item whose position in the first target cabinet image differs from its position in the second target cabinet image is the target item. Subsequently, target movement trajectory indication information is obtained based on the positions of the target items in the first and second target cabinet images to indicate the target movement trajectory. Alternatively, obtaining the target movement trajectory of a target item can be understood as sending two images from multiple cabinet images and receiving target movement trajectory indication information to indicate the target movement trajectory.

[0082] It should be noted that the first target cabinet image and the second target cabinet image can be any two images from multiple cabinet images, or they can be two images determined from multiple cabinet images according to a preset sampling time interval threshold that satisfy the sampling time interval being greater than or equal to the preset sampling time interval threshold.

[0083] In one embodiment of this disclosure, the third target cabinet image can be understood as an image of a moving target item among multiple cabinet images.

[0084] In one embodiment of this disclosure, the image of the target item area is captured in the third target cabinet image according to the target movement trajectory. This can be understood as capturing an image in the third target cabinet image that includes the at least two positions indicated by the target movement trajectory, i.e., the target item area image. The at least two positions should include the position of the target item in the first target cabinet image, i.e., the starting position of movement, and the position of the target item in the second target cabinet image, i.e., the ending position of movement of the target item.

[0085] In step S104, object recognition is performed based on the target object region image to obtain a second object recognition result.

[0086] In one embodiment of this disclosure, the method of performing object recognition based on the target object region image to obtain a second object recognition result can be understood as performing object recognition on the target object region image according to a pre-acquired algorithm, or it can be achieved by sending the target object region image and receiving the second object recognition result obtained by other devices or systems, such as a cloud server, through image recognition on the target object region image.

[0087] It should be noted that, as Figure 5 As shown, the order of steps S102-S103 and step S104 can be reversed, or steps S102-S103 and step S104 can be executed simultaneously. This disclosure does not show the specifics of this.

[0088] In step S105, matching is performed based on the first item recognition result and the second item recognition result to obtain the matching item recognition result.

[0089] In one embodiment of this disclosure, matching based on the first item identification result and the second item identification result can be understood as determining the completely identical parts of the first item identification result and the second item identification result, and obtaining the matching item identification result based on the completely identical parts; it can also be understood as determining the parts of the first item identification result and the second item identification result with a similarity higher than a similarity threshold, and obtaining the matching item identification result based on the parts with a similarity higher than the similarity threshold.

[0090] For example, consider a display case used to provide unmanned vending services. When a customer wants to purchase an item from the display case, they can use a mobile communication terminal to scan the QR code on the surface of the display case to access the corresponding cloud authorization server and send a statistical authorization request to the cloud authorization server, i.e., requesting to purchase the item from the display case. After the cloud authorization server approves the statistical authorization request, it sends statistical authorization information to the display case. The display case receives the statistical authorization information sent by the cloud server and, in response, unlocks the display case door. After the door is unlocked, the customer can open the door and take the item from the display case through the item entrance / exit. After taking the item, the customer can close the door. From the time the door is opened until it is closed, the door image acquisition device at the display case door continuously captures images of the item entrance / exit, i.e., the door image, while the cabinet image acquisition device at the display case body continuously captures images of the item entrance / exit and the display area, i.e., the cabinet body image. The display case can acquire an image of the area corresponding to the hand in the cabinet door image, and perform item recognition on the image of the area to obtain a first item recognition result. It also acquires the target movement trajectory of the target item and, based on the target movement trajectory, crops an image of the target item area from a third target cabinet image. Item recognition is then performed on the target item area image to obtain a second item recognition result. The first and second item recognition results are then matched to obtain a matched item recognition result. The display case then sends this matched item recognition result to a cloud-based statistics server, which deducts the customer's account fee based on the result. The images used for the first and second item recognition results were collected from at least two significantly different locations. Even if the target item is obscured in an image collected from one location, the probability of the target item being obscured in images collected from other locations is low due to the significant differences between the two locations. Therefore, the accuracy of at least one item recognition result in the first and second item recognition results will not be affected by the obscuration of the target item (i.e., the item removed from the display case) in the collected images. This ensures that the item removed from the display case can be accurately identified based on the obtained matching item recognition results, thereby improving the reliability of deducting fees from the customer's account.

[0091] For example, let's illustrate this using a display case to provide unmanned vending services. When maintenance personnel, such as convenience store clerks, need to replenish the display case, they can use a mobile communication terminal to scan the QR code on the surface of the display case to access the corresponding cloud authorization server and send a replenishment request to the server. Upon receiving the authorization request, the cloud server sends authorization information to the display case, granting permission for the maintenance personnel to replenish the items. The display case receives the authorization information from the cloud server and unlocks its door in response. Once unlocked, the maintenance personnel can open the door and insert the scanned items into the display case through the item inlet / outlet. After replenishing, the maintenance personnel can close the door. From the time the door is opened until it is closed, the door image acquisition device continuously captures images of the item inlet / outlet (door image), while the cabinet image acquisition device continuously captures images of the item inlet / outlet and the display area (cabinet image). The display case can acquire an image of the area corresponding to the hand in the cabinet door image, and perform item recognition on the image of the area to obtain a first item recognition result. It can also acquire the target movement trajectory of the target item, and capture an image of the target item area in a third target cabinet image based on the target movement trajectory. Based on the target item area image, it can perform item recognition to obtain a second item recognition result. It can then match the first item recognition result and the second item recognition result to obtain a matching item recognition result. The cloud statistics server can then determine the quantity, type, and location of items that maintenance personnel have added to the display case based on the item recognition result, so as to determine the quantity, type, and location of items stored in the display case after the items are moved in. The images used for the first and second item recognition results were collected from at least two different locations with significant differences. Even if the target item is occluded in the image collected from one location, the probability of the target item being occluded in the images collected from other locations within the at least two different locations is small due to the significant differences between the two locations. Therefore, the accuracy of at least one item recognition result in the first and second item recognition results will not be affected by the occlusion of the target item (i.e., the item moved from the display case) in the collected image. This ensures that the item moved from the display case can be accurately determined based on the obtained matching item recognition results, thereby improving the reliability of the quantity, type, and location of the items stored in the display case after the item is moved.

[0092] For example, consider a display case used to provide item retrieval services to a target user, where the target user can be understood as a user belonging to a specific unit or department. When a user needs to purchase an item from the display case, they can use a mobile communication terminal to scan the QR code on the surface of the display case to access the server corresponding to the display case and send an identity authentication request to the server, i.e., requesting authentication of the user's identity; alternatively, they can use a corresponding identification (such as an employee badge, ID card, etc.) to scan the QR code on the display case, which will then send an identity authentication request to the corresponding server. When the display case receives item retrieval authorization information from the server, it can be understood that the server has confirmed the user as the target user, and the display case can unlock its door in response to the item retrieval authorization information. After the door is unlocked, the user can open the door and retrieve the item from the display case through the item access window. After retrieving the item, the user can close the door. From the moment the cabinet door is opened until it is closed, the cabinet door image acquisition device continuously captures images of the item entry / exit points, i.e., the cabinet door image. Simultaneously, the cabinet body image acquisition device continuously captures images of the item entry / exit points and the display area, i.e., the cabinet body image. The display cabinet can acquire images of the area corresponding to the hand in the cabinet door image, and perform item recognition on the area image to obtain a first item recognition result. It also acquires the target movement trajectory of the target item and, based on the target movement trajectory, extracts the target item area image from a third target cabinet image. Item recognition is then performed on the target item area image to obtain a second item recognition result. The first and second item recognition results are matched to obtain a matched item recognition result. This allows the cloud-based statistics server to determine the items taken by the user from the display cabinet, and further, to determine the types and quantities of remaining items in the display cabinet. The images used for the first and second item recognition results are collected from at least two different locations with significant differences. Even if the target item is occluded in the image collected from one location, the probability of the target item being occluded in the images collected from other locations within the at least two different locations is small due to the significant differences between the two locations. Therefore, the accuracy of at least one item recognition result in the first and second item recognition results will not be affected by the occlusion of the target item (i.e., the item removed from the display case) in the collected image. This ensures that the items removed from the display case can be accurately determined based on the obtained matching item recognition results, thereby improving the reliability of the quantity, type, and location of the items removed by the user from the display case, and / or the quantity, type, and location of the items remaining in the display case after the items are removed.

[0093] The above technical solution acquires cabinet door images captured by a cabinet door image acquisition device and multiple cabinet body images captured by a cabinet body image acquisition device. Since the cabinet door image acquisition device is located at the cabinet door of the display cabinet, and the cabinet body image acquisition device is located at the cabinet body, the cabinet door images and cabinet body images are acquired from at least two significantly different locations. The solution then acquires an image of the area corresponding to the hand in the cabinet door image and performs item recognition on this area image to obtain a first item recognition result. The second item recognition result is based on the image acquired from the location of the cabinet door, indicating that the item is being moved by the hand (i.e., being moved). The system identifies the target item (entering or leaving the display case); acquires the target item's movement trajectory; and extracts an image of the target item region from the third target cabinet image based on the target movement trajectory. The target item is defined as an item whose position in the first target cabinet image differs from its position in the second target cabinet image. Both the first and second target cabinet images are multiple cabinet images, and the third target cabinet image is also a multiple cabinet image, acquired between the acquisition times of the first and second target cabinet images. Therefore, the target item region extracted from the third target cabinet image... The target item in the product area image is in a moving state (i.e., being moved into or out of the display case); item recognition is performed based on the target item area image to obtain a second item recognition result, which is the recognition result of the target item being moved into or out of the display case based on the image taken from the location of the display case; matching is performed based on the first item recognition result and the second item recognition result to obtain a matched item recognition result, wherein the images on which the first item recognition result and the second item recognition result are based during item recognition are taken from at least two significantly different locations, and Even if the target item is occluded in an image acquired from one location, the probability of the target item being occluded in images acquired from other locations within those two locations is relatively small due to the significant differences between the two locations. Therefore, the accuracy of at least one item recognition result in the first item recognition result and the second item recognition result will not be affected by the occlusion of the target item (i.e., the item moved out or into the display case) in the acquired image. This ensures that the item moved out or into the display case can be accurately determined based on the obtained matching item recognition result, thereby improving the reliability of recognizing items moved out or into the display case.

[0094] In one implementation of this disclosure, obtaining the image of the retrieval area corresponding to the hand in the cabinet door image includes:

[0095] A pre-trained hand detection model is obtained, and the cabinet door image is input into the hand detection model to obtain the hand detection information output by the hand detection model. The hand detection information is used to indicate the position of the hand in the cabinet door image.

[0096] Obtain an image of the picking area based on hand detection information;

[0097] Perform item recognition on the retrieval area to obtain the first item recognition result, including:

[0098] Obtain a pre-trained first item category detection model, and input the captured region image into the first item category detection model to obtain the first item recognition result output by the first item category detection model.

[0099] In one embodiment of this disclosure, the hand detection model and the first item category detection model can be pre-stored in a display case or obtained from other devices or systems. The hand detection model and the first item category detection model can be a neural network (NN) model, a convolutional neural network (CNN) model, or a long short-term memory (LSTM) model, etc.

[0100] In this embodiment, by acquiring a pre-trained hand detection model and inputting the cabinet door image into the hand detection model to obtain the hand detection information output by the hand detection model, and by acquiring the retrieval area image based on the hand detection information, it can be ensured that the retrieval area image includes items moved by the hand with a high probability. By acquiring a pre-trained first item category detection model and inputting the retrieval area image into the first item category detection model to obtain the first item recognition result output by the first item category detection model, it can be ensured that the accuracy of the item information of the retrieval area indicated by the first item recognition result is high.

[0101] In one implementation of this disclosure, the method further includes;

[0102] The scanning device acquires the scanning information collected by the scanning device and determines the item to be scanned based on the scanning information. The scanning device is installed on the cabinet door or cabinet body.

[0103] In response to a mismatch between the first item recognition result and the scanned item, at least one of the hand annotation information and the first annotation information of the scanned item is obtained. The hand annotation information is used to indicate the position of the hand corresponding to the scanned item in the cabinet door image, and the first annotation information of the scanned item is used to indicate the scanned item in the cabinet door image.

[0104] The cabinet door image is used as input, and the hand annotation information is used as output to train the hand detection model;

[0105] And / or, based on the first annotation information of the scanned item, crop the first area image of the scanned item in the cabinet door image, and use the first area image of the scanned item as input and the item information of the scanned item as output to train the first item category detection model.

[0106] In one embodiment of this disclosure, obtaining the barcode information collected by the barcode scanning device can be achieved by receiving an image of the display cabinet door sent by the barcode scanning device, reading barcode information pre-stored in the display cabinet, or receiving barcode information sent by other devices or systems. The barcode information can be used to indicate the item information of the scanned item, which may include at least one of the following: quantity, type, and price.

[0107] In one embodiment of this disclosure, the hand marking information can be sent by a server or input through the human-computer interaction device of the display case.

[0108] In one embodiment of this disclosure, the first labeling information of the scanned item can be sent by the server or input through the human-computer interaction device of the display case.

[0109] In one implementation of this disclosure, before obtaining the hand marking information or the first marking information of the scanned item input through the human-computer interaction device of the display case, an image of the cabinet door can also be displayed.

[0110] In one embodiment of this disclosure, the display of cabinet door images can be achieved through a human-computer interaction device on the display cabinet, such as a display screen, or through other devices or systems.

[0111] In one embodiment of this disclosure, the hand marking information or the first marking information of the scanned item input through the human-computer interaction device of the display case can be obtained by acquiring the hand marking information or the first marking information of the scanned item input by the user through at least one of the keyboard, touch screen, and touchpad on the display case; it can also be obtained by acquiring the voice information input by the user through the microphone on the display case and obtaining the hand marking information or the first marking information of the scanned item through voice recognition; or it can be obtained by acquiring the image information of the user's actions through the camera on the display case and obtaining the hand marking information or the first marking information of the scanned item through image recognition.

[0112] For example, the display case can display a cabinet door image through a graphical user interface (GUI) on a touchscreen, and obtain hand annotation information input actions or scanned item first annotation information input actions corresponding to the cabinet door image displayed in the GUI, inputted by the user via the touchscreen. The GUI is an interactive interface for an application (APP) running on the display case. Furthermore, the GUI can also display one or more visual prompts on the touchscreen that allow the user to perform hand annotation information input actions or scanned item first annotation information input actions. These visual prompts can be text, graphics, or any combination thereof. Hand annotation information input actions or scanned item first annotation information input actions may include contact with the touchscreen. In some embodiments, in addition to visual cues, the display case may also provide non-visual cues to indicate the progress of hand annotation input or the input of the first annotation information of the scanned item. These non-visual cues may include audio cues (e.g., sound) or physical cues (e.g., vibration). In some embodiments, the hand annotation input or the input of the first annotation information of the scanned item is a predetermined gesture performed on the touchscreen. The gesture used here refers to the movement of an object / accessory in contact with the touchscreen. For example, the predetermined gesture may include contacting the touchscreen at the location of the hand in the cabinet door image displayed on the touchscreen (initialization gesture), and breaking contact after maintaining continuous contact with the touchscreen for more than a preset contact time threshold (completing the gesture); the predetermined gesture may also include contacting the touchscreen at the location of the scanned item in the cabinet door image displayed on the touchscreen (initialization gesture), and breaking contact after maintaining continuous contact with the touchscreen for more than a preset contact time threshold (completing the gesture).

[0113] For ease of explanation, in the process of acquiring the user's hand marking information and the first marking information of the scanned item, and in other embodiments described below, touch on the touchscreen will be described as being performed by the user using at least one hand and one or more fingers. However, it should be understood that the touch can also be performed using any suitable object or accessory, such as a stylus, finger, etc. The touch may include: one or more taps on the touchscreen, maintaining continuous contact with the touchscreen, moving the contact point while maintaining continuous contact, interrupting the contact, or any combination thereof.

[0114] The display case detects touch on the touchscreen. If the touch does not correspond to an attempt to perform a hand gesture input or a first-order label input for the scanned item, or if the touch corresponds to a failed or abandoned attempt by the user to perform a hand gesture input or a first-order label input for the scanned item, then the display case will not acquire the corresponding label information. For example, if the first-order label input for the scanned item involves contacting the touchscreen at the location of the scanned item in the image displayed on the touchscreen, and the contact is interrupted after exceeding a preset contact time threshold, but the detected contact is a series of random taps on the touchscreen, then the touch does not correspond to the first-order label input for the scanned item. Similarly, if the hand gesture input involves contacting the touchscreen at the location of a hand in the image displayed on the touchscreen, and the contact is interrupted after exceeding a preset contact time threshold, but the detected contact is a series of random taps on the touchscreen, then the touch does not correspond to the hand gesture input.

[0115] If the contact corresponds to the successful execution of the action of inputting the first label information of the scanned item, that is, the user successfully executes the action of inputting the first label information of the scanned item, then the display cabinet can obtain the corresponding first label information of the scanned item based on the action of inputting the first label information of the scanned item.

[0116] If the contact corresponds to the successful execution of the hand annotation information input action, that is, the user successfully performed the hand annotation information input action, then the display case can obtain the corresponding hand annotation information based on the hand annotation information input action.

[0117] Figure 8 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figure 9 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figure 10 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figures 8-10 This diagram illustrates the graphical user interface (GUI) of the display case at different moments during the execution of the hand annotation input action.

[0118] exist Figure 8 In the middle, the user indicates with finger 61 that finger 61 has not yet touched the touch screen 62 of the display case, at which time the display case will not obtain the corresponding hand annotation information.

[0119] exist Figure 9In this system, the user initiates the hand annotation input action by touching the touchscreen 62 of the display case with their finger 61. Specifically, the user continuously touches the touchscreen 62 with their finger 61 and moves their finger 61 along the outline of a hand 64 in the cabinet door image displayed by the graphical user interface GUI 63. The display case then begins to record the movement trajectory 65 of the finger 61. It should be noted that when the finger 61 is still in contact with the touchscreen 62, the display case will not obtain the corresponding hand annotation information based on the recorded movement trajectory.

[0120] exist Figure 10 In this process, the user ends the hand annotation information input action by removing their finger 61 from the touch screen 62 of the display case. Specifically, in response to the finger 61 being out of contact with the touch screen 62 for more than a preset time difference, the display case identifies at least one hand 64 in the cabinet door image displayed in the graphical user interface GUI 63 based on the recorded movement trajectory, and obtains hand annotation information to indicate the hand 64.

[0121] Figure 11 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figure 12 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figure 13 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figures 11-13 The diagram shows the graphical user interface (GUI) of the display case at different moments during the execution of the first labeling information input action of the scanned item.

[0122] exist Figure 11 In the middle, the user indicates with finger 71 that finger 71 has not yet touched the touch screen 72 of the display case. At this time, the display case will not obtain the corresponding first label information of the scanned item.

[0123] exist Figure 12 In this system, the user initiates the input of the first label information for the item to be scanned by touching the touchscreen 72 of the display case with their finger 71. Specifically, the user continuously touches the touchscreen 72 with their finger 71 and moves their finger 71 along the outline of the item 74 to be scanned in the cabinet door image displayed by the graphical user interface GUI 73. The display case then begins to record the movement trajectory 75 of the finger 71. It should be noted that when the finger 71 is still in contact with the touchscreen 72, the display case will not obtain the corresponding first label information for the item to be scanned based on the recorded movement trajectory.

[0124] exist Figure 13In this process, the user ends the input of the first label information of the scanned item by removing their finger 71 from the touch screen 72 of the display case. Specifically, in response to the finger 71 leaving the touch screen 72 for more than a preset time difference, the display case determines at least one scanned item 74 in the cabinet door image displayed in the graphical user interface GUI 73 based on the recorded movement trajectory, and obtains the first label information of the scanned item 74 to indicate the scanned item 74. In one embodiment of this disclosure, the item information of the scanned item can be understood as indicating at least one of the category, quantity, and price of the scanned item.

[0125] In one embodiment of this disclosure, the first area image can be understood as the image of the area where the scanned item is located.

[0126] In this embodiment, by acquiring the scanning information collected by the scanning device and determining the item to be scanned based on the scanning information, in response to a mismatch between the first item recognition result and the item being scanned, at least one of hand annotation information and the first annotation information of the item being scanned is acquired. The cabinet door image is used as input, and the hand annotation information is used as output to train the hand detection model; and / or, based on the first annotation information of the item being scanned, a first region image of the item being scanned is cropped from the cabinet door image, and the first region image of the item being scanned is used as input, and the item information of the item being scanned is used as output to train the first item category detection model. This ensures that in cases of hand detection mismatch, the hand detection model can be trained. When the model fails to successfully identify a hand in the cabinet door image, and / or when the first item category detection model fails to successfully identify an item in the retrieval area image, the trained hand detection model should be able to learn the pattern between the previously unidentified hand and the cabinet door image, and / or the first item category detection model should be able to learn the pattern between the previously unidentified scanned item and the retrieval area image, to ensure a high success rate for identifying hands in the cabinet door image based on the trained hand detection model, and / or to ensure a high success rate for identifying items in the retrieval area image based on the trained first item category detection model.

[0127] In one implementation of this disclosure, the cabinet door image is used as input, and hand annotation information is used as output. Before training the hand detection model, the method further includes:

[0128] Receive the first update weight parameters sent by the first edge server, and update the hand detection model according to the first update weight parameters;

[0129] Using cabinet door images as input and hand annotation information as output, a hand detection model is trained, including:

[0130] The cabinet door image is used as input, and the hand annotation information is used as output to train the updated hand detection model;

[0131] The method further includes:

[0132] In response to the failure of the trained hand detection model to converge, the first gradient update vector is obtained based on the trained hand detection model and sent to the first edge server.

[0133] Alternatively, in response to the convergence of the trained hand detection model, the trained hand detection model can be stored as the target hand detection model.

[0134] In one implementation of this disclosure, a first edge server aggregates gradient update vectors and updates the weight parameters of the hand detection model on the first edge server based on the aggregated gradient update vectors to obtain updated weight parameters. The first edge server can be a cloud server or a server provided by a display case control service provider. It should be noted that one first edge server can correspond to one or more display cases. For example, a display case control service provider can divide its managed area into multiple blocks, and multiple display cases in each block can correspond to one first edge server.

[0135] The hand detection model on the first edge server can be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.

[0136] In one implementation of this disclosure, the updated weight parameters received by the display case from the first edge server are obtained by the first edge server aggregating gradient update vectors sent by multiple display cases and updating the weight parameters of the hand detection model on the first edge server according to the aggregated gradient update vectors. Therefore, the updated hand detection model on the display case can reflect the common patterns between the corresponding image and the location of the hand in the image, which were learned by the hand detection model on the first edge server in the previous training round. Then, the cabinet door image is used as input, and the hand annotation information is used as output to train the updated hand detection model. This allows the updated hand detection model on the display case to learn not only the common patterns but also the patterns between the cabinet door image and the hand in the cabinet door image, which are acquired by the display case itself. This enables the trained hand detection model on the display case to learn the private patterns between the cabinet door image and the hand in the cabinet door image. When the trained hand detection model on the display case fails to converge, it indicates that the trained hand detection model on the display case still needs to be trained further. The hand detection model on the subsequent display case obtains the gradient update vector and sends it out. This allows the first edge server to continue obtaining the corresponding update weight parameters based on the gradient update vectors uploaded from multiple display cases, thereby continuing to train the hand detection model on each display case. When the trained hand detection model on the display case converges, it can be considered that the converged hand detection model on the display case can accurately identify the hands in the cabinet door images it has acquired. The converged hand detection model on the display case can be stored as the target hand detection model, that is, a model with high accuracy in identifying hands in cabinet door images.

[0137] In the above technical solution, on the one hand, the final target hand detection model can be a model that learns both common rules and private rules, and its accuracy in recognizing hands in cabinet door images is relatively high; on the other hand, since the process of continuing to train the hand detection model on each display cabinet is jointly executed by the display cabinet and the first edge server, compared with the process of further training the hand detection model by the display cabinet or the server alone, less processing resources are required and the training speed is faster.

[0138] In one implementation of this disclosure, the method further includes taking the image of the first region of the item to be scanned as input and the item information of the item to be scanned as output, before training the first item category detection model:

[0139] Receive the second update weight parameters sent by the second edge server, and update the first item category detection model according to the second update weight parameters;

[0140] The first region image of the item to be scanned is used as input, and the item information of the item to be scanned is used as output to train the first item category detection model, including:

[0141] The first region image of the item to be scanned is used as input, and the item information of the item to be scanned is used as output to train the updated first item category detection model.

[0142] The method also includes:

[0143] In response to the failure of the first item category detection model after training to converge, the second gradient update vector is obtained based on the first item category detection model after training and sent to the second edge server;

[0144] Alternatively, in response to the convergence of the trained first item category detection model, the trained first item category detection model is stored as the target first item category detection model.

[0145] In one implementation of this disclosure, a second edge server is used to aggregate gradient update vectors and update the weight parameters of the first item category detection model on the second edge server according to the aggregated gradient update vectors to obtain updated weight parameters. The second edge server can be a cloud server or a server provided by a display case control service provider. It should be noted that one second edge server can correspond to one or more display cases. For example, a display case control service provider can divide its managed area into multiple blocks, and multiple display cases in each block can correspond to one second edge server.

[0146] The first item category detection model on the second edge server can be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.

[0147] In one implementation of this disclosure, the updated weight parameters sent by the second edge server and received by the display cabinet are obtained by the second edge server aggregating the gradient update vectors sent by multiple display cabinets and updating the weight parameters of the first item category detection model on the second edge server according to the aggregated gradient update vectors. Therefore, the updated first item category detection model on the display cabinet can reflect the common pattern between the first area image and the item information of the scanned item in the first area image that the first item category detection model on the second edge server learned in the previous training. Then, the first region image is used as input, and the item information of the scanned item is used as output to train the updated first item category detection model. This allows the updated first item category detection model on the display case to learn not only common patterns but also personalized patterns specific to the first region image acquired by the display case itself and the item information of the scanned item within that image. This enables the trained first item category detection model on the display case to learn the private patterns between the first region image acquired by the display case and the item information of the scanned item within that image. If the trained first item category detection model on the display case fails to converge, it indicates that the trained model still needs further training. The gradient update vector is obtained from the first item category detection model on the trained display case and sent to the second edge server. This allows the second edge server to continue to obtain the corresponding update weight parameters based on the gradient update vectors uploaded from multiple display cases, thereby continuing to train the first item category detection model on each display case. When the trained first item category detection model on the display case converges, it can be considered that the converged first item category detection model on the display case can accurately identify the item information in the retrieval area image input to the first item category detection model. The converged first item category detection model on the display case can be stored as the target first item category detection model, that is, a model with high accuracy in identifying items in the retrieval area image.

[0148] In the above technical solution, on the one hand, the final target first item category detection model can be a model that learns both common rules and private rules, and its accuracy in identifying items in the retrieval area image is relatively high; on the other hand, since the process of continuing to train the first item category detection model on each display case is jointly performed by the display case and the second edge server, compared with the display case or server alone further training the first item category detection model, the required processing resources are less and the training speed is faster.

[0149] In one implementation of this disclosure, before obtaining the pre-trained hand detection model, the method further includes:

[0150] The angle between the cabinet door and the cabinet body when acquiring images of the cabinet door;

[0151] Obtain a pre-trained hand detection model, including:

[0152] Obtain a pre-trained hand detection model corresponding to the angle of the cabinet door.

[0153] In one implementation of this disclosure, the included angle between the cabinet door and the cabinet body can be understood as being acquired by an angle sensing device located at the cabinet body, the cabinet door, or the door hinge connecting the cabinet body and the cabinet door at the moment of acquiring the cabinet door image. Alternatively, it can be understood as being acquired by other angle sensing devices outside the display cabinet, such as an infrared goniometer located near the display cabinet, at the moment of acquiring the cabinet door image.

[0154] In one implementation of this disclosure, the cabinet door angle between the cabinet door and the cabinet body when acquiring the cabinet door image can be understood as receiving cabinet door angle information sent by an angle sensing device, or reading cabinet door angle information stored in the display cabinet, or receiving cabinet door angle information sent by other devices or systems. The cabinet door angle information is used to indicate the angle value of the cabinet door angle and the acquisition time of the angle value of the cabinet door angle. The cabinet door angle between the cabinet door and the cabinet body when acquiring the cabinet door image is determined by matching the acquisition time of the angle value of the cabinet door angle with the acquisition time of the cabinet door image.

[0155] In one implementation of this disclosure, obtaining a pre-trained hand detection model corresponding to the cabinet door angle can be understood as searching among multiple hand detection models based on the cabinet door angle to determine any hand detection model whose corresponding angle matches the cabinet door angle. Matching the cabinet door angle can be understood as the corresponding angle being exactly the same as the cabinet door angle, or as the angle difference between the corresponding angle and the cabinet door angle being less than or equal to an angle difference threshold.

[0156] In the above technical solution, by acquiring the cabinet door angle between the cabinet door and the cabinet body when acquiring the cabinet door image, and acquiring a pre-trained hand detection model corresponding to the cabinet door angle, it can be ensured that the hand detection model has a high success rate in detecting hands in the cabinet door image of disassembled machine under the cabinet door angle, thereby improving the accuracy of the acquired hand detection information.

[0157] In one implementation of this disclosure, object recognition is performed based on an image of a target object region, including:

[0158] Obtain a pre-trained second item category detection model, and input the target item region image into the second item category detection model to obtain the second item recognition result output by the second item category detection model;

[0159] The method further includes:

[0160] The scanning device acquires the scanning information collected by the scanning device and determines the item to be scanned based on the scanning information. The scanning device is installed on the cabinet door or cabinet body.

[0161] In response to a mismatch between the second item recognition result and the scanned item, the second annotation information of the scanned item is obtained. The second annotation information of the scanned item is used to indicate the scanned item in multiple cabinet images.

[0162] The movement trajectory of the scanned item is obtained based on the second label information of the scanned item, and the second area image of the scanned item is extracted from any cabinet image among multiple cabinet images based on the movement trajectory of the scanned item.

[0163] The second area image of the item to be scanned is used as input, and the item recognition result corresponding to the item to be scanned is used as output to train the second item category detection model.

[0164] In one embodiment of this disclosure, the second item category detection model can be pre-stored in a display case or obtained from other devices or systems. The second item category detection model can be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.

[0165] By acquiring a pre-trained second item category detection model and inputting the target item region image into the second item category detection model to obtain the second item recognition result output by the second item category detection model, it can be ensured that the accuracy of the item information in the target item region image indicated by the second item recognition result is high.

[0166] In one embodiment of this disclosure, obtaining the barcode information collected by the barcode scanning device can be achieved by receiving an image of the display cabinet door sent by the barcode scanning device, reading barcode information pre-stored in the display cabinet, or receiving barcode information sent by other devices or systems. The barcode information can be used to indicate the item information of the scanned item, which may include at least one of the following: quantity, type, and price.

[0167] In one embodiment of this disclosure, the second labeling information of the scanned item can be sent by the server or input through the human-computer interaction device of the display case.

[0168] In one implementation of this disclosure, multiple images of the display case can be displayed before obtaining the second label information of the scanned item input through the human-computer interaction device of the display case.

[0169] In one embodiment of this disclosure, the display of cabinet images can be achieved through a human-computer interaction device on the display cabinet, such as a display screen, or through other devices or systems.

[0170] In one embodiment of this disclosure, the second labeling information of the scanned item input through the human-computer interaction device of the display case can be obtained by acquiring the second labeling information of the scanned item input by the user through at least one of the keyboard, touch screen, and touchpad on the display case; it can also be obtained by acquiring the user's voice information through the microphone on the display case and obtaining the second labeling information of the scanned item by performing voice recognition on the voice information; or it can be obtained by acquiring the image information of the user's actions through the camera on the display case and obtaining the second labeling information of the scanned item by performing image recognition on the image information.

[0171] For example, the display case can display a cabinet image through a graphical user interface (GUI) on a touchscreen, and obtain the second labeling information of the scanned item corresponding to the cabinet image displayed in the GUI, inputted by the user via the touchscreen. The second labeling information of the scanned item can be obtained based on the user's input. The GUI is the interactive interface of an application (APP) running on the display case. Further, the GUI can also display one or more visual prompts on the touchscreen for the user to perform the input action of the second labeling information of the scanned item. These visual prompts provide the user with hints or reminders to perform the input action. These visual prompts can be text, graphics, or any combination thereof. The input action of the second labeling information of the scanned item may include contact with the touchscreen. In some embodiments, in addition to visual prompts, the display case may also provide non-visual prompts to indicate the progress of the input action of the second labeling information of the scanned item. These non-visual prompts may include audio prompts (e.g., sound) or physical prompts (e.g., vibration). In some embodiments, the input action of the second labeling information of the scanned item is a predetermined gesture performed on the touchscreen. The gestures used here refer to the movement of an object / accessory in contact with the touchscreen. For example, the predetermined gesture may include touching the touchscreen at the location of the hand in the cabinet image displayed on the touchscreen (initialization gesture), and breaking the contact after maintaining continuous contact with the touchscreen for more than a preset contact time threshold (completing the gesture); the predetermined gesture may also include touching the touchscreen at the location of the scanned item in the cabinet image displayed on the touchscreen (initialization gesture), and breaking the contact after maintaining continuous contact with the touchscreen for more than a preset contact time threshold (completing the gesture).

[0172] For ease of explanation, in the process of acquiring the second labeling information of the scanned item input by the user, and in other embodiments described below, contact on the touchscreen will be described as being performed by the user using at least one hand and one or more fingers. However, it should be understood that the contact can also be performed using any suitable object or accessory, such as a stylus, finger, etc. The contact may include: one or more taps on the touchscreen, maintaining continuous contact with the touchscreen, moving the contact point while maintaining continuous contact, interrupting the contact, or any combination thereof.

[0173] The display case detects touches on the touchscreen. If the touch does not correspond to an attempt to input the second labeling information of the scanned item, or if the touch corresponds to a failed or abandoned attempt by the user to input the second labeling information of the scanned item, then the display case will not acquire the corresponding labeling information. For example, if the input of the second labeling information of the scanned item involves contacting the touchscreen at the location of the scanned item in the image displayed on the touchscreen, and the contact is interrupted after the duration of contact exceeds a preset contact time threshold, but the detected contact is a series of random taps on the touchscreen, then the contact does not correspond to the input of the second labeling information of the scanned item.

[0174] If the contact corresponds to the successful execution of the action of inputting the second label information of the scanned item, that is, the user successfully executes the action of inputting the second label information of the scanned item, then the display cabinet can obtain the corresponding second label information of the scanned item based on the action of inputting the second label information of the scanned item.

[0175] Figure 14 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figure 15 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figure 16 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figures 14-16 The diagram shows the graphical user interface (GUI) of the display case at different moments during the execution of the second labeling information input action for the scanned item.

[0176] exist Figure 14 In the middle, the user indicates with finger 81 that finger 81 has not yet touched the touch screen 82 of the display case. At this time, the display case will not obtain the corresponding second label information of the scanned item.

[0177] exist Figure 15In this system, the user initiates the input of the second labeling information for the scanned item by touching the touchscreen 82 of the display case with their finger 81. Specifically, the user continuously touches the touchscreen 82 with their finger 81 and moves their finger 81 along the outline of the scanned item 84 in the display case image displayed by the graphical user interface GUI 83. The display case then begins to record the movement trajectory 85 of the finger 81. It should be noted that when the finger 81 is still in contact with the touchscreen 82, the display case will not obtain the corresponding second labeling information for the scanned item based on the recorded movement trajectory.

[0178] exist Figure 16 In this process, the user ends the input of the second labeling information of the scanned item by removing their finger 81 from the touch screen 82 of the display case. Specifically, in response to the finger 81 leaving the touch screen 82 for more than a preset time difference, the display case determines at least one scanned item 84 in the display case image displayed in the graphical user interface 83 based on the recorded movement trajectory, and obtains the second labeling information of the scanned item 84. In one embodiment of this disclosure, the item information of the scanned item can be understood as indicating at least one of the category, quantity, and price of the scanned item.

[0179] In one embodiment of this disclosure, the second area image can be understood as an image of the area where the scanned item is located.

[0180] In this embodiment, by acquiring the scanning information collected by the scanning device and determining the scanned item based on the scanning information, in response to a mismatch between the second item recognition result and the scanned item, second annotation information of the scanned item is acquired. This second annotation information is used to indicate the scanned item in multiple cabinet images. The movement trajectory of the scanned item is obtained based on the second annotation information. A second region image of the scanned item is extracted from any one of the multiple cabinet images based on the movement trajectory. This second region image is used as input, and the corresponding item recognition result is used as output to train the second item category detection model. This scheme ensures that when the second item category detection model fails to successfully identify the scanned item in the cabinet image, the trained second item category detection model can learn the pattern between the previously unidentified scanned item and the cabinet image, ensuring a high success rate for item recognition in cabinet images based on the trained second item category detection model.

[0181] In one implementation of this disclosure, the second region image of the item to be scanned is used as input, and the item recognition result corresponding to the item to be scanned is used as output. Before training the second item category detection model, the method further includes:

[0182] Receive the third update weight parameters sent by the third edge server, and update the second item category detection model according to the third update weight parameters;

[0183] The second item category detection model is trained by taking the image of the second region of the item to be scanned as input and the item recognition result corresponding to the item to be scanned as output, including:

[0184] The second region image of the item to be scanned is used as input, and the item recognition result corresponding to the item to be scanned is used as output to train the updated second item category detection model.

[0185] The method also includes:

[0186] In response to the failure of the trained second item category detection model to converge, the third gradient update vector is obtained based on the trained second item category detection model and sent to the third edge server;

[0187] Alternatively, in response to the convergence of the trained second item category detection model, the trained second item category detection model is stored as the target second item category detection model.

[0188] In one implementation of this disclosure, a third edge server is used to aggregate gradient update vectors and update the weight parameters of the second item category detection model on the third edge server according to the aggregated gradient update vectors to obtain updated weight parameters. The third edge server can be a cloud server or a server provided by a display case control service provider. It should be noted that one third edge server can correspond to one or more display cases. For example, a display case control service provider can divide its managed area into multiple blocks, and multiple display cases in each block can correspond to one third edge server.

[0189] The second item category detection model on the third edge server can be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.

[0190] In one implementation of this disclosure, the updated weight parameters sent by the third edge server and received by the display cabinet are obtained by the third edge server aggregating the gradient update vectors sent by multiple display cabinets and updating the weight parameters of the second item category detection model on the third edge server according to the aggregated gradient update vectors. Therefore, the updated second item category detection model on the display cabinet can reflect the common pattern between the second region image and the item information of the scanned item in the second region image, which was learned by the second item category detection model on the third edge server in the previous training. Then, the second region image is used as input, and the item information of the scanned item is used as output to train the updated second item category detection model. This allows the updated second item category detection model on the display case to learn not only common patterns but also personalized patterns specific to the second region image acquired by the display case itself and the item information of the scanned item within that image. This enables the trained second item category detection model on the display case to learn the private patterns between the second region image acquired by the display case and the item information of the scanned item within that image. If the trained second item category detection model fails to converge, it indicates that the model still needs further training. After training, the second item category detection model on the display case obtains and sends the gradient update vector. This allows the third edge server to continue obtaining the corresponding update weight parameters based on the gradient update vectors uploaded from multiple display cases, thereby continuing to train the second item category detection model on each display case. When the trained second item category detection model on the display case converges, it can be considered that the converged second item category detection model on the display case can accurately identify the item information in the target item region image input to the second item category detection model. The converged second item category detection model on the display case can be stored as the target second item category detection model, that is, a model with high accuracy in identifying items in the target item region image.

[0191] In the above technical solution, on the one hand, the final target second item category detection model can be a model that learns both common rules and private rules, and its accuracy in identifying items in the target item region image is high; on the other hand, since the process of continuing to train the second item category detection model on each display case is jointly performed by the display case and the third edge server, compared with the display case or server alone further training the second item category detection model, less processing resources are required and the training speed is faster.

[0192] In one implementation of this disclosure, the cabinet door image is any video frame in the cabinet door video, the multiple cabinet body images are multiple video frames in the cabinet body video, the cabinet door video is acquired by the cabinet door image acquisition device in response to the cabinet door unlocking to start recording and in response to the cabinet door locking to end recording, and the cabinet body video is acquired by the cabinet image acquisition device in response to the cabinet door unlocking to start recording and in response to the cabinet door locking to end recording.

[0193] In one embodiment of this disclosure, cabinet door unlocking can be understood as the cabinet door locking device on the display cabinet responding to an unlocking command to unlock the cabinet door, allowing the cabinet door to rotate or slide relative to the cabinet body of the display cabinet to open the item entrance / exit of the display cabinet; it can also be understood as the cabinet door locking device on the display cabinet being triggered by a corresponding unlocking operation, causing the cabinet door locking device itself to be set to an unlocked state, thereby allowing the cabinet door to rotate or slide relative to the cabinet body of the display cabinet to open the item entrance / exit of the display cabinet, through which users can move items out of the item display area or move items into the item display area.

[0194] In one embodiment of this disclosure, determining whether a cabinet door is locked can be achieved by receiving cabinet door locking status information sent by a cabinet door locking device on the display cabinet, and determining whether the cabinet door is locked based on the cabinet door locking status information. Alternatively, cabinet door locking status information sent by other devices or systems can be received, and the cabinet door can be determined based on the cabinet door locking status information. For example, an infrared detection device independent of the display cabinet can detect whether the cabinet door is locked, and send cabinet door locking status information to the display cabinet based on the detection result; alternatively, a camera independent of the display cabinet, such as a security camera, can acquire images including all or part of the display cabinet, perform image recognition based on the images, and send cabinet door locking status information to the display cabinet based on the image recognition.

[0195] In this embodiment, considering that items can only be moved out of or into the display case when the display case door is unlocked, by limiting the door image to any video frame in the door video, and the multiple cabinet images to multiple video frames in the cabinet video, the door video is acquired by the door image acquisition device in response to the door unlocking and in response to the door locking ... locking and in response to the door unlocking and in response to the door locking and in response to the door locking and in response to the door unlocking and in response to the

[0196] In one implementation of this disclosure, the first item recognition result is used to indicate at least one item category of the item being picked up by hand in the cabinet door image, and the second item recognition result is used to indicate the category probability of at least two item categories corresponding to the item in the target item area image;

[0197] Matching is performed based on the first item recognition result and the second item recognition result to obtain the matched item recognition result, including:

[0198] In response to the fact that at least one item category indicated by the first item identification result includes the target item category, and the target item category has the highest category probability among the category probabilities indicated by the second item identification result, a matching item identification result for indicating the target item category is obtained.

[0199] In this embodiment, by obtaining a matching item recognition result for indicating the target item category in response to at least one item category indicated by the first item recognition result including the target item category, and the target item category having the highest category probability among the category probabilities indicated by the second item recognition result, the accuracy of the obtained matching item recognition result can be improved.

[0200] Figure 17 A flowchart illustrating the overall process of a display case control method according to an embodiment of the present disclosure is shown, as follows: Figure 12 As shown, the display case control methods include:

[0201] In step S201, the cabinet door image acquired by the cabinet door image acquisition device and multiple cabinet body images acquired by the cabinet body image acquisition device are obtained.

[0202] In step S202, the included angle between the cabinet door and the cabinet body is obtained when acquiring the cabinet door image.

[0203] In step S203, a pre-trained hand detection model corresponding to the angle of the cabinet door is obtained, and the cabinet door image is input into the hand detection model to obtain the hand detection information output by the hand detection model, and the retrieval area image is obtained based on the hand detection information.

[0204] In step S204, a pre-trained first item category detection model is obtained, and the image of the captured region is input into the first item category detection model to obtain the first item recognition result output by the first item category detection model.

[0205] In step S205, the target movement trajectory of the target item is obtained, and the target item area image is cropped from the third target cabinet image according to the target movement trajectory.

[0206] In step S206, a pre-trained second item category detection model is obtained, and the target item region image is input into the second item category detection model to obtain the second item recognition result output by the second item category detection model.

[0207] In step S207, in response to the fact that at least one item category indicated by the first item identification result includes the target item category, and the target item category has the highest category probability among the category probabilities indicated by the second item identification result, a matching item identification result for indicating the target item category is obtained.

[0208] In step S208, the scanning information collected by the scanning device is obtained, and the item being scanned is determined based on the scanning information.

[0209] In step S209, in response to the mismatch between the first item recognition result and the scanned item, at least one of the hand annotation information and the first annotation information of the scanned item is obtained.

[0210] In step S210, the first update weight parameter sent by the first edge server is received, and the hand detection model is updated according to the first update weight parameter.

[0211] In step S211, the cabinet door image is used as input and the hand annotation information is used as output to train the updated hand detection model.

[0212] In step S212, in response to the failure of the trained hand detection model to converge, the first gradient update vector is obtained based on the trained hand detection model and sent to the first edge server.

[0213] In step S213, in response to the convergence of the trained hand detection model, the trained hand detection model is stored as the target hand detection model.

[0214] In step S214, the first area image of the scanned item is cropped from the cabinet door image according to the first label information of the scanned item.

[0215] In step S215, the second update weight parameter sent by the second edge server is received, and the first item category detection model is updated according to the second update weight parameter.

[0216] In step S216, the first area image of the scanned item is used as input, and the item information of the scanned item is used as output to train the updated first item category detection model.

[0217] In step S217, in response to the failure of the trained first item category detection model to converge, a second gradient update vector is obtained based on the trained first item category detection model and sent to the second edge server.

[0218] In step S218, in response to the convergence of the trained first item category detection model, the trained first item category detection model is stored as the target first item category detection model.

[0219] In step S219, in response to the mismatch between the second item identification result and the scanned item, the second labeling information of the scanned item is obtained.

[0220] In step S220, the movement trajectory of the scanned item is obtained according to the second labeling information of the scanned item, and the second area image of the scanned item is cropped from any one of the multiple cabinet images according to the movement trajectory of the scanned item.

[0221] In step S221, the third update weight parameter sent by the third edge server is received, and the second item category detection model is updated according to the third update weight parameter.

[0222] In step S222, the second area image of the item to be scanned is used as input, and the item recognition result corresponding to the item to be scanned is used as output to train the second item category detection model.

[0223] In step S223, in response to the failure of the trained second item category detection model to converge, the third gradient update vector is obtained based on the trained second item category detection model and sent to the third edge server.

[0224] In step S224, in response to the convergence of the trained second item category detection model, the trained second item category detection model is stored as the target second item category detection model.

[0225] This disclosure also discloses an electronic device. Figure 18 A schematic structural block diagram of an electronic device according to an embodiment of the present disclosure is shown, such as... Figure 18 As shown, the electronic device 600 includes a memory 601 and a processor 602; wherein the memory 601 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 602 to implement the above method steps.

[0226] Figure 19 This is a schematic diagram of the structure of a computer system suitable for implementing the display case control method according to an embodiment of the present disclosure. For example... Figure 19As shown, the computer system 700 includes a processing unit 701, which can execute various processes described above based on a program stored in a read-only memory (ROM) 702 or a program loaded from a storage section 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0227] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed. The processing unit 701 can be implemented as a CPU, GPU, TPU, FPGA, NPU, etc.

[0228] In particular, according to embodiments of this disclosure, the methods described above with reference to the accompanying drawings can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a readable medium thereof, the computer program containing program code for performing the methods in the drawings. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. For example, embodiments of this disclosure include a readable storage medium storing computer instructions that, when executed by a processor, implement program code for performing the methods in the drawings.

[0229] In another aspect, this disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the apparatus described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs that are used by one or more processors to perform the methods described in this disclosure.

[0230] In addition, this disclosure also provides a computer program product storing a computer program that, when executed by a processor, enables the processor to at least implement the methods provided in the foregoing embodiments.

[0231] Figure 20 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown. Figure 21 A schematic top view of a display case according to one embodiment of the present disclosure is shown. Figure 20 as well as Figure 21 As shown, the display case 800 includes a cabinet body 801, a cabinet door 802, a first cabinet body image acquisition device 803, a second cabinet body image acquisition device 804, a cabinet door image acquisition device 805, and a processing device.

[0232] The cabinet door 802 is rotatably connected to the cabinet body 801 and is used to open or close the item access entrance 811 of the cabinet body 801;

[0233] The cabinet 801 includes an inner cavity 821, which is connected to the outside of the cabinet 801 through an item inlet / outlet 811. The inner cavity 821 is used to store items.

[0234] The first cabinet image acquisition device 803 and the second cabinet image acquisition device 804 are both connected to the top surface of the inner cavity 821 of the cabinet. The first cabinet image acquisition device 803 and the second cabinet image acquisition device 804 are used to acquire images of the item entrance 811 from different directions.

[0235] The cabinet door image acquisition device 805 is connected to the side of the cabinet door 802 closest to the cabinet body 801, and the position of the cabinet door image acquisition device 805 matches the position of the door handle 812 of the cabinet door 802. The door handle 812 is connected to the side of the cabinet door 802 furthest from the cabinet body 801.

[0236] The processing device is communicatively connected to the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device. The processing device is used to execute the method provided in the foregoing embodiments.

[0237] In one embodiment of this disclosure, collecting images of the item entry / exit points can be understood as collecting images of all or part of the item entry / exit points. Through the collected images, it is possible to determine whether the user moves items out of the cabinet cavity from the item entry / exit points or whether the user moves items into the cabinet cavity from the item entry / exit points.

[0238] In one embodiment of this disclosure, the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device can be cameras or other devices with image acquisition functions.

[0239] In one embodiment of this disclosure, the processing apparatus may include one or more processing units, such as an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a memory, a video codec, a digital signal processor, a baseband processor, and / or a neural network processor. The different processing units may be independent devices or integrated into one or more processors.

[0240] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A display case control method, characterized in that, The display cabinet includes a cabinet body, a cabinet door, a first cabinet body image acquisition device, a second cabinet body image acquisition device, and a cabinet door image acquisition device; the first cabinet body image acquisition device and the second cabinet body image acquisition device are both connected to the top surface of the cabinet body cavity, and are used to acquire images of the item entrance / exit from different directions; the cabinet door image acquisition device is connected to the side of the cabinet door closest to the cabinet body, and the position of the cabinet door image acquisition device matches the position of the door handle, the door handle being connected to the side of the cabinet door furthest from the cabinet body; the method includes: Acquire cabinet door images captured by the cabinet door image acquisition device, as well as multiple cabinet images captured by the first cabinet body image acquisition device and the second cabinet body image acquisition device; Obtain the image of the area corresponding to the hand in the cabinet door image, and perform item recognition on the image of the area to obtain a first item recognition result; The step of acquiring the image of the area corresponding to the hand in the cabinet door image, and performing item recognition on the image of the area to obtain a first item recognition result, includes: A pre-trained hand detection model is obtained, and the cabinet door image is input into the hand detection model to obtain hand detection information output by the hand detection model. The hand detection information is used to indicate the position of the hand in the cabinet door image. The image of the grasping area is obtained based on the hand detection information; A pre-trained first item category detection model is obtained, and the image of the picking area is input into the first item category detection model to obtain the first item recognition result output by the first item category detection model; The target movement trajectory of the target item is obtained, and the target item area image is cropped from the third target cabinet image according to the target movement trajectory. The target item is an item whose position in the first target cabinet image is different from its position in the second target cabinet image. The first target cabinet image, the second target cabinet image, and the third target cabinet image all belong to the multiple cabinet images. The acquisition time of the third target cabinet image is between the acquisition time of the first target cabinet image and the acquisition time of the second target cabinet image. Item recognition is performed based on the target item region image to obtain a second item recognition result; Match the first item recognition result and the second item recognition result to obtain the matching item recognition result; The method further includes: The scanning device acquires the scanning information collected by the scanning device and determines the item to be scanned based on the scanning information. The scanning device is installed on the cabinet door or cabinet body. In response to a mismatch between the first item recognition result and the scanned item, at least one of hand annotation information and first annotation information of the scanned item is obtained. The hand annotation information is used to indicate the position of the hand corresponding to the scanned item in the cabinet door image, and the first annotation information of the scanned item is used to indicate the scanned item in the cabinet door image. Receive the first update weight parameter sent by the first edge server, and update the hand detection model according to the first update weight parameter; The cabinet door image is used as input, and the hand annotation information is used as output to train the updated hand detection model; the first region image of the scanned item is extracted from the cabinet door image according to the first annotation information of the scanned item, and the first region image of the scanned item is used as input, and the item information of the scanned item is used as output to train the first item category detection model. If the trained hand detection model fails to converge, a first gradient update vector is obtained based on the trained hand detection model and sent to the first edge server; or, if the trained hand detection model converges, the trained hand detection model is stored as the target hand detection model.

2. The display case control method according to claim 1, characterized in that, Before training the first item category detection model by taking the first region image of the scanned item as input and the item information of the scanned item as output, the method further includes: Receive the second update weight parameters sent by the second edge server, and update the first item category detection model according to the second update weight parameters; The step of training the first item category detection model by taking the first region image of the scanned item as input and the item information of the scanned item as output includes: The first area image of the scanned item is used as input, and the item information of the scanned item is used as output to train the updated first item category detection model. The method further includes: In response to the failure of the trained first item category detection model to converge, a second gradient update vector is obtained based on the trained first item category detection model and sent to the second edge server; Alternatively, in response to the convergence of the trained first item category detection model, the trained first item category detection model is stored as the target first item category detection model.

3. The display case control method according to claim 1, characterized in that, Before obtaining the pre-trained hand detection model, the method further includes: The included angle between the cabinet door and the cabinet body when acquiring the cabinet door image; The process of obtaining a pre-trained hand detection model includes: Obtain a pre-trained hand detection model corresponding to the included angle of the cabinet door.

4. The display case control method according to claim 1, characterized in that, The step of identifying the object based on the target object region image includes: A pre-trained second item category detection model is obtained, and the target item region image is input into the second item category detection model to obtain the second item recognition result output by the second item category detection model; The method further includes: The scanning device acquires the scanning information collected by the scanning device and determines the item to be scanned based on the scanning information. The scanning device is installed on the cabinet door or cabinet body. In response to the second item identification result not matching the scanned item, second annotation information of the scanned item is obtained, and the second annotation information of the scanned item is used to indicate the scanned item in the multiple cabinet images; The scanning item's movement trajectory is obtained based on the second labeling information of the scanned item, and the second area image of the scanned item is extracted from any one of the multiple cabinet images based on the scanning item's movement trajectory. The second item category detection model is trained by taking the second region image of the scanned item as input and the item recognition result corresponding to the scanned item as output.

5. The display case control method according to claim 4, characterized in that, Before training the second item category detection model by taking the second region image of the scanned item as input and the item recognition result corresponding to the scanned item as output, the method further includes: Receive the third update weight parameter sent by the third edge server, and update the second item category detection model according to the third update weight parameter; The second item category detection model is trained by taking the second region image of the scanned item as input and the item recognition result corresponding to the scanned item as output, including: The second region image of the scanned item is used as input, and the item recognition result corresponding to the scanned item is used as output to train the updated second item category detection model. The method further includes: In response to the failure of the trained second item category detection model to converge, a third gradient update vector is obtained based on the trained second item category detection model and sent to the third edge server; Alternatively, in response to the convergence of the trained second item category detection model, the trained second item category detection model is stored as the target second item category detection model.

6. The display case control method according to any one of claims 1-5, characterized in that, The cabinet door image is any video frame in the cabinet door video, and the multiple cabinet body images are multiple video frames in the cabinet body video. The cabinet door video is acquired by the cabinet door image acquisition device in response to the cabinet door unlocking and in response to the cabinet door locking, and the cabinet body video is acquired by the cabinet image acquisition device in response to the cabinet door unlocking and in response to the cabinet door locking, and in response to the cabinet door locking, the recording starts and ends.

7. The display case control method according to any one of claims 1-5, characterized in that, The first item recognition result is used to indicate at least one item category of the item being picked up by hand in the cabinet door image, and the second item recognition result is used to indicate the category probability of at least two item categories corresponding to the item in the target item area image; The step of matching based on the first item identification result and the second item identification result to obtain a matched item identification result includes: In response to the fact that at least one item category indicated by the first item recognition result includes the target item category, and that the target item category has the highest category probability among the category probabilities indicated by the second item recognition result, a matching item recognition result for indicating the target item category is obtained.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method of any one of claims 1-7.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by a processor, the computer instructions implement the method of any one of claims 1-7.

10. A display case, characterized in that, The display case includes a cabinet body, cabinet doors, a first cabinet body image acquisition device, a second cabinet body image acquisition device, a cabinet door image acquisition device, and a processing device. The cabinet door is rotatably connected to the cabinet body and is used to open or close the item entrance / exit of the cabinet body. The cabinet includes an internal cavity, which is connected to the outside of the cabinet through the item inlet / outlet, and is used to store items. Both the first cabinet image acquisition device and the second cabinet image acquisition device are connected to the top surface of the inner cavity of the cabinet. The first cabinet image acquisition device and the second cabinet image acquisition device are used to acquire images of the item entrance and exit from different directions respectively. The cabinet door image acquisition device is connected to the side of the cabinet door closest to the cabinet body, and the position of the cabinet door image acquisition device matches the position of the door handle of the cabinet door. The door handle is connected to the side of the cabinet door furthest from the cabinet body. The processing device is communicatively connected to the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device, and the processing device is used to execute the method of any one of claims 1-7.

Images