Display cabinet control method, apparatus, medium, and display cabinet

Abstract

The display cabinet control method, device, medium and display cabinet disclosed by the embodiments of the present disclosure comprise: acquiring a to-be-detected image; acquiring object position information of at least one object in the to-be-detected image and body node information of at least one user in the to-be-detected image; and in response to determining that the to-be-detected image comprises at least one alarm body node according to the body key node information and that the to-be-detected image comprises at least one alarm target object according to the object position information, generating alarm information. This technical solution can generate alarm information when a user may make an action of using a target object (i.e. an object stored in a display cabinet) to achieve the purpose of prompting, avoid the user from arbitrarily using the object in the display cabinet, ensure that an object that has been used is not placed in the display cabinet after being locked in the display cabinet, avoid other users from removing the object used by the user from the display cabinet, and improve the user experience.

Description

Technical Field

[0001] This disclosure relates to the field of control technology, specifically to a display cabinet control method, equipment, medium, and display cabinet. Background Technology

[0002] In recent years, to facilitate user access to product information, merchants and businesses have often placed items in display cases, simultaneously storing and showcasing goods. When users need to remove or add items, they can open the display case themselves. In this scenario, the display case can capture images of the items inside and upload the data. Other devices or systems, such as servers or cloud platforms, can then use these images to identify users or items removed from the display case for statistical purposes. Summary of the Invention

[0003] This disclosure provides a display cabinet control method, device, medium, and display cabinet to solve the problem in related technologies where it is impossible to accurately determine the items moved out or into the display cabinet based on image recognition results.

[0004] Firstly, this disclosure provides a display cabinet control method.

[0005] Specifically, display case control methods include:

[0006] Acquire the image to be detected;

[0007] Obtain the item location information of at least one item in the image to be detected and the body node information of at least one user in the image to be detected. The item location information is used to indicate the location of the corresponding item, and the body node information is used to indicate the location of at least one body node of the corresponding user.

[0008] In response to determining that the image to be detected includes at least one alarm body node based on the body key node information, and determining that the image to be detected includes at least one alarm target item based on the item location information, an alarm message is generated, wherein the alarm target item is a target item whose location matches the location corresponding to the alarm node.

[0009] In a second aspect, embodiments of this disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method as described in the first aspect or any implementation thereof.

[0010] Thirdly, this disclosure provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the method as described in the first aspect or any implementation thereof.

[0011] Fourthly, this disclosure provides a computer program product including computer instructions that, when executed by a processor, implement the method as described in the first aspect or any implementation thereof.

[0012] Fifthly, this disclosure provides a display cabinet, which includes a cabinet body, cabinet doors, a first cabinet body image acquisition device, a second cabinet body image acquisition device, a cabinet door image acquisition device, and a processing device.

[0013] The cabinet door is rotatably connected to the cabinet body and is used to open or close the cabinet's entrance / exit for items.

[0014] The cabinet includes an internal cavity, which is connected to the outside of the cabinet through an item inlet / outlet. The internal cavity is used to store items.

[0015] Both the first cabinet image acquisition device and the second cabinet image acquisition device are connected to the top surface of the cabinet's internal cavity. The first cabinet image acquisition device and the second cabinet image acquisition device are used to acquire images of the item's entrance and exit from different directions.

[0016] The cabinet door image acquisition device is connected to the side of the cabinet door closest to the cabinet body, and the position of the cabinet door image acquisition device matches the position of the cabinet door handle, while the door handle is connected to the side of the cabinet door furthest from the cabinet body.

[0017] The processing device is communicatively connected to the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device. The processing device is used to execute the method described in the first aspect or any implementation thereof.

[0018] The technical solutions provided in this disclosure may have the following beneficial effects:

[0019] The technical solution provided in this disclosure acquires an image to be detected, and acquires the item location information of at least one item in the image and the body node information of at least one user in the image. In response to determining that the image to be detected includes at least one alarm body node based on the body node information, and determining that the image to be detected includes at least one alarm target item based on the item location information, alarm information is generated. The alarm target item is a target item whose location matches the location corresponding to the alarm node. Wherein, when the body node of at least one user in the image to be detected includes at least one alarm body node, it can be understood that the user may be performing an action related to using the item. And when the location of at least one target item in the image to be detected matches the location corresponding to the alarm node, it can be understood that the user may be using the item by performing an action related to using the item. For example, an alarm body node can be understood as a hand node that is constantly approaching a mouth node and is within a certain distance of the mouth node. If the position of at least one target item matches the position corresponding to the hand node, the user may move the item they are holding towards their mouth, preparing to drink, eat, or bite the item. Alternatively, an alarm body node can be understood as another hand node that is constantly approaching a hand node and is within a certain distance of that hand node. If the position of at least one target item matches the position corresponding to that hand node, the user may move the item held in one hand towards the other hand, preparing to tear the item with both hands. Therefore, the above solution can generate alarm information to prompt users when they are likely to use the target item (i.e., the item stored in the display case), thereby preventing users from using the items in the display case without authorization. This also prevents used items from remaining in the display case after it has been locked, and prevents other users from removing items used by the user, thus improving the user experience.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

[0022] Figure 1 A schematic structural block diagram of a display cabinet according to an embodiment of the present disclosure is shown.

[0023] Figure 2 A schematic structural block diagram of a motherboard according to an embodiment of the present disclosure is shown.

[0024] Figure 3 A schematic structural block diagram of a control panel according to an embodiment of the present disclosure is shown.

[0025] Figure 4 A schematic structural block diagram of a power management module according to an embodiment of the present disclosure is shown.

[0026] Figure 5 A flowchart illustrating a display case control method according to an embodiment of the present disclosure is shown.

[0027] Figure 6 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown.

[0028] Figure 7 A schematic top view of a display case according to one embodiment of the present disclosure is shown.

[0029] Figure 8 A schematic diagram of an image to be detected according to an embodiment of the present disclosure is shown.

[0030] Figure 9 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0031] Figure 10 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0032] Figure 11 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0033] Figure 12 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0034] Figure 13 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0035] Figure 14 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown.

[0036] Figure 15 A flowchart illustrating the overall process of a display case control method according to an embodiment of the present disclosure is shown.

[0037] Figure 16 A schematic structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0038] Figure 17 This is a schematic diagram of the structure of a computer system suitable for implementing a display case control method according to an embodiment of the present disclosure.

[0039] Figure 18 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown.

[0040] Figure 19 A schematic top view of a display case according to one embodiment of the present disclosure is shown. Detailed Implementation

[0041] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement them. Furthermore, for clarity, portions unrelated to the description of the exemplary embodiments have been omitted from the drawings.

[0042] In this disclosure, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, figures, steps, behaviors, components, parts or combinations thereof disclosed in this specification, and do not preclude the possibility of the presence or addition of one or more other features, figures, steps, behaviors, components, parts or combinations thereof.

[0043] It should also be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0044] As mentioned above, with the development of technology and the improvement of people's living standards, merchants or enterprises no longer simply place goods on shelves. Instead, to facilitate users' understanding of product information, they can place items in display cases, thus simultaneously storing and displaying the goods. When users need to remove items from the display case or put items into it, they can open the display case themselves and perform the corresponding operations.

[0045] In recent years, the number of display cases put into operation has gradually increased. During the use of display cases, merchants or enterprises generally need to monitor the items in the display cases to identify the users who move items out of or into the display cases, as well as the items that are moved out of or into the display cases, and to make statistics based on the above information.

[0046] In one embodiment, after the display case is unlocked, at least one image can be captured by at least one camera installed on the display case. After the display case is locked, image recognition is performed on the image. Based on the image recognition result, it can be determined whether the items in the display case have been removed by the user or whether the user has moved items into the display case when the display case is in the unlocked state. The statistical result is then uploaded so that other devices or systems, such as servers or the cloud, can know the quantity and type of the remaining items in the display case based on the statistical result. In addition, further settlement can be performed based on the statistical result.

[0047] While the above solution can identify items removed from or moved into the display case for appropriate settlement or statistics, it raises concerns that some users may use items in the display case without authorization after unlocking it (e.g., opening and drinking bottled beverages without permission), leaving the used items in the display case even after it is locked. This could lead to other users removing used items from the display case after unlocking it, potentially harming users' health and damaging the user experience.

[0048] Therefore, how to accurately determine whether a user has used an item in the display case is an increasingly urgent problem that needs to be solved.

[0049] In view of the above-mentioned defects, a display cabinet control method is proposed in one embodiment of this disclosure.

[0050] The display cabinet control method provided in this application embodiment can be applied to display cabinets, which can have a temperature control function. The temperature control function can be a cooling function, such as a refrigerated display cabinet, a frozen display cabinet, a refrigerator, a wine cabinet, a cosmetic preservation cabinet, etc.; the temperature control function can also be a heating function, such as a warming cabinet, a heated display cabinet, a hot beverage cabinet, etc. This application embodiment does not limit the specific type of display cabinet.

[0051] For example, Figure 1 A schematic structural block diagram of a display cabinet according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the display case 100 may include a compressor 11, a condenser 12, a throttling element 13, and an evaporator 14. The compressor 11, condenser 12, throttling element 13, and evaporator 14 are connected by pipes filled with refrigerant to form a closed pipeline, which constitutes a refrigeration system or heating system capable of circulating refrigerant.

[0052] In one embodiment of this application, the display cabinet includes a cabinet body and a cabinet door, wherein a control board and a power management module may be installed in the cabinet body, and a main board may be installed in the cabinet door.

[0053] In one embodiment of this application, Figure 2 A schematic structural block diagram of a motherboard according to an embodiment of the present disclosure is shown, such as... Figure 2 As shown, the motherboard 200 includes a processor 201, random access memory 202, flash memory 203, wireless LAN Bluetooth module 204, gyroscope 205, pressure sensor 206, microphone 207, speaker 208, camera 209, and cellular communication module 210.

[0054] In one embodiment of this application, Figure 3A schematic structural block diagram of a control panel according to an embodiment of the present disclosure is shown, such as... Figure 3 As shown, the control board 300 includes a power input interface 301, a power output interface 302, a metering chip 303, a microcontroller chip 304, a real-time clock chip, a light switch interface 305, a temperature control switch interface 306, an evaporator fan interface 307, a compressor interface 308, a condenser fan interface 309, a temperature sensor interface 310, a communication interface 311, and a power interface 312.

[0055] In one embodiment of this application, Figure 4 A schematic structural block diagram of a power management module according to an embodiment of the present disclosure is shown, such as... Figure 4 As shown, the power management module 400 includes an AC-to-DC conversion module 401, a charging management module 402, and a battery 403. The power management module 400 supplies power to the motherboard and control board and manages the charging and discharging of the battery. The power management module 400 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 400 may also be located within the processor.

[0056] In one embodiment of this application, the display case further includes a display screen. The display case implements its display function through a graphics processor, a display screen, and an application processor. The graphics processor is a microprocessor for image processing, connected to the display screen and the application processor. The graphics processor is used to perform mathematical and geometric calculations and for graphics rendering. The processor may include one or more graphics processors that execute program instructions to generate or modify display information.

[0057] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the display cabinet. In other embodiments of this application, the display cabinet may include more or fewer components than illustrated, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, by combining different components, the display cabinet in the embodiments of this application may be any of the following: a retail cabinet, a heated cabinet, a refrigerated cabinet, a freezer, a combination cabinet, or a display case.

[0058] Figure 5 A flowchart illustrating a display case control method according to an embodiment of the present disclosure is shown, such as... Figure 5 As shown, the display case control method includes the following steps S101-S103:

[0059] In step S101, the image to be detected is acquired.

[0060] In one embodiment of this disclosure, acquiring the image to be detected can be achieved by receiving an image from the display case's image acquisition device, reading an image pre-stored in the display case, or receiving an image from another device or system. The image to be detected may include one or more still images, or one or more segments of moving images or videos. The image to be detected can be understood as including images of all or part of the users using the display case. Based on the image to be detected, the actions performed by the user can be determined.

[0061] In one embodiment of this disclosure, the image acquisition device for the display case can be installed at the top or bottom of the display case door, on the side wall of the door near the cabinet body, on the side wall of the door away from the cabinet body, or on the door handle; alternatively, the image acquisition device can be connected to the top, side, or bottom surface of the inner wall of the cabinet body, or it can be connected to the top, side, or bottom surface of the outer wall of the cabinet body. This embodiment of the disclosure does not impose specific limitations on the exact location of the image acquisition device for the display case.

[0062] For example, Figure 6 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown. Figure 7 A schematic top view of a display case according to one embodiment of the present disclosure is shown, such as... Figure 6 as well as Figure 7 As shown, the display case includes a cabinet body 501, a cabinet door 502, and an image acquisition device. The image acquisition device includes a cabinet door image acquisition device 503 and a cabinet body image acquisition device 504. The cabinet door image acquisition device 503 is connected to the side wall of the cabinet door 502 near the cabinet body 501, and the position of the cabinet door image acquisition device 503 matches the position of the cabinet door handle 512 on the cabinet door 502. The cabinet body image acquisition device 504 is connected to the top surface of the inner wall of the cabinet body 501.

[0063] The cabinet 501 includes a display area 511 and an item entrance / exit 521. The display area 511 is connected to the outside of the cabinet 501 through the item entrance / exit 521. The display area 511 is used to accommodate items stored in the display cabinet. Items in the display area 511 can be moved out of the display cabinet through the item entrance / exit 521, or items can be moved into the display area 511 through the item entrance / exit 521.

[0064] The cabinet doors can be rotatably connected to the cabinet body, or they can be slidably connected to the cabinet body. Alternatively, the cabinet doors can be connected to the cabinet body by folding. It should be noted that this application does not specifically limit the way the cabinet doors are connected to the cabinet body. For ease of understanding, this is just a general description. Figure 6 as well as Figure 7The following explanation uses the rotatable connection between cabinet door 502 and cabinet body 501 as an example. Cabinet door 502 is used to open or close the item access entrance 521.

[0065] The cabinet door image acquisition device 503 can be used to acquire images of at least one of the following: all display areas 511, some display areas 511, all item entrances / exits 521, and some item entrances / exits 521, in order to obtain images to be inspected. It should be noted that the display cabinet may include only one cabinet door image acquisition device 503, or it may include multiple cabinet door image acquisition devices 503 that acquire images to be inspected from different directions. This disclosure does not impose a specific limitation on the number of cabinet door image acquisition devices 503.

[0066] The cabinet image acquisition device 504 can be used to acquire images of at least one of the following: all display areas 511, some display areas 511, all item entrances / exits 521, and some item entrances / exits 521, in order to obtain images to be inspected. It should be noted that the display cabinet may include only one cabinet image acquisition device 504, or it may include multiple cabinet image acquisition devices 504 that acquire images to be inspected from different directions. This disclosure does not impose a specific limitation on the number of cabinet image acquisition devices 504.

[0067] In step S102, the item location information of at least one item in the image to be detected and the body node information of at least one user in the image to be detected are obtained.

[0068] Among them, the item location information is used to indicate the location of the corresponding item, and the body node information is used to indicate the location of at least one body node of the corresponding user.

[0069] In one embodiment of this disclosure, the corresponding item can be understood as an item stored in a display case. The corresponding item may include bottled beverages, packaged food, fruits, pastries, cooked food, etc. This application does not specifically limit the type of corresponding item.

[0070] In one embodiment of this disclosure, the position of a corresponding item can be understood as the relative position between the corresponding item and the display case. This relative position can be understood as the coordinates of the item in a relative coordinate system, which can be a coordinate system with any position on the display case as its pole; alternatively, the relative position can also be understood as the distance between the corresponding item and at least two positions on the display case. The position of a corresponding item can also be understood as the relative position between the corresponding item and at least one body node of the corresponding user. This relative position can be understood as the distance between the item and at least one body node of the user, or as the coordinates of the item in a relative coordinate system, which can be a coordinate system with any body node of the user as its pole. The position of a corresponding item can also be understood as the absolute position of the corresponding item, which can be understood as the coordinates of the corresponding item in a geodetic coordinate system.

[0071] In one embodiment of this disclosure, obtaining the item location information of at least one item in an image to be detected can be understood as performing item location information recognition on the image to be detected according to a pre-acquired algorithm to obtain the item location information of at least one item in the image to be detected; alternatively, the image to be detected can be sent, and a first item location information recognition result obtained by other devices or systems, such as a cloud server, through performing item location information recognition on the image to be detected can be received, and the item location information of at least one item in the image to be detected can be obtained according to the first item location information recognition result.

[0072] In one embodiment of this disclosure, a body node can be understood as a node used to indicate a user's body movements, and the position of this node may coincide with the position of joints that the user uses frequently. For example, Figure 8 A schematic diagram showing an image to be detected according to an embodiment of the present disclosure is provided, such as... Figure 8 As shown, in the image to be detected, the user's body nodes may include at least one of the following: hand node 601, wrist node 602, elbow node 603, shoulder node 604, neck node 605, head node 606, waist node 607, hip node 608, knee node 609, ankle node 610, and foot node 611.

[0073] In one embodiment of this disclosure, the position of a body node can be understood as the relative position between the body node and the display case. This relative position can be understood as the coordinates of the body node in a relative coordinate system, which can be a coordinate system with any position on the display case as its pole; or, the relative position can also be understood as the distance between the body node and at least two positions on the display case. The position of a body node can also be understood as the relative position between the body node and the corresponding item. This relative position can be understood as the distance between the body node and the corresponding item, or it can be understood as the coordinates of the body node in a relative coordinate system, which can be a coordinate system with any corresponding item as its pole. The position of a body node can also be understood as the absolute position of the body node, which can be understood as the coordinates of the body node in a geodetic coordinate system.

[0074] In one embodiment of this disclosure, obtaining the body node information of at least one user in the image to be detected can be understood as performing body node information recognition on the image to be detected according to a pre-acquired algorithm to obtain the body node information of at least one item in the image to be detected; alternatively, the image to be detected can be sent, and a first body node information recognition result obtained by other devices or systems, such as a cloud server, through the body node information recognition of the image to be detected can be received, and the body node information of at least one item in the image to be detected can be obtained according to the first body node information recognition result.

[0075] In step S103, in response to determining that the image to be detected includes at least one alarm body node based on the body key node information, and determining that the image to be detected includes at least one alarm target item based on the item location information, alarm information is generated.

[0076] Among them, the alarm target item is the target item whose location matches the location corresponding to the alarm node.

[0077] In one embodiment of this disclosure, the image to be detected includes at least one alarm body node. This can be understood as an action performed by a body part corresponding to at least one of the user's body nodes in the image, an action that may involve the use of an object. This action may leave traces of use on the object or damage the object itself. For example, an alarm body node can be understood as a hand node that is constantly approaching a mouth node and is less than a certain distance from the mouth node. If the position of at least one target object matches the position corresponding to the hand node, the user may move the object they are holding towards their mouth, preparing to drink, eat, or bite the object, thereby damaging the object. Alternatively, an alarm body node can be understood as another hand node that is constantly approaching a hand node and is less than a certain distance from that hand node. If the position of at least one target object matches the position corresponding to that hand node, the user may move the object held in one hand towards the other hand, preparing to tear the object with both hands, thereby damaging the object.

[0078] In one embodiment of this disclosure, the target item can be understood as an item stored in a corresponding display case. Item recognition can be performed on the image to be detected, and the target item in the image can be determined based on the item recognition result. Item recognition on the image to be detected can be understood as performing item recognition on the image according to a pre-acquired algorithm, or it can be achieved by sending the image to be detected and receiving item recognition results obtained by other devices or systems, such as a cloud server, through image recognition of the image to be detected.

[0079] In one embodiment of this disclosure, a target item whose location matches the location corresponding to an alarm node can be understood as the target item's location coinciding with the alarm node's location, or the distance between the target item and the alarm node being less than or equal to an item-node distance threshold. A match between the location of at least one target item in the image to be detected and the location corresponding to an alarm node can be understood as a user moving, touching, or holding the target item using a body part corresponding to the alarm node.

[0080] In one embodiment of this disclosure, the alarm information can be understood as a warning to the user that they may use the corresponding item stored in the display case. The alarm information can be displayed through human-computer interaction devices on the display case, such as a display screen, speaker, vibration device, etc., or it can be sent to other devices or systems, such as a smart wireless communication terminal paired with the display case, so that other devices or systems can display the alarm information.

[0081] For example, consider a display case used to provide unmanned vending services. When a customer wants to purchase an item from the display case, they can use a mobile communication terminal to scan the QR code on the surface of the display case to access the corresponding cloud authorization server and send a statistical authorization request to the cloud authorization server, i.e., requesting to purchase the item from the display case. After the cloud authorization server approves the statistical authorization request, it sends statistical authorization information to the display case. The display case receives the statistical authorization information sent by the cloud server and, in response, unlocks the display case door. After the door is unlocked, the customer can open the door and take the item from the display case through the item entrance / exit. After taking the item, the customer can close the door. From the time the door is opened until it is closed, the door image acquisition device at the display case door continuously acquires images of the item entrance / exit (i.e., the door image), while the cabinet image acquisition device at the display case body continuously acquires images of the item entrance / exit and the display area (i.e., the cabinet body image). Both the door image and the cabinet body image are images to be detected. The display case can acquire the location information of at least one item in the image to be detected and the body node information of at least one customer in the image to be detected. When the body node information determines that the body node of at least one customer in the image to be detected includes at least one alarm body node, and the location information of at least one target item in the image to be detected matches the location of the alarm node, i.e., the image to be detected includes at least one alarm target item, it can be considered that the customer may have used the item stored in the display case. Since the customer can put the used item back in the display case at any time before the cabinet door is closed, other customers may take the used item out of the display case. To avoid this situation, an alarm message can be generated and displayed through any one of the following alarm devices on the display case: speaker, display screen, vibration device, and other alarm devices. This prevents the customer from continuing to use the item stored in the display case, thereby ensuring that other customers do not remove the used item from the display case and improving the user experience.

[0082] For example, consider a display case used to provide item retrieval services to a target user, where the target user can be understood as a user belonging to a specific unit or department. When a user needs to purchase an item from the display case, they can use a mobile communication terminal to scan the QR code on the surface of the display case to access the server corresponding to the display case and send an identity authentication request to the server, i.e., requesting authentication of the user's identity; alternatively, they can use a corresponding identification (such as an employee badge, ID card, etc.) to scan the QR code on the display case, which will then send an identity authentication request to the corresponding server. When the display case receives item retrieval authorization information from the server, it can be understood that the server has confirmed the user as the target user, and the display case can unlock its door in response to the item retrieval authorization information. After the door is unlocked, the user can open the door and retrieve the item from the display case through the item access window. After retrieving the item, the user can close the door. From the moment the cabinet door is opened until it is closed, the cabinet door image acquisition device at the cabinet door can continuously acquire images of the item entrance and exit, i.e., cabinet door images. At the same time, the cabinet body image acquisition device at the cabinet body can continuously acquire images of the item entrance and exit and the display area, i.e., cabinet body images. Both the cabinet door images and the cabinet body images are images to be detected. The display case can acquire the location information of at least one item in the image to be detected, as well as the body node information of at least one user in the image to be detected. When the body node information determines that the body node of at least one user in the image to be detected includes at least one alarm body node, and the location information of at least one target item in the image to be detected matches the location of the alarm node, i.e., the image to be detected includes at least one alarm target item, it can be considered that the user may have used the item stored in the display case. Since the user can put the used item back into the display case at any time before the cabinet door is closed, other users may take the used item out of the display case. To avoid this situation, an alarm message can be generated and displayed through any one of the following alarm devices on the display case: speaker, display screen, vibration device, and other alarm devices. This prevents the user from continuing to use the item stored in the display case, thus ensuring that other users do not remove the used item from the display case and improving the user experience.

[0083] In the above technical solution, by acquiring an image to be detected, and acquiring the item location information of at least one item in the image and the body node information of at least one user in the image, an alarm message is generated in response to determining, based on the body node information, that the body nodes of at least one user in the image include at least one alarm body node, and based on the item location information, determining that the position of at least one target item in the image matches the position corresponding to the alarm node. Specifically, when the body nodes of at least one user in the image include at least one alarm body node, it can be understood that the user may be performing an action related to using the item, and the matching of the position of at least one target item in the image with the position corresponding to the alarm node can be understood as the user possibly using the item by performing an action related to using the item. For example, an alarm body node can be understood as a hand node that is constantly approaching a mouth node and is within a certain distance of the mouth node. If the position of at least one target item matches the position corresponding to the hand node, the user may move the item they are holding towards their mouth, preparing to drink, eat, or bite the item. Alternatively, an alarm body node can be understood as another hand node that is constantly approaching a hand node and is within a certain distance of that hand node. If the position of at least one target item matches the position corresponding to that hand node, the user may move the item held in one hand towards the other hand, preparing to tear the item with both hands. Therefore, the above solution can generate alarm information to prompt users when they are likely to use the target item (i.e., the item stored in the display case), thereby preventing users from using the items in the display case without authorization. This also prevents used items from remaining in the display case after it has been locked, and prevents other users from removing items used by the user, thus improving the user experience.

[0084] In one implementation of this disclosure, in step S102, obtaining the item location information of at least one item in the image to be detected can be achieved through the following steps:

[0085] Obtain a pre-trained object location detection model and input the image to be detected into the object location detection model to obtain the object location information output by the object location detection model;

[0086] Obtain body node information for at least one user in the image to be detected, including:

[0087] Obtain a pre-trained body node detection model and input the image to be detected into the body node detection model to obtain the body node information output by the body node detection model.

[0088] In one embodiment of this disclosure, the item location detection model and the body node detection model can be pre-stored in a display case or obtained from other devices or systems. The item location detection model and the body node detection model can be a neural network (NN) model, a convolutional neural network (CNN) model, or a long short-term memory (LSTM) model, etc.

[0089] In the above technical solution, by acquiring a pre-trained object location detection model and inputting the image to be detected into the object location detection model to obtain the object location information output by the object location detection model, the accuracy of the acquired object location information can be ensured to be high. Similarly, by acquiring a pre-trained body node detection model and inputting the image to be detected into the body node detection model to obtain the body node information output by the body node detection model, the accuracy of the acquired body node information can be ensured to be high.

[0090] In one implementation of this disclosure, the method further includes the following steps:

[0091] In response to an alarm message, the system acquires the item position correction information input from the human-computer interaction device of the display case. In response to a mismatch between the item position correction information and the item position information, the system uses the image to be detected as input and the item position correction information as output to train the item position detection model.

[0092] And / or, in response to an alarm message, acquire body node correction information input from the human-computer interaction device of the display case; in response to a mismatch between the body node correction information and the body node information, use the image to be detected as input and the body node correction information as output to train the body node detection model.

[0093] In one implementation of this disclosure, the image to be detected can be displayed before the item position correction information input by the human-computer interaction device of the display case is retrieved.

[0094] In one embodiment of this disclosure, the image to be detected can be displayed through a human-computer interaction device on a display case, such as a display screen, or through other devices or systems.

[0095] In one embodiment of this disclosure, obtaining item position correction information input by the human-computer interaction device of the display case can be achieved by obtaining the user-inputted item position correction information through at least one of the keyboard, touch screen, and touchpad on the display case; it can also be achieved by obtaining the user-inputted voice information through the microphone on the display case and performing voice recognition on the voice information to obtain the item position correction information; or it can be achieved by obtaining image information of the user's actions through the camera on the display case and performing image recognition on the image information to obtain the item position correction information.

[0096] For example, the display case can display the image to be detected through a graphical user interface (GUI) on a touchscreen, and acquire the user's input actions (item position correction information input or body node correction information input) corresponding to the image to be detected displayed in the GUI. The display case can then acquire the user-inputted item position correction information or body node correction information based on the input actions. The GUI is the interactive interface of an application (APP) running on the display case. Further, the GUI can also display one or more visual prompts on the touchscreen that allow the user to perform the input actions (item position correction information input or body node correction information input). These visual prompts can be text, graphics, or any combination thereof. The input actions (item position correction information input or body node correction information input) may involve contact with the touchscreen. In some embodiments, in addition to visual prompts, the display case can also provide non-visual prompts to indicate the progress of the input actions (item position correction information input or body node correction information input). The non-visual prompts may include audio prompts (e.g., sound) or physical prompts (e.g., vibration). In some embodiments, the item position correction information input action or body node correction information input action is a predetermined gesture performed on the touchscreen. The gesture used here is the movement of an object / accessory in contact with the touchscreen. For example, the predetermined gesture may include contacting the touchscreen at the location of the hand in the image to be detected displayed on the touchscreen (initialization gesture), and breaking the contact after maintaining continuous contact with the touchscreen for more than a preset contact time threshold (completion of the gesture); the predetermined gesture may also include contacting the touchscreen at the location of the item being scanned in the image to be detected displayed on the touchscreen (initialization gesture), and breaking the contact after maintaining continuous contact with the touchscreen for more than a preset contact time threshold (completion of the gesture).

[0097] For ease of illustration, in the process of acquiring user-inputted object position correction information or body node correction information, and in other embodiments described below, touch on the touchscreen will be described as being performed by the user using at least one hand and one or more fingers. However, it should be understood that the touch can also be performed using any suitable object or accessory, such as a stylus, finger, etc. The touch may include: one or more taps on the touchscreen, maintaining continuous contact with the touchscreen, moving the contact point while maintaining continuous contact, interrupting the contact, or any combination thereof.

[0098] The display case detects touches on the touchscreen. If the touch does not correspond to an attempt to perform an item position correction information input action or a body node correction information input action, or if the touch corresponds to a failed or abandoned attempt by the user to perform such an action, then the display case will not acquire the corresponding information. For example, if the item position correction information input action involves contacting the touchscreen at the location of an item in the image displayed on the touchscreen, and the contact is interrupted after exceeding a preset contact time threshold, but the detected contact is a series of random taps on the touchscreen, then the touch does not correspond to an item position correction information input action. Similarly, if the body node correction information input action involves contacting the touchscreen at the location of a body node in the image displayed on the touchscreen, and the contact is interrupted after exceeding a preset contact time threshold, but the detected contact is a series of random taps on the touchscreen, then the touch does not correspond to a body node correction information input action.

[0099] If the contact corresponds to the successful execution of the item position correction information input action, that is, the user successfully executes the item position correction information input action, then the display cabinet can obtain the corresponding item position correction information based on the item position correction information input action.

[0100] If the contact corresponds to the successful execution of a body node correction information input action, that is, if the user successfully executes the body node correction information input action, then the display case can obtain the corresponding body node correction information based on the body node correction information input action.

[0101] Figure 9 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figure 10 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figure 11 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figures 9-11This diagram illustrates the graphical user interface (GUI) of the display case at different moments during the execution of the item position correction information input action.

[0102] exist Figure 9 In the middle, the user indicates with finger 61 that finger 61 has not yet touched the touch screen 62 of the display case, at which time the display case will not obtain the corresponding item position correction information.

[0103] exist Figure 10 In this process, the user initiates the input of item position correction information by touching the touchscreen 62 of the display case with their finger 61. Specifically, the user continuously touches the touchscreen 62 with their finger 61 and moves their finger 61 along the trajectory of the corresponding item 64 in the image to be detected displayed by the graphical user interface 63. The display case then begins to record the movement trajectory 65 of the finger 61. It should be noted that when the finger 61 is not removed from the touchscreen 62, the display case will not obtain the corresponding item position correction information based on the recorded movement trajectory.

[0104] exist Figure 11 In this process, the user ends the input of item position correction information by removing their finger 61 from the touch screen 62 of the display case. Specifically, in response to the finger 61 being out of contact with the touch screen 62 for more than a preset time difference, the display case determines the position of the corresponding item 64 in the image to be detected in the cabinet door image displayed in the graphical user interface GUI 63 based on the recorded movement trajectory, and obtains item position correction information to indicate the position of the corresponding item 64 in the image to be detected.

[0105] Figure 12 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figure 13 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figure 14 A graphical user interface (GUI) diagram of a display case according to an embodiment of the present disclosure is shown. Figures 12-14 The diagram shows the graphical user interface (GUI) of the display case at different moments during the execution of the body node correction information input action.

[0106] exist Figure 12 In the middle, the user indicates with finger 71 that finger 71 has not yet touched the touch screen 72 of the display case, at which time the display case will not obtain the corresponding body node correction information.

[0107] exist Figure 13In this system, the user initiates the input of body node correction information by touching the touchscreen 72 of the display case with their finger 71. Specifically, the user continuously touches the touchscreen 72 with their finger 71, moving the finger 71 along the trajectory of at least one body node 74 in the image to be detected displayed by the graphical user interface 73. The display case then begins recording the movement trajectory 75 of the finger 71. It should be noted that when the finger 71 is not removed from the touchscreen 72, the display case will not obtain the corresponding body node correction information based on the recorded movement trajectory.

[0108] exist Figure 14 In this process, the user ends the action of inputting body node correction information by removing their finger 71 from the touch screen 72 of the display case. Specifically, in response to the finger 71 leaving the touch screen 72 for more than a preset time difference, the display case determines the position of at least one body node 74 in the image to be detected in the cabinet door image displayed in the graphical user interface GUI 73 based on the recorded movement trajectory, and obtains body node correction information to indicate the position of at least one body node 74 in the image to be detected.

[0109] In the above technical solution, in response to alarm information, the item position correction information input by the human-computer interaction device of the display case is obtained. In response to a mismatch between the item position correction information and the item position information, the image to be detected is used as input and the item position correction information is used as output to train the item position detection model. This allows the trained item position detection model to learn the patterns between the item position and the image to be detected that it could not learn successfully before, ensuring a high success rate in recognizing the position of the corresponding item in the image to be detected based on the trained item position detection model. And / or, in response to alarm information, the body node correction information input by the human-computer interaction device of the display case is obtained. In response to a mismatch between the body node correction information and the body node information, the image to be detected is used as input and the body node correction information is used as output to train the body node detection model. This allows the body node detection model to learn the patterns between the position of the body nodes that it could not learn successfully before and the image to be detected, ensuring a high success rate in recognizing the position of the body nodes in the image to be detected based on the trained body node detection model.

[0110] In one implementation of this disclosure, in response to a mismatch between the item position correction information and the item position information, before training the item position detection model by taking the image to be detected as input and the item position correction information as output, the method further includes the following steps:

[0111] Receive the first update weight parameter sent by the first edge server, and update the item location detection model according to the first update weight parameter;

[0112] In response to a mismatch between the item position correction information and the item position information, the image to be detected is used as input, and the item position correction information is used as output to train the item position detection model, including:

[0113] In response to a mismatch between the item position correction information and the item position information, the image to be detected is used as input, and the item position correction information is used as output to train the updated item position detection model.

[0114] The method also includes the following steps:

[0115] In response to the failure of the trained object location detection model to converge, the first gradient update vector is obtained based on the trained object location detection model and sent to the first edge server.

[0116] Alternatively, in response to the convergence of the trained item location detection model, the trained item location detection model is stored as the target item location detection model.

[0117] In one implementation of this disclosure, a first edge server aggregates gradient update vectors and updates the weight parameters of the item location detection model on the first edge server based on the aggregated gradient update vectors to obtain updated weight parameters. The first edge server can be a cloud server or a server provided by a display case control service provider. It should be noted that one first edge server can correspond to one or more display cases. For example, a display case control service provider can divide its managed area into multiple blocks, and multiple display cases in each block can correspond to one first edge server.

[0118] The item location detection model on the first edge server can be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.

[0119] In one implementation of this disclosure, the updated weight parameters received by the display case from the first edge server are obtained by the first edge server aggregating gradient update vectors sent by multiple display cases and updating the weight parameters of the item position detection model on the first edge server according to the aggregated gradient update vectors. Therefore, the updated item position detection model on the display case can reflect the common rules between the corresponding image and the position of the item in the image, which were learned by the item position detection model on the first edge server in the previous training round. Then, the image to be detected is used as input, and the item position correction information is used as output to train the updated item position detection model. This allows the updated item position detection model on the display case to learn not only the common rules but also the rules between the image to be detected and the position of the item in the image acquired by the display case itself. This enables the trained item position detection model on the display case to learn the private rules between the image to be detected and the position of the item in the image acquired by the display case itself. When the trained item position detection model on the display case fails to converge, it indicates that the trained item position detection model still needs to be trained further. After training, the item location detection model on the display case obtains gradient update vectors and sends them out. This allows the first edge server to continue obtaining corresponding update weight parameters based on the gradient update vectors uploaded from multiple display cases, thereby continuing to train the item location detection model on each display case. When the trained item location detection model on the display case converges, it can be considered that the converged item location detection model on the display case can accurately identify the item location in the image to be detected. The converged item location detection model on the display case can be stored as the target item location detection model, that is, a model with high accuracy in identifying the item location in the image to be detected.

[0120] In the above technical solution, on the one hand, the final target item location detection model can be a model that learns both common rules and private rules, and its accuracy in identifying the location of items in the image to be detected is high; on the other hand, since the process of continuing to train the item location detection model on each display case is jointly performed by the display case and the first edge server, compared with the display case or server alone further training the item location detection model, the required processing resources are less and the training speed is faster.

[0121] In one implementation of this disclosure, in response to a mismatch between body node correction information and body node information, the method further includes the following steps before training the body node detection model, using the image to be detected as input and the body node correction information as output:

[0122] Receive the second update weight parameters sent by the second edge server, and update the body node detection model according to the second update weight parameters;

[0123] In response to a mismatch between body node correction information and body node information, the body node detection model is trained by taking the image to be detected as input and the body node correction information as output, including:

[0124] In response to the mismatch between body node correction information and body node information, the image to be detected is used as input and the body node correction information is used as output to train the updated body node detection model.

[0125] The method also includes the following steps:

[0126] In response to the failure of the trained body node detection model to converge, the second gradient update vector is obtained based on the trained body node detection model and sent to the second edge server.

[0127] Alternatively, in response to the convergence of the trained body node detection model, the trained body node detection model is stored as the target body node detection model.

[0128] In one implementation of this disclosure, a second edge server is used to aggregate gradient update vectors and update the weight parameters of the body node detection model on the second edge server according to the aggregated gradient update vectors to obtain updated weight parameters. The second edge server can be a cloud server or a server provided by a display case control service provider. It should be noted that one second edge server can correspond to one or more display cases. For example, a display case control service provider can divide its managed area into multiple blocks, and multiple display cases in each block can correspond to one second edge server.

[0129] The body node detection model on the second edge server can be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.

[0130] In one implementation of this disclosure, the updated weight parameters received by the display case from the second edge server are obtained by the second edge server aggregating gradient update vectors sent by multiple display cases and updating the weight parameters of the body node detection model on the second edge server according to the aggregated gradient update vectors. Therefore, the updated body node detection model on the display case can reflect the common patterns between the corresponding image and the body nodes in the image learned by the body node detection model on the second edge server in the previous training round. Then, the image to be detected is used as input, and the body node correction information is used as output to train the updated body node detection model. This allows the updated body node detection model on the display case to not only learn the common patterns but also to learn the patterns between the image to be detected and the body nodes in the image acquired by the display case itself. This enables the trained body node detection model on the display case to learn the private patterns between the image to be detected and the body nodes in the image acquired by the display case itself. When the trained body node detection model on the display case fails to converge, it indicates that the trained body node detection model on the display case still needs to be trained further. After training, the body node detection model on the display case obtains and sends the gradient update vector. This allows the second edge server to continue obtaining the corresponding update weight parameters based on the gradient update vectors uploaded from multiple display cases, thereby continuing to train the body node detection model on each display case. When the trained body node detection model on the display case converges, it can be considered that the converged body node detection model on the display case can accurately identify the body nodes in the image to be detected. The converged body node detection model on the display case can be stored as the target body node detection model, that is, a model with high accuracy in identifying body nodes in the image to be detected.

[0131] In the above technical solution, on the one hand, the final target body node detection model can be a model that learns both common rules and private rules, and its accuracy in recognizing body nodes in the image to be detected is high; on the other hand, since the process of continuing to train the body node detection model on each display case is jointly executed by the display case and the second edge server, compared with the further training of the body node detection model by the display case or the server alone, the required processing resources are less and the training speed is faster.

[0132] In one implementation of this disclosure, the number of images to be detected includes at least a first image to be detected and a second image to be detected, wherein the acquisition time of the first image to be detected is earlier than the acquisition time of the second image to be detected;

[0133] Alarm body nodes may include:

[0134] The first body node has a first position in the first image to be detected and a second position in the second image to be detected. The first distance between the first position and the position of the second body node in the first image to be detected is greater than the second distance between the second position and the position of the second body node in the second image to be detected. The first body node includes a left hand node and a right hand node, and the second body node includes a head node.

[0135] In the above technical solution, by limiting the number of images to at least include a first image to be detected and a second image to be detected whose acquisition time is earlier than that of the second image to be detected, and by limiting the alarm body node to include a first body node whose position is a first position in the first image to be detected and whose position is a second position in the second image to be detected, the accuracy of the generated alarm information can be improved. This is because the first distance between the first position and the position of the second body node in the first image to be detected is greater than the second distance between the second position and the position of the second body node in the second image to be detected. The first body node includes a left hand node and a right hand node, and the second body node includes a head node. Therefore, the image to be detected includes at least one alarm body node, which can be understood as the user possibly moving their left or right hand toward their head, and the hand moving toward the head is holding an item stored in a display case. In this situation, the probability that the user wants to bite the item held in their hand is high, that is, the probability that the user is damaging the item stored in the display case held in their hand is high.

[0136] In one implementation of this disclosure, the number of images to be detected includes at least a first image to be detected and a second image to be detected, wherein the acquisition time of the first image to be detected is earlier than the acquisition time of the second image to be detected;

[0137] The alarm body node includes a third body node and a fourth body node, wherein the distance difference between the third distance and the fourth distance is less than or equal to the distance difference threshold. The third distance is the distance between the position of the third body node in the first image to be detected and the position of the fourth body node in the first image to be detected. The fourth distance is the distance between the position of the third body node in the second image to be detected and the position of the fourth body node in the second image to be detected. The third body node is either a left-hand node or a right-hand node, and the fourth body node is either a left-hand node or a right-hand node.

[0138] In the above technical solution, by limiting the number of images to be detected to at least include a first image to be detected and a second image to be detected whose acquisition time is earlier than that of the second image to be detected, and by limiting the alarm body node to include a third body node and a fourth body node, where the distance difference between the third distance and the fourth distance is less than or equal to the distance difference threshold, the third distance is the distance between the position of the third body node in the first image to be detected and the position of the fourth body node in the first image to be detected, and the fourth distance is the distance between the position of the third body node in the second image to be detected and the position of the fourth body node in the second image to be detected, the third body node is one of the left-hand node and the right-hand node, and the fourth body node is the other of the left-hand node and the right-hand node, the inclusion of the alarm body node in the image to be detected can be understood as the user possibly continuously squeezing or tearing the items stored in the display case they are holding with their left and right hands. Under such circumstances, the probability of the user damaging the items stored in the display case they are holding is high, thereby improving the accuracy of the generated alarm information.

[0139] In one implementation of this disclosure, the number of images to be detected includes at least a first image to be detected and a second image to be detected. The acquisition time of the first image to be detected is earlier than the acquisition time of the second image to be detected, and the time difference between the acquisition time of the first image to be detected and the acquisition time of the second image to be detected is less than or equal to a time difference threshold.

[0140] Alarm body nodes, including

[0141] A fifth body node is defined as a node whose position in the first image to be detected is greater than or equal to a distance threshold from its position in the second image to be detected. The fifth body node includes a left-hand node and a right-hand node.

[0142] In the above technical solution, by limiting the number of images to be detected to at least include a first image to be detected and a second image to be detected, and the acquisition time of the first image to be detected is earlier than the acquisition time of the second image to be detected, the time difference between the acquisition time of the first image to be detected and the acquisition time of the second image to be detected is less than or equal to a time difference threshold, and the alarm body node is limited to a fifth body node whose position in the first image to be detected and the distance between its position in the second image to be detected is greater than or equal to a distance threshold, wherein, since the fifth body node includes a left-hand node and a right-hand node, the inclusion of an alarm body node in the image to be detected can be understood as the user possibly holding an item stored in the display case with their left or right hand and moving the item at high speed to tear or smash the item, the probability of the user damaging the item stored in the display case is high, thereby improving the accuracy of the generated alarm information.

[0143] In one implementation of this disclosure, before generating alarm information in response to determining that the image to be detected includes at least one alarm body node based on body key node information and determining that the image to be detected includes at least one alarm target item based on item location information, the method further includes the following steps:

[0144] Obtain the item category information of at least one item in the image to be detected. The item category information is used to indicate the category of the corresponding item.

[0145] Identify at least one target item in the image to be detected based on the item category information.

[0146] In one embodiment of this disclosure, obtaining the item category information of at least one item in the image to be detected can be understood as performing item category recognition on the image to be detected according to a pre-acquired algorithm, or it can be achieved by sending the image to be detected and receiving item category information obtained by other devices or systems, such as a cloud server, through item category recognition on the image to be detected.

[0147] In one embodiment of this disclosure, determining at least one target item in the image to be detected based on item category information can be understood as matching the item category information with pre-acquired storage item category information used to indicate the items stored in the display case, and determining, based on the matching result, that at least one item stored in the display case in the image to be detected is the target item.

[0148] For example, when the item category information indicates that the item category is carbonated soft drink, and the saved item category information indicates that the item category is tea beverage, soda water, carbonated soft drink, since tea beverage, soda water, carbonated soft drink includes carbonated soft drink, it can be determined that the item in the image to be detected that the item category is carbonated soft drink is the target item.

[0149] In the above technical solution, by obtaining the item category information of at least one item in the image to be detected, and determining at least one target item in the image to be detected based on the item category information, the accuracy of the target item can be improved.

[0150] In one implementation of this disclosure, the image to be detected is multiple video frames in the cabinet door video, the cabinet door video is captured by the cabinet door image acquisition device in response to the cabinet door unlocking and starting recording, and the cabinet door image acquisition device is set at the cabinet door of the display cabinet.

[0151] Alternatively, the image to be detected may be multiple video frames from the cabinet video, which is captured by the cabinet image acquisition device in response to the cabinet door unlocking and recording begins. The cabinet image acquisition device is located on the cabinet of the display case.

[0152] In one embodiment of this disclosure, cabinet door unlocking can be understood as the cabinet door locking device on the display cabinet responding to an unlocking command to unlock the cabinet door, allowing the cabinet door to rotate or slide relative to the cabinet body of the display cabinet to open the item entrance / exit of the display cabinet; it can also be understood as the cabinet door locking device on the display cabinet being triggered by a corresponding unlocking operation, causing the cabinet door locking device itself to be set to an unlocked state, thereby allowing the cabinet door to rotate or slide relative to the cabinet body of the display cabinet to open the item entrance / exit of the display cabinet, through which users can move items out of the item display area or move items into the item display area.

[0153] In one embodiment of this disclosure, determining whether a cabinet door is locked can be achieved by receiving cabinet door locking status information sent by a cabinet door locking device on the display cabinet, and determining whether the cabinet door is locked based on the cabinet door locking status information. Alternatively, cabinet door locking status information sent by other devices or systems can be received, and the cabinet door can be determined based on the cabinet door locking status information. For example, an infrared detection device independent of the display cabinet can detect whether the cabinet door is locked, and send cabinet door locking status information to the display cabinet based on the detection result; alternatively, a camera independent of the display cabinet, such as a security camera, can acquire images including all or part of the display cabinet, perform image recognition based on the images, and send cabinet door locking status information to the display cabinet based on the image recognition.

[0154] In this embodiment, considering that items can only be moved out of or into the display case when the display case door is unlocked, by limiting the images to be detected to multiple video frames from the door video, or multiple images to be detected to multiple video frames from the cabinet body video, the door video is acquired by the door image acquisition device in response to the door unlocking and in response to the door locking and in response to the door locking and in response to the door locking and in response to the door unlocking and in response to the door locking and in response to the cabinet body video, the probability of the images to be detected including items moved out or into the display case can be increased, the number of images to be processed can be reduced, and data processing resources can be saved.

[0155] Figure 15 A flowchart illustrating the overall process of a display case control method according to an embodiment of the present disclosure is shown, as follows: Figure 15 As shown, the display case control methods include:

[0156] In step S201, the image to be detected is acquired.

[0157] In step S202, a pre-trained object location detection model is obtained, and the image to be detected is input into the object location detection model to obtain the object location information output by the object location detection model.

[0158] In step S203, a pre-trained body node detection model is obtained, and the image to be detected is input into the body node detection model to obtain the body node information output by the body node detection model.

[0159] In step S204, the item category information of at least one item in the image to be detected is obtained. The item category information is used to indicate the category of the corresponding item.

[0160] In step S205, at least one target item in the image to be detected is determined based on the item category information.

[0161] In step S206, in response to determining that the image to be detected includes at least one alarm body node based on the body key node information, and determining that the image to be detected includes at least one alarm target item based on the item location information, alarm information is generated.

[0162] In step S207, the first update weight parameter sent by the first edge server is received, and the item location detection model is updated according to the first update weight parameter.

[0163] In step S208, in response to the alarm information, the item position correction information input by the human-computer interaction device of the display case is obtained.

[0164] In step S209, in response to the mismatch between the item position correction information and the item position information, the image to be detected is used as input and the item position correction information is used as output to train the updated item position detection model.

[0165] In step S210, in response to the failure of the trained item location detection model to converge, the first gradient update vector is obtained based on the trained item location detection model and sent to the first edge server.

[0166] In step S211, in response to the convergence of the trained item location detection model, the trained item location detection model is stored as the target item location detection model.

[0167] In step S212, the second update weight parameter sent by the second edge server is received, and the body node detection model is updated according to the second update weight parameter.

[0168] In step S213, in response to the alarm information, body node correction information input by the human-computer interaction device of the display case is obtained.

[0169] In step S214, in response to the mismatch between the body node correction information and the body node information, the image to be detected is used as input and the body node correction information is used as output to train the updated body node detection model.

[0170] In step S215, in response to the failure of the trained body node detection model to converge, the second gradient update vector is obtained based on the trained body node detection model and sent to the second edge server.

[0171] In step S216, in response to the convergence of the trained body node detection model, the trained body node detection model is stored as the target body node detection model.

[0172] This disclosure also discloses an electronic device. Figure 16 A schematic structural block diagram of an electronic device according to an embodiment of the present disclosure is shown, such as... Figure 16 As shown, the electronic device 700 includes a memory 701 and a processor 702; wherein the memory 701 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 702 to implement the above method steps.

[0173] Figure 17 This is a schematic diagram of the structure of a computer system suitable for implementing the display case control method according to an embodiment of the present disclosure. For example... Figure 17 As shown, the computer system 800 includes a processing unit 801, which can execute various processes described above based on a program stored in a read-only memory (ROM) 802 or a program loaded from a storage section 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0174] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed. The processing unit 801 can be implemented as a CPU, GPU, TPU, FPGA, NPU, etc.

[0175] In particular, according to embodiments of this disclosure, the methods described above with reference to the accompanying drawings can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a readable medium thereof, the computer program containing program code for performing the methods in the drawings. In such embodiments, the computer program can be downloaded and installed from a network via a communication section 809, and / or installed from a removable medium 811. For example, embodiments of this disclosure include a readable storage medium storing computer instructions that, when executed by a processor, implement program code for performing the methods in the drawings.

[0176] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0177] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The described units or modules can also be located in a processor, and the names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0178] In another aspect, this disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the apparatus described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs that are used by one or more processors to perform the methods described in this disclosure.

[0179] In addition, this disclosure also provides a computer program product storing a computer program that, when executed by a processor, enables the processor to at least implement the methods provided in the foregoing embodiments.

[0180] Figure 18 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown. Figure 19 A schematic top view of a display case according to one embodiment of the present disclosure is shown. Figure 18 as well as Figure 19 As shown, the display case 900 includes a cabinet body 901, a cabinet door 902, a first cabinet body image acquisition device 903, a second cabinet body image acquisition device 904, a cabinet door image acquisition device 905, and a processing device.

[0181] Cabinet door 902 is rotatably connected to cabinet body 901 and is used to open or close the item access entrance 911 of cabinet body 901;

[0182] The cabinet 901 includes an inner cavity 921, which is connected to the outside of the cabinet 901 through an item inlet / outlet 911. The inner cavity 921 is used to store items.

[0183] The first cabinet image acquisition device 903 and the second cabinet image acquisition device 904 are both connected to the top surface of the cabinet's inner cavity 921. The first cabinet image acquisition device 903 and the second cabinet image acquisition device 904 are used to acquire images of the item entrance / exit 911 from different directions.

[0184] The cabinet door image acquisition device 905 is connected to the side of the cabinet door 902 closest to the cabinet body 901, and the position of the cabinet door image acquisition device 905 matches the position of the door handle 912 of the cabinet door 902. The door handle 912 is connected to the side of the cabinet door 902 away from the cabinet body 901.

[0185] The processing device is communicatively connected to the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device. The processing device is used to execute the method provided in the foregoing embodiments.

[0186] In one embodiment of this disclosure, collecting images of the item entry / exit points can be understood as collecting images of all or part of the item entry / exit points. Through the collected images, it is possible to determine whether the user moves items out of the cabinet cavity from the item entry / exit points or whether the user moves items into the cabinet cavity from the item entry / exit points.

[0187] In one embodiment of this disclosure, the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device can be cameras or other devices with image acquisition functions.

[0188] In one embodiment of this disclosure, the processing apparatus may include one or more processing units, such as an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a memory, a video codec, a digital signal processor, a baseband processor, and / or a neural network processor. The different processing units may be independent devices or integrated into one or more processors.

[0189] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A display case control method, characterized in that, The method includes: Acquire images to be detected; the number of images to be detected includes at least a first image to be detected and a second image to be detected, wherein the acquisition time of the first image to be detected is earlier than the acquisition time of the second image to be detected; The system obtains the item location information of at least one item in the image to be detected and the body node information of at least one user in the image to be detected. The item location information is used to indicate the location of the corresponding item, and the body node information is used to indicate the location of at least one body node of the corresponding user. In response to determining, based on the body node information, that the image to be detected includes at least one alarm body node, and based on the item location information, determining that the image to be detected includes at least one alarm target item, an alarm message is generated. The alarm target item is a target item whose location matches the location corresponding to the alarm body node. The alarm body node includes: a first body node whose location in the first image to be detected is a first location and whose location in the second image to be detected is a second location. A first distance between the first location and the location of the second body node in the first image to be detected is greater than a second distance between the second location and the location of the second body node in the second image to be detected. The first body node includes a left-hand node and a right-hand node, and the second body node includes a head node.

2. The display case control method according to claim 1, characterized in that, The step of obtaining the item location information of at least one item in the image to be detected includes: Obtain a pre-trained object location detection model, and input the image to be detected into the object location detection model to obtain the object location information output by the object location detection model; Obtaining body node information of at least one user in the image to be detected, including: A pre-trained body node detection model is obtained, and the image to be detected is input into the body node detection model to obtain the body node information output by the body node detection model.

3. The display case control method according to claim 1, characterized in that, The method further includes: In response to the alarm information, the item position correction information input by the human-computer interaction device of the display case is obtained. In response to the mismatch between the item position correction information and the item position information, the image to be detected is used as input and the item position correction information is used as output to train the item position detection model. And / or, in response to the alarm information, acquire body node correction information input by the human-computer interaction device of the display case; in response to a mismatch between the body node correction information and the body node information, use the image to be detected as input and the body node correction information as output to train the body node detection model.

4. The display case control method according to claim 3, characterized in that, Before training the item position detection model in response to a mismatch between the item position correction information and the item position information, the method further includes: Receive the first update weight parameter sent by the first edge server, and update the item location detection model according to the first update weight parameter; In response to a mismatch between the item position correction information and the item position information, the method of training the item position detection model by taking the image to be detected as input and the item position correction information as output includes: In response to a mismatch between the item position correction information and the item position information, the method further includes training the updated item position detection model by taking the image to be detected as input and the item position correction information as output. In response to the failure of the trained object location detection model to converge, the first gradient update vector is obtained based on the trained object location detection model and sent to the first edge server. Alternatively, in response to the convergence of the trained item location detection model, the trained item location detection model is stored as the target item location detection model.

5. The display case control method according to claim 3, characterized in that, Before training the body node detection model in response to a mismatch between the body node correction information and the body node information, the method further includes: Receive the second update weight parameter sent by the second edge server, and update the body node detection model according to the second update weight parameter; In response to a mismatch between the body node correction information and the body node information, the body node detection model is trained by taking the image to be detected as input and the body node correction information as output, including: In response to a mismatch between the body node correction information and the body node information, the image to be detected is used as input and the body node correction information is used as output to train the updated body node detection model. The method further includes: In response to the failure of the trained body node detection model to converge, the second gradient update vector is obtained based on the trained body node detection model and sent to the second edge server. Alternatively, in response to the convergence of the trained body node detection model, the trained body node detection model is stored as the target body node detection model.

6. The display case control method according to any one of claims 1-5, characterized in that, The alarm body node includes a third body node and a fourth body node, wherein the distance difference between the third distance and the fourth distance is less than or equal to a distance difference threshold. The third distance is the distance between the position of the third body node in the first image to be detected and the position of the fourth body node in the first image to be detected. The fourth distance is the distance between the position of the third body node in the second image to be detected and the position of the fourth body node in the second image to be detected. The third body node is one of a left-hand node and a right-hand node, and the fourth body node is the other of a left-hand node and a right-hand node.

7. The display case control method according to any one of claims 1-5, characterized in that, The time difference between the acquisition time of the first image to be detected and the acquisition time of the second image to be detected is less than or equal to the time difference threshold. The alarm body node includes: A fifth body node whose position in the first image to be detected is greater than or equal to a distance threshold between its position in the second image to be detected, wherein the fifth body node includes a left-hand node and a right-hand node.

8. The display case control method according to any one of claims 1-5, characterized in that, Before generating alarm information in response to determining, based on the body node information, that the image to be detected includes at least one alarm body node, and based on the item location information, determining that the image to be detected includes at least one alarm target item, the method further includes: Obtain item category information for at least one item in the image to be detected, wherein the item category information is used to indicate the category of the corresponding item; At least one target item in the image to be detected is determined based on the item category information.

9. The display case control method according to any one of claims 1-5, characterized in that, The image to be detected is multiple video frames in the cabinet door video, which is captured by the cabinet door image acquisition device in response to the cabinet door unlocking and recording begins. The cabinet door image acquisition device is installed at the cabinet door of the display cabinet. Alternatively, the image to be detected may be multiple video frames from the cabinet video, which is captured by the cabinet image acquisition device in response to the cabinet door unlocking and recording begins. The cabinet image acquisition device is located on the cabinet of the display cabinet.

10. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method of any one of claims 1-9.

11. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by a processor, the computer instructions implement the method of any one of claims 1-9.

12. A display case, characterized in that, The display case includes a cabinet body, cabinet doors, a first cabinet body image acquisition device, a second cabinet body image acquisition device, a cabinet door image acquisition device, and a processing device. The cabinet door is rotatably connected to the cabinet body and is used to open or close the item entrance / exit of the cabinet body. The cabinet includes an internal cavity, which is connected to the outside of the cabinet through the item inlet / outlet, and is used to store items. Both the first cabinet image acquisition device and the second cabinet image acquisition device are connected to the top surface of the cabinet's inner cavity. The first cabinet image acquisition device and the second cabinet image acquisition device are used to acquire images of the item's entrance and exit from different directions. The cabinet door image acquisition device is connected to the side of the cabinet door closest to the cabinet body, and the position of the cabinet door image acquisition device matches the position of the door handle of the cabinet door. The door handle is connected to the side of the cabinet door furthest from the cabinet body. The processing device is communicatively connected to the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device, and the processing device is used to execute the method of any one of claims 1-9.

Images