Display cabinet control method, apparatus, medium, and display cabinet

Abstract

The display cabinet control method, device, medium and display cabinet disclosed by the embodiments of the present disclosure comprise the following steps: acquiring at least one cabinet door image and a cabinet door included angle angle value corresponding to each cabinet door image; acquiring depth information of an article located in a target region in the cabinet door image according to the cabinet door image and the cabinet door included angle angle value; and acquiring position information of an idle display position of at least one article not accommodated in the cabinet body according to the depth information. The technical solution can determine the idle display position in the display cabinet at a lower cost and a higher accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of control technology, specifically to a display cabinet control method, equipment, medium, and display cabinet. Background Technology

[0002] In recent years, to facilitate user access to product information, merchants and businesses have often placed items in display cases, simultaneously storing and showcasing goods. When users need to remove or add items, they can open the display case themselves. In this scenario, the display case can capture images of the items inside and upload the data. Other devices or systems, such as servers or cloud platforms, can then use these images to determine available display spaces after items are moved in or out, allowing for statistical analysis or adjustments to the display case's inventory. Summary of the Invention

[0003] This disclosure provides a display cabinet control method, device, medium, and display cabinet to solve the problem in related technologies that it is impossible to determine the vacant display positions of unoccupied items in a display cabinet with low cost and high accuracy.

[0004] Firstly, this disclosure provides a display cabinet control method.

[0005] Specifically, the display case control method includes:

[0006] Acquire at least one cabinet door image and the cabinet door angle value corresponding to each cabinet door image. The cabinet door images are acquired by a cabinet door image acquisition device, which is set at the cabinet door of the display cabinet. The cabinet door angle value is the angle between the cabinet door and the cabinet body of the display cabinet when the cabinet door image is acquired.

[0007] Depth information of items located in the target area in the cabinet door image is obtained based on the cabinet door image and the included angle value of the cabinet door;

[0008] Based on depth information, obtain the location information of at least one empty display space inside the cabinet that does not contain any items.

[0009] In one implementation of this disclosure, the number of cabinet door images is greater than or equal to one;

[0010] Based on the cabinet door image and the included angle value of the cabinet door, obtain the depth information of the items located in the target area in the cabinet door image, including:

[0011] A pre-trained first depth information estimation model is obtained. The cabinet door images and the cabinet door angle values ​​corresponding to each cabinet door image are input into the first depth information estimation model to obtain the depth information of the items located in the target area output by the first depth information estimation model.

[0012] In one implementation of this disclosure, the method further includes:

[0013] The number of available front spaces for the aisles inside the cabinet is determined based on the available display space.

[0014] Image recognition is performed on at least one cabinet door image to obtain the empty space marking recognition result. Based on the empty space marking recognition result, the number of empty spaces in the image recognition of at least one cargo aisle inside the cabinet is obtained. The empty space marking recognition result is used to indicate the empty space marking on the surface of each cargo aisle inside the cabinet that is used to carry items.

[0015] In response to a mismatch between the number of empty spaces in front of the target cargo channel and the number of empty spaces in front of the target cargo channel, the target depth information corresponding to the target cargo channel is obtained based on the number of empty spaces in front of the target cargo channel.

[0016] The cabinet door image is used as input, and the target depth information is used as output to train the first depth information estimation model.

[0017] In one implementation of this disclosure, before training the first depth information estimation model, the method further includes: (The method takes the cabinet door image as input and the target depth information as output.)

[0018] Receive the first update weight parameters sent by the first edge server, and update the first depth information estimation model according to the first update weight parameters;

[0019] Using the cabinet door image as input and target depth information as output, the first depth information estimation model is trained, including:

[0020] The cabinet door image is used as input, and the target depth information is used as output to train the updated first depth information estimation model.

[0021] The method also includes:

[0022] In response to the convergence of the first depth information estimation model after training, the first gradient update vector is obtained based on the first depth information estimation model after training, and the first gradient update vector is sent to the first edge server.

[0023] Alternatively, in response to the convergence of the first depth information estimation model after training, the first depth information estimation model after training is stored as the target first depth information estimation model.

[0024] In one implementation of this disclosure, the number of cabinet door images is greater than or equal to two;

[0025] Based on the cabinet door image and the included angle value of the cabinet door, obtain the depth information of the items located in the target area in the cabinet door image, including:

[0026] Obtain the second depth information estimation model corresponding to each pre-trained cabinet door angle value, input the cabinet door angle value and the cabinet door image corresponding to the cabinet door angle value into the second depth information estimation model corresponding to the cabinet door angle value, so as to obtain the sub-depth information of at least one item in the cabinet door image output by each second depth information estimation model.

[0027] The depth information of the items located in the target area is obtained by estimating the sub-depth information output by the model based on each second depth information.

[0028] In one implementation of this disclosure, the method further includes:

[0029] The number of available front spaces for the aisles inside the cabinet is determined based on the available display space.

[0030] Image recognition is performed on at least one cabinet door image to obtain the empty space marking recognition result. Based on the empty space marking recognition result, the number of empty spaces in the image recognition of at least one cargo aisle inside the cabinet is obtained. The empty space marking recognition result is used to indicate the empty space marking on the surface of each cargo aisle inside the cabinet that is used to carry items.

[0031] In response to a mismatch between the number of empty spaces in front of the target cargo channel and the number of empty spaces in front of the target cargo channel, the target depth information corresponding to the target cargo channel is obtained based on the number of empty spaces in front of the target cargo channel.

[0032] The cabinet door angle value and the corresponding cabinet door image are used as inputs to the second depth information estimation model corresponding to the cabinet door angle value, and the target depth information is used as the output. The second depth information estimation model is trained.

[0033] In one implementation of this disclosure, the cabinet door angle value and the cabinet door image corresponding to the cabinet door angle value are used as inputs to a second depth information estimation model corresponding to the cabinet door angle value, and the target depth information is used as the output. Before training each second depth information estimation model, the method further includes...

[0034] Receive the second update weight parameters corresponding to each second depth information estimation model sent by the second edge server, and update each second depth information estimation model according to the corresponding second update weight parameters;

[0035] The cabinet door angle value and the corresponding cabinet door image are used as inputs to the second depth information estimation model corresponding to the cabinet door angle value. The target depth information is used as the output. Each second depth information estimation model is trained, including:

[0036] The cabinet door angle value and the cabinet door image corresponding to the cabinet door angle value are used as inputs to the second depth information estimation model corresponding to the cabinet door angle value, and each updated second depth information estimation model is trained.

[0037] The method also includes:

[0038] In response to the convergence of the trained second depth information estimation model, the first gradient update vector corresponding to the trained second depth information estimation model is obtained according to the trained second depth information estimation model, and the first gradient update vector is sent to the first edge server.

[0039] Alternatively, in response to the convergence of the trained second depth information estimation model, the trained second depth information estimation model is stored as the target second depth information estimation model.

[0040] In one implementation of this disclosure, obtaining the included angle value of the cabinet door corresponding to each cabinet door image includes:

[0041] Obtain a pre-trained cabinet door angle estimation model, input each cabinet door image into the cabinet door angle estimation model, and obtain the cabinet door angle value output by the cabinet door angle estimation model corresponding to each cabinet door image;

[0042] Alternatively, obtain the cabinet door image acquisition time for each cabinet door image, and obtain the cabinet door angle value acquired by the angle sensor at the cabinet door image acquisition time for each cabinet door image. The angle sensor is used to acquire the angle value between the cabinet door and the cabinet body.

[0043] In one implementation of this disclosure, the cabinet door image is a video frame image in the cabinet door video, and the cabinet door video is acquired by the cabinet door image acquisition device in response to the cabinet door unlocking to start recording and in response to the cabinet door locking to end recording.

[0044] In a second aspect, embodiments of this disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method as described in the first aspect or any implementation thereof.

[0045] Thirdly, this disclosure provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the method as described in the first aspect or any implementation thereof.

[0046] Fourthly, this disclosure provides a computer program product including computer instructions that, when executed by a processor, implement the method as described in the first aspect or any implementation thereof.

[0047] Fifthly, this disclosure provides a display cabinet, which includes a cabinet body, cabinet doors, a first cabinet body image acquisition device, a second cabinet body image acquisition device, a cabinet door image acquisition device, and a processing device.

[0048] The cabinet door is rotatably connected to the cabinet body and is used to open or close the cabinet's entrance / exit for items.

[0049] The cabinet includes an internal cavity, which is connected to the outside of the cabinet through an item inlet / outlet. The internal cavity is used to store items.

[0050] Both the first cabinet image acquisition device and the second cabinet image acquisition device are connected to the top surface of the cabinet's internal cavity. The first cabinet image acquisition device and the second cabinet image acquisition device are used to acquire images of the item's entrance and exit from different directions.

[0051] The cabinet door image acquisition device is connected to the side of the cabinet door closest to the cabinet body, and the position of the cabinet door image acquisition device matches the position of the cabinet door handle, while the door handle is connected to the side of the cabinet door furthest from the cabinet body.

[0052] The processing device is communicatively connected to the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device. The processing device is used to execute the method described in the first aspect or any implementation thereof.

[0053] The technical solutions provided in this disclosure may have the following beneficial effects:

[0054] The above technical solution acquires at least one cabinet door image captured by a cabinet door image acquisition device installed at the cabinet door of the display case, and the cabinet door angle value corresponding to the angle between the cabinet door and the cabinet body when the cabinet door image was acquired. Based on the cabinet door image and the cabinet door angle value, the depth information of the items located in the target area in the cabinet door image is obtained. Since the depth information of the items in the cabinet door image can be obtained with high accuracy based on the cabinet door image acquired at the corresponding cabinet door angle, the depth information is used to indicate the distance of the items in the cabinet door image to the item image acquisition device of the cabinet door image acquisition device. Then, the position information of at least one empty display position in the cabinet that does not contain an item is obtained based on the depth information. Since the distance from the cabinet door image acquisition device to the corresponding display position image acquisition device at the corresponding cabinet door angle can be considered known, the distance of the item image acquisition device and the distance of the display position image acquisition device can be used to determine whether the corresponding display position contains an item, thereby determining the position information of the empty display position in the cabinet. In the above solution, since the depth information obtained has a high accuracy rate, the accuracy of the vacant display location obtained based on the depth information is also high. Furthermore, since only the cabinet door image acquisition device set at the cabinet door is required to acquire the cabinet door image, fewer cabinet door image acquisition devices are needed, reducing the cost of the display cabinet. Therefore, the above solution achieves the goal of determining the vacant display location in the display cabinet with a low cost and a high accuracy rate.

[0055] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0056] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

[0057] Figure 1 A schematic structural block diagram of a display cabinet according to an embodiment of the present disclosure is shown.

[0058] Figure 2 A schematic structural block diagram of a motherboard according to an embodiment of the present disclosure is shown.

[0059] Figure 3 A schematic structural block diagram of a control panel according to an embodiment of the present disclosure is shown.

[0060] Figure 4 A schematic structural block diagram of a power management module according to an embodiment of the present disclosure is shown.

[0061] Figure 5 A flowchart illustrating a display case control method according to an embodiment of the present disclosure is shown.

[0062] Figure 6 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown.

[0063] Figure 7 A schematic top view of a display case according to one embodiment of the present disclosure is shown.

[0064] Figure 8 A schematic diagram showing an image of a cabinet door according to an embodiment of the present disclosure.

[0065] Figure 9 A schematic top view of a display case according to one embodiment of the present disclosure is shown.

[0066] Figure 10 A flowchart illustrating the overall process of a display case control method according to an embodiment of the present disclosure is shown.

[0067] Figure 11 A schematic structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0068] Figure 12 This is a schematic diagram of the structure of a computer system suitable for implementing a display case control method according to an embodiment of the present disclosure.

[0069] Figure 13 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown.

[0070] Figure 14 A schematic top view of a display case according to one embodiment of the present disclosure is shown. Detailed Implementation

[0071] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement them. Furthermore, for clarity, portions unrelated to the description of the exemplary embodiments have been omitted from the drawings.

[0072] In this disclosure, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, figures, steps, behaviors, components, parts or combinations thereof disclosed in this specification, and do not preclude the possibility of the presence or addition of one or more other features, figures, steps, behaviors, components, parts or combinations thereof.

[0073] It should also be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0074] As mentioned above, with the development of technology and the improvement of people's living standards, merchants or enterprises no longer simply place goods on shelves. Instead, to facilitate users' understanding of product information, they can place items in display cases, thus simultaneously storing and displaying the goods. When users need to remove items from the display case or put items into it, they can open the display case themselves and perform the corresponding operations.

[0075] In recent years, the number of display cases put into operation has gradually increased. During the use of display cases, merchants or enterprises generally need to monitor the items in the display cases to determine the empty display positions in the display cases after items are moved out or in, so as to make corresponding statistics or adjust the items contained in the display cases.

[0076] In one embodiment, an item-accommodating sensor (e.g., infrared sensor, capacitive sensor, etc.) corresponding to each display location in the display case can be installed at a corresponding position (e.g., above or below the display location). The sensor signal sent by the corresponding item-accommodating sensor is acquired, and the presence or absence of an item in that display location is determined based on the sensor signal. However, this approach requires a large number of item-accommodating sensors in the display case due to the need to install them at each display location, increasing the cost of determining vacant display locations without items.

[0077] In another embodiment, after the display case is unlocked, an image corresponding to the display case can be captured. Based on the image, the moved items that were removed or moved into the display case can be determined, and a pre-stored display plan for the display case items can be obtained. This display plan indicates the specific display position of the corresponding items in the display case. Based on the display plan and the determined moved items, the available display positions in the display case after the items are removed or moved into the display case are obtained. However, in this solution, since users may arbitrarily move other items in the display case when moving items into or out of the display case, the actual display position of the items in the display case may differ from the display position indicated by the display plan. This results in a relatively poor accuracy rate for obtaining the available display positions for the moved items based on the display plan.

[0078] Therefore, determining the vacant display locations of unused items in display cases with low cost and high accuracy is an increasingly urgent problem that needs to be solved.

[0079] In view of the above-mentioned defects, a display cabinet control method is proposed in one embodiment of this disclosure.

[0080] The display cabinet control method provided in this application embodiment can be applied to display cabinets, which can have a temperature control function. The temperature control function can be a cooling function, such as a refrigerated display cabinet, a frozen display cabinet, a refrigerator, a wine cabinet, a cosmetic preservation cabinet, etc.; the temperature control function can also be a heating function, such as a warming cabinet, a heated display cabinet, a hot beverage cabinet, etc. This application embodiment does not limit the specific type of display cabinet.

[0081] For example, Figure 1 A schematic structural block diagram of a display cabinet according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the display case 100 may include a compressor 11, a condenser 12, a throttling element 13, and an evaporator 14. The compressor 11, condenser 12, throttling element 13, and evaporator 14 are connected by pipes filled with refrigerant to form a closed pipeline, which constitutes a refrigeration system or heating system capable of circulating refrigerant.

[0082] In one embodiment of this application, the display cabinet includes a cabinet body and a cabinet door, wherein a control board and a power management module may be installed in the cabinet body, and a main board may be installed in the cabinet door.

[0083] In one embodiment of this application, Figure 2 A schematic structural block diagram of a motherboard according to an embodiment of the present disclosure is shown, such as... Figure 2 As shown, the motherboard 200 includes a processor 201, random access memory 202, flash memory 203, wireless LAN Bluetooth module 204, gyroscope 205, pressure sensor 206, microphone 207, speaker 208, camera 209, and cellular communication module 210.

[0084] In one embodiment of this application, Figure 3 A schematic structural block diagram of a control panel according to an embodiment of the present disclosure is shown, such as... Figure 3 As shown, the control board 300 includes a power input interface 301, a power output interface 302, a metering chip 303, a microcontroller chip 304, a real-time clock chip, a light switch interface 305, a temperature control switch interface 306, an evaporator fan interface 307, a compressor interface 308, a condenser fan interface 309, a temperature sensor interface 310, a communication interface 311, and a power interface 312.

[0085] In one embodiment of this application, Figure 4 A schematic structural block diagram of a power management module according to an embodiment of the present disclosure is shown, such as... Figure 4As shown, the power management module 400 includes an AC-to-DC conversion module 401, a charging management module 402, and a battery 403. The power management module 400 supplies power to the motherboard and control board and manages the charging and discharging of the battery. The power management module 400 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 400 may also be located within the processor.

[0086] In one embodiment of this application, the display case further includes a display screen. The display case implements its display function through a graphics processor, a display screen, and an application processor. The graphics processor is a microprocessor for image processing, connected to the display screen and the application processor. The graphics processor is used to perform mathematical and geometric calculations and for graphics rendering. The processor may include one or more graphics processors that execute program instructions to generate or modify display information.

[0087] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the display cabinet. In other embodiments of this application, the display cabinet may include more or fewer components than illustrated, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, by combining different components, the display cabinet in the embodiments of this application may be any of the following: a retail cabinet, a heated cabinet, a refrigerated cabinet, a freezer, a combination cabinet, or a display case.

[0088] Figure 5 A flowchart illustrating a display case control method according to an embodiment of the present disclosure is shown, such as... Figure 5 As shown, the display case control method includes the following steps S101-S103:

[0089] In step S101, at least one cabinet door image and the cabinet door angle value corresponding to each cabinet door image are obtained.

[0090] The cabinet door image is captured by a cabinet door image acquisition device, which is installed at the cabinet door of the display cabinet. The included angle of the cabinet door is the angle between the cabinet door and the cabinet body of the display cabinet when the cabinet door image is captured.

[0091] In one embodiment of this disclosure, acquiring a cabinet door image can be achieved by receiving a cabinet door image from a cabinet door image acquisition device, reading a cabinet door image pre-stored in the display case, or receiving a cabinet door image from another device or system. A cabinet door image can include one or more static images, or one or more dynamic images or videos. A cabinet door image can be understood as including images of all or part of the display case, or images of all or part of the items within the display case.

[0092] In one embodiment of this disclosure, the cabinet door image acquisition device can be installed on the top or bottom of the cabinet door, or on the side wall of the cabinet door near the cabinet body, or on the side wall of the cabinet door away from the cabinet body, or on the door handle of the cabinet door. This application does not limit the specific installation location of the cabinet door image acquisition device.

[0093] For example, Figure 6 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown. Figure 7 A schematic top view of a display case according to one embodiment of the present disclosure is shown, such as... Figure 6 as well as Figure 7 As shown, the display case includes a cabinet body 501, a cabinet door 502, and a cabinet door image acquisition device 503, wherein the cabinet door image acquisition device 503 is connected to the side of the cabinet door 502 near the cabinet body 501.

[0094] The cabinet 501 includes a display area 511 and an item entrance / exit 521. The display area 511 is connected to the outside of the cabinet 501 through the item entrance / exit 521. The display area 511 is used to accommodate items stored in the display cabinet. Items in the display area 511 can be moved out of the display cabinet through the item entrance / exit 521, or items can be moved into the display area 511 through the item entrance / exit 521.

[0095] The cabinet doors can be rotatably connected to the cabinet body, or they can be slidably connected to the cabinet body. Alternatively, the cabinet doors can be connected to the cabinet body by folding. It should be noted that this application does not specifically limit the way the cabinet doors are connected to the cabinet body. For ease of understanding, this is just a general description. Figure 6 as well as Figure 7 The following explanation uses the rotatable connection between cabinet door 502 and cabinet body 501 as an example. Cabinet door 502 is used to open or close the item access entrance 521.

[0096] The cabinet door image acquisition device 503 can be used to acquire images of at least one of the following: the entire display area 511, a portion of the display area 511, and the item entrance / exit 521, in order to obtain cabinet door images. It should be noted that the display cabinet may include only one cabinet door image acquisition device 503, or it may include multiple cabinet door image acquisition devices 503 that acquire cabinet door images from different directions. This disclosure does not impose a specific limitation on the number of cabinet door image acquisition devices 503.

[0097] In one embodiment of this disclosure, obtaining the cabinet door angle value corresponding to each cabinet door image can be done by receiving the cabinet door angle value sent by an angle sensing device, reading the cabinet door angle value stored in the display cabinet in advance, or receiving the cabinet door angle value sent by other devices or systems.

[0098] In step S102, the depth information of the items located in the target area in the cabinet door image is obtained based on the cabinet door image and the cabinet door angle value.

[0099] In one embodiment of this disclosure, the target area can be understood as the image area in the cabinet door image that corresponds to the display position inside the cabinet, given the included angle value of the cabinet door. A cabinet door image may include one target area or multiple target areas.

[0100] For example, Figure 8 A schematic diagram showing an image of a cabinet door according to an embodiment of the present disclosure, such as... Figure 8 As shown, the cabinet door image includes multiple aisles 505 in the cabinet body for storing items. Each aisle 505 can be used to store multiple items 601. The area 602 where each aisle 505 is located in the cabinet door image can be understood as the target area.

[0101] In one embodiment of this disclosure, the depth information of an item located in a target area in a cabinet door image can be understood as indicating the distance from an item in at least one target area in the cabinet door image to the cabinet door image acquisition device.

[0102] In one embodiment of this disclosure, the depth information of an item located in a target area in a cabinet door image is obtained based on the cabinet door image and the cabinet door angle value. This can be understood as calculating the depth information of an item located in a target area in a cabinet door image based on a pre-acquired algorithm, according to the cabinet door image and the cabinet door angle value. Alternatively, it can be understood as acquiring a pre-trained depth information model, inputting the cabinet door image and the cabinet door angle value into the depth information model, and obtaining the depth information of an item located in at least one target area in the cabinet door image output by the depth information model.

[0103] In step S103, the location information of at least one empty display position inside the cabinet that does not contain any items is obtained based on the depth information.

[0104] In one embodiment of this disclosure, obtaining the location information of at least one vacant display location within the cabinet that does not contain an item based on depth information can be understood as calculating the location information of at least one vacant display location based on a pre-obtained location information algorithm and depth information, or it can be understood as obtaining a pre-trained location information model, inputting depth information into the location information model, and obtaining the location information of at least one vacant display location output by the location information model.

[0105] For example, Figure 9 A schematic top view of a display case according to one embodiment of the present disclosure is shown, such as... Figure 8As shown, depth information can be used to indicate the actual item distance 514 from the item closest to the cabinet door image acquisition device 503 in the aisle 505 shown in the cabinet door image to the cabinet door image acquisition device 503. When the angle 512 between the cabinet body 501 and the cabinet door 502 is the cabinet door angle value, the item distance from the item closest to the cabinet door image acquisition device 503 in the aisle 505 containing the corresponding number of items to the target item distance of the cabinet door image acquisition device 503 can be considered as known. When the actual item distance 514 matches the target item distance, it can be understood that the aisle 505 contains the corresponding number of items. Based on the corresponding number of items, it can be determined whether there is an empty space in the aisle 505 that does not contain items, thereby obtaining the location information of at least one empty display space.

[0106] For example, let's illustrate this using a display case to provide unmanned vending services. When a customer wants to purchase an item from the display case, they can use a mobile communication terminal to scan the QR code on the surface of the display case to access the corresponding cloud authorization server and send a statistical authorization request to the cloud authorization server, i.e., requesting to purchase the item from the display case. After the cloud authorization server approves the statistical authorization request, it sends statistical authorization information to the display case. The display case receives the statistical authorization information from the cloud server and, in response, unlocks its door. After the door is unlocked, the customer can open it and take the item from the display case through the item access window. After taking the item, the customer can close the door. From the moment the cabinet door is opened until it is closed, the cabinet door image acquisition device continuously captures images of the cabinet interior to obtain door images and the corresponding door angle value for each image. The display cabinet then uses these images and angle values ​​to determine the depth information of items located in the target area within the door image. This target area corresponds to the aisles within the display cabinet. Based on the depth information, the system determines how many items can still be accommodated in at least one aisle, i.e., the location information of at least one unoccupied display space within the cabinet. The display cabinet then sends this unoccupied display space location information to a cloud-based statistics server. The server can then perform statistical analysis based on this information and generate corresponding billing information. This information can be used to deduct fees from customer accounts or to generate item allocation plans for the display cabinet.

[0107] For example, let's illustrate this using a display case to provide unmanned vending services. When maintenance personnel, such as convenience store clerks, need to replenish the display case, they can use a mobile communication terminal to scan the QR code on the surface of the display case to access the corresponding cloud authorization server and send a replenishment request to the server. Upon receiving the authorization request, the cloud server sends authorization information to the display case, granting permission for the maintenance personnel to replenish the item. The display case receives the authorization information from the cloud server and unlocks its door in response. Once unlocked, the maintenance personnel can open the door and insert the scanned item into the display case through the item inlet / outlet. After replenishing the item, the maintenance personnel can close the door. From the moment the cabinet door is opened until it is closed, the cabinet door image acquisition device continuously captures images of the cabinet interior to obtain door images and acquires the door angle value corresponding to each image. The display cabinet then uses these images and angle values ​​to determine the depth information of items located in the target area within the door image. This target area corresponds to the aisles within the display cabinet. Based on the depth information, the system determines how many items can still be accommodated in at least one aisle, i.e., the location information of at least one vacant display position within the cabinet. The display cabinet then sends this vacant display position location information to a cloud-based statistics server. This allows the server to evaluate maintenance work based on this information or generate further item allocation plans for the display cabinet.

[0108] For example, consider a display case used to provide item retrieval services to a target user, where the target user can be understood as a user belonging to a specific unit or department. When a user needs to purchase an item from the display case, they can use a mobile communication terminal to scan the QR code on the surface of the display case to access the server corresponding to the display case and send an identity authentication request to the server, i.e., requesting authentication of the user's identity; alternatively, they can use a corresponding identification (such as an employee badge, ID card, etc.) to scan the QR code on the display case, which will then send an identity authentication request to the corresponding server. When the display case receives item retrieval authorization information from the server, it can be understood that the server has confirmed the user as the target user, and the display case can unlock its door in response to the item retrieval authorization information. After the door is unlocked, the user can open the door and retrieve the item from the display case through the item access window. After retrieving the item, the user can close the door. From the moment the cabinet door is opened until it is closed, the cabinet door image acquisition device continuously captures images of the cabinet interior to obtain door images and the corresponding door angle value for each image. The display cabinet then uses these images and angle values ​​to determine the depth information of items located in the target area within the door image. This target area corresponds to the aisles within the display cabinet. Based on the depth information, the system determines how many items can still be accommodated in at least one aisle, i.e., the location information of at least one unoccupied display space within the cabinet. The display cabinet then sends this unoccupied display space location information to a cloud-based statistics server, enabling the server to perform statistical analysis or generate appropriate item allocation plans for the display cabinet.

[0109] In the above scheme, at least one cabinet door image is acquired by a cabinet door image acquisition device installed at the cabinet door of the display case, and the cabinet door angle value corresponding to the angle between the cabinet door and the cabinet body when the cabinet door image is acquired. The depth information of the items located in the target area in the cabinet door image is obtained based on the cabinet door image and the cabinet door angle value. Since the depth information of the items in the cabinet door image can be obtained with high accuracy based on the cabinet door image acquired at the corresponding cabinet door angle, the depth information is used to indicate the distance of the items in the cabinet door image to the item image acquisition device of the cabinet door image acquisition device. Then, the location information of at least one empty display position in the cabinet that does not contain an item is obtained based on the depth information. Since the distance of the cabinet door image acquisition device to the display position image acquisition device of the corresponding display position at the corresponding cabinet door angle can be regarded as known, the location information of the empty display position in the cabinet can be determined by the distance of the item image acquisition device and the distance of the display position image acquisition device. In the above solution, since the depth information obtained has a high accuracy rate, the accuracy of the vacant display location obtained based on the depth information is also high. Furthermore, since only the cabinet door image acquisition device set at the cabinet door is required to acquire the cabinet door image, fewer cabinet door image acquisition devices are needed, reducing the cost of the display cabinet. Therefore, the above solution achieves the goal of determining the vacant display location in the display cabinet with a low cost and a high accuracy rate.

[0110] In one implementation of this disclosure, the number of cabinet door images is greater than or equal to one;

[0111] In step S102, the depth information of the items located in the target area in the cabinet door image is obtained based on the cabinet door image and the included angle value of the cabinet door. This can be achieved through the following steps:

[0112] A pre-trained first depth information estimation model is obtained. The cabinet door images and the cabinet door angle values ​​corresponding to each cabinet door image are input into the first depth information estimation model to obtain the depth information of the items located in the target area output by the first depth information estimation model.

[0113] In one embodiment of this disclosure, the first depth information estimation model may be pre-stored in a display case or obtained from other devices or systems. The first depth information estimation model may be a neural network (NN) model, a convolutional neural network (CNN) model, or a long short-term memory (LSTM) model, etc.

[0114] In the above scheme, by obtaining a pre-trained first depth information estimation model, the cabinet door image and the cabinet door angle value corresponding to each cabinet door image are input into the first depth information estimation model to obtain the depth information of the item located in the target area output by the first depth information estimation model, which can improve the accuracy of the obtained depth information.

[0115] In one implementation of this disclosure, the method further includes the following steps:

[0116] The number of available front spaces for the aisles inside the cabinet is determined based on the available display space.

[0117] Image recognition is performed on at least one cabinet door image to obtain the empty space marking recognition result, and the number of empty spaces in the image recognition front of at least one cargo aisle inside the cabinet is obtained based on the empty space marking recognition result.

[0118] The empty space marking recognition result is used to indicate the empty space marking on the surface of each cargo aisle inside the cabinet that is used to carry items.

[0119] In response to a mismatch between the number of empty spaces in front of the target cargo channel and the number of empty spaces in front of the target cargo channel, the target depth information corresponding to the target cargo channel is obtained based on the number of empty spaces in front of the target cargo channel.

[0120] The cabinet door image is used as input, and the target depth information is used as output to train the first depth information estimation model.

[0121] In one embodiment of this disclosure, the number of empty spaces in front of the cargo aisle can be understood as the number of positions between the nearest item to the cabinet door and the cabinet door that the cargo aisle can still hold (i.e., positions that the cargo aisle can still be used to hold items).

[0122] In one embodiment of this disclosure, a vacancy marker can be understood as being placed at a corresponding position on the surface of the cargo channel used to hold items, and having a corresponding pattern. By performing image recognition on the corresponding pattern, a corresponding vacancy marker recognition result can be obtained, which can indicate the location of the corresponding position, i.e., the location of the vacancy marker. The vacancy marker recognition for different vacancy positions within the same cargo channel can be different; furthermore, the vacancy marker recognition for different vacancy positions within different cargo channels can be different. The vacancy marker can be affixed to the surface of the cargo channel used to hold items, or it can be understood as being integrally formed with the cargo channel.

[0123] In one embodiment of this disclosure, the number of empty spaces in front of the image recognition of the cargo lane can be understood as the number of empty spaces in front of the cargo lane indicated by the empty space marking recognition result.

[0124] Specifically, when an item is placed on the surface of a certain location within a certain aisle in a cabinet door image, the empty space marker at that location will be obscured by the item, preventing the image recognition of the cabinet door from obtaining the corresponding empty space marker. However, when the surface of an aisle in a cabinet door image between the nearest item to the cabinet door and the door itself is empty, the area from the nearest item to the door can be considered a front empty space within that aisle. The empty space marker at this front empty space will not be obscured by the item, and image recognition of the cabinet door image can obtain the corresponding empty space marker. Based on this empty space marker recognition result, the number of front empty spaces in that aisle can be determined. Furthermore, the specific location of each front empty space in that aisle can also be obtained.

[0125] In one embodiment of this disclosure, performing image recognition on at least one cabinet door image to obtain a vacancy mark recognition result can be understood as performing image recognition on at least one cabinet door image based on a pre-acquired vacancy mark recognition algorithm to obtain a vacancy mark recognition result, or it can be understood as acquiring a pre-trained vacancy mark recognition model, inputting at least one cabinet door image into the vacancy mark recognition model, and obtaining the vacancy mark recognition result output by the vacancy mark recognition model.

[0126] In one embodiment of this disclosure, the number of image recognition empty spaces in front of the target cargo channel does not match the number of empty spaces in front of the target cargo channel. This can be understood as the number of image recognition empty spaces in front of the target cargo channel being different from the number of empty spaces in front of the target cargo channel. It can also be understood as the difference between the number of image recognition empty spaces in front of the target cargo channel and the number of empty spaces in front of the target cargo channel being greater than or equal to the threshold of the difference in the number of empty spaces in front of the target cargo channel.

[0127] In one embodiment of this disclosure, the target depth information corresponding to the target cargo channel can be understood as the distance from the nearest item to the cabinet door in the target cargo channel to the cabinet door image acquisition device.

[0128] In one embodiment of this disclosure, obtaining target depth information corresponding to the target cargo channel based on the number of empty spaces in front of the image recognition can be understood as determining the position of the nearest item to the cabinet door in the target cargo channel based on the number of empty spaces in front of the image recognition, obtaining the distance from the nearest item to the cabinet door image acquisition device based on the position of the nearest item and the cabinet door angle value, and obtaining target depth information corresponding to the target cargo channel based on the distance from the nearest item to the cabinet door image acquisition device.

[0129] In the above scheme, the number of empty spaces in front of the aisles inside the cabinet is obtained based on the vacant display positions, and image recognition is performed on at least one cabinet door image to obtain the empty space marking recognition result. Based on the empty space marking recognition result, the number of empty spaces in front of at least one aisle inside the cabinet is obtained through image recognition. Since only the empty space markings in the cabinet door image need to be recognized when obtaining the empty space marking recognition result, the recognition difficulty is lower than obtaining the depth information of the items in the cabinet door image. Therefore, the accuracy of the number of empty spaces in front of the image recognition is higher than the accuracy of the number of empty spaces in front of the target aisle. When the number of empty spaces in front of the image recognition of the target aisle does not match the number of empty spaces in front of the target aisle, it can be considered that the error in the number of empty spaces in front of the target aisle is due to inaccurate depth information. To improve the accuracy of subsequently acquired depth information, a method is used to address the mismatch between the number of empty spaces in front of the target cargo channel and the number of empty spaces in front of the target cargo channel. Based on the number of empty spaces in front of the target cargo channel with higher accuracy, target depth information with higher accuracy corresponding to the target cargo channel is acquired. The cabinet door image is used as input, and the target depth information with higher accuracy is used as output to train the first depth information estimation model. This allows the first depth information estimation model to learn the pattern between the depth information of the cabinet door image and the items in the cabinet door image that it could not learn before, ensuring that the accuracy of the depth information acquired subsequently is higher.

[0130] In one implementation of this disclosure, before training the first depth information estimation model by taking the cabinet door image as input and the target depth information as output, the method further includes the following steps:

[0131] Receive the first update weight parameters sent by the first edge server, and update the first depth information estimation model according to the first update weight parameters;

[0132] Using the cabinet door image as input and the target depth information as output, the first depth information estimation model can be trained through the following steps:

[0133] The cabinet door image is used as input, and the target depth information is used as output to train the updated first depth information estimation model.

[0134] The method also includes the following steps:

[0135] In response to the convergence of the first depth information estimation model after training, the first gradient update vector is obtained based on the first depth information estimation model after training, and the first gradient update vector is sent to the first edge server.

[0136] Alternatively, in response to the convergence of the first depth information estimation model after training, the first depth information estimation model after training is stored as the target first depth information estimation model.

[0137] In one implementation of this disclosure, a first edge server aggregates gradient update vectors and updates the weight parameters of a first depth estimation model on the first edge server based on the aggregated gradient update vectors to obtain updated weight parameters. The first edge server can be a cloud server or a server provided by a display case control service provider. It should be noted that one first edge server can correspond to one or more display cases. For example, a display case control service provider can divide its managed area into multiple blocks, and multiple display cases in each block can correspond to one first edge server.

[0138] The first depth estimation model on the first edge server can be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.

[0139] In one implementation of this disclosure, the updated weight parameters sent by the first edge server and received by the display cabinet are obtained by the first edge server aggregating the first gradient update vectors sent by multiple display cabinets and updating the weight parameters of the first depth estimation model on the first edge server according to the aggregated gradient update vectors. Therefore, the updated first depth estimation model on the display cabinet can reflect the common pattern between the depth information of the cabinet door image and the items in the cabinet door image that the first depth estimation model on the first edge server learned in the previous training. Then, using the cabinet door image as input and the target depth information as output, the updated first depth estimation model is trained. This allows the updated first depth estimation model on the display cabinet to learn not only common patterns but also personalized patterns between the depth information of the cabinet door image and the items within it. This enables the trained first depth estimation model on the display cabinet to learn the unique patterns between the cabinet door image and the items within it. If the trained first depth estimation model fails to converge, it indicates that further training is needed. The first depth estimation model on the display case obtains and sends the gradient update vector, which enables the first edge server to continue to obtain the corresponding update weight parameters based on the gradient update vectors uploaded by multiple display cases, thereby continuing to train the first depth estimation model on each display case. When the trained first depth estimation model on the display case converges, it can be considered that the converged first depth estimation model on the display case can accurately identify the depth information of the items in the cabinet door image it has acquired. The converged first depth estimation model on the display case can be stored as the target first depth estimation model, that is, a model with high accuracy in identifying the depth information of the items in the cabinet door image.

[0140] In the above technical solution, on the one hand, the final target first depth estimation model can be a model that learns both common rules and private rules, and its accuracy in recognizing the depth information of items in the cabinet door image is relatively high; on the other hand, since the process of continuing to train the first depth estimation model on each display cabinet is jointly executed by the display cabinet and the first edge server, compared with the display cabinet or server alone further training the first depth estimation model, the required processing resources are less and the training speed is faster.

[0141] In one implementation of this disclosure, the number of cabinet door images is greater than or equal to two;

[0142] In step S102, the depth information of the items located in the target area in the cabinet door image is obtained based on the cabinet door image and the included angle value of the cabinet door. This can be achieved through the following steps:

[0143] Obtain the second depth information estimation model corresponding to the included angle value of each cabinet door in the pre-trained model. Input the included angle value of the cabinet door and the cabinet door image corresponding to the included angle value into the second depth information estimation model corresponding to the included angle value of the cabinet door to obtain the sub-depth information of at least one item in the cabinet door image output by each second depth information estimation model.

[0144] The depth information of the items located in the target area is obtained by estimating the sub-depth information output by the model based on each second depth information.

[0145] In one embodiment of this disclosure, the second depth information estimation model can be pre-stored in a display case or obtained from other devices or systems. The second depth information estimation model can be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.

[0146] In one embodiment of this disclosure, the depth information of an item located in the target area is obtained based on the sub-depth information output by each second depth information estimation model. This can be understood as taking the average of multiple sub-depth information to obtain the depth information of the item located in the target area, or it can be understood as taking the median value of multiple sub-depth information to obtain the depth information of the item located in the target area.

[0147] In the above scheme, the pre-trained second depth information estimation model corresponding to each cabinet door angle value can be understood as a model that has learned the relationship between the depth information of the cabinet door image and the items in the cabinet door image for the corresponding cabinet door angle value. Therefore, by obtaining the pre-trained second depth information estimation model corresponding to each cabinet door angle value, the cabinet door angle value and the cabinet door image corresponding to the cabinet door angle value are input into the second depth information estimation model corresponding to the cabinet door angle value to obtain the sub-depth information of at least one item in the cabinet door image output by each second depth information estimation model. This information ensures that the sub-depth information accurately reflects the distance between the object in the target area and the cabinet door image device in the cabinet door image acquired at the corresponding cabinet door angle value. Then, by obtaining the depth information of the object in the target area based on the sub-depth information output by each second depth information estimation model, it can be ensured that the obtained depth information can reflect the distance between the object in the target area and the cabinet door image device in cabinet door images acquired based on multiple different cabinet door angle values. This avoids the low accuracy of depth information due to blurry or poor quality of a certain cabinet door image, ensuring that the accuracy of depth information is high and relatively stable.

[0148] In one implementation of this disclosure, the method further includes the following steps:

[0149] The number of available front spaces for the aisles inside the cabinet is determined based on the available display space.

[0150] Image recognition is performed on at least one cabinet door image to obtain the empty space marking recognition result, and the number of empty spaces in the image recognition front of at least one cargo aisle inside the cabinet is obtained based on the empty space marking recognition result.

[0151] The empty space marking recognition result is used to indicate the empty space marking on the surface of each cargo aisle inside the cabinet that is used to carry items.

[0152] In response to a mismatch between the number of empty spaces in front of the target cargo channel and the number of empty spaces in front of the target cargo channel, the target depth information corresponding to the target cargo channel is obtained based on the number of empty spaces in front of the target cargo channel.

[0153] The cabinet door angle value and the corresponding cabinet door image are used as inputs to the second depth information estimation model corresponding to the cabinet door angle value, and the target depth information is used as the output. The second depth information estimation model is trained.

[0154] In one embodiment of this disclosure, the number of empty spaces in front of the cargo aisle can be understood as the number of positions between the nearest item to the cabinet door and the cabinet door that the cargo aisle can still hold (i.e., positions that the cargo aisle can still be used to hold items).

[0155] In one embodiment of this disclosure, a vacancy marker can be understood as being placed at a corresponding position on the surface of the cargo channel used to hold items, and having a corresponding pattern. By performing image recognition on the corresponding pattern, a corresponding vacancy marker recognition result can be obtained, which can indicate the location of the corresponding position, i.e., the location of the vacancy marker. The vacancy marker recognition for different vacancy positions within the same cargo channel can be different; furthermore, the vacancy marker recognition for different vacancy positions within different cargo channels can be different. The vacancy marker can be affixed to the surface of the cargo channel used to hold items, or it can be understood as being integrally formed with the cargo channel.

[0156] In one embodiment of this disclosure, the number of empty spaces in front of the image recognition of the cargo lane can be understood as the number of empty spaces in front of the cargo lane indicated by the empty space marking recognition result.

[0157] Specifically, when an item is placed on the surface of a certain location within a certain aisle in a cabinet door image, the empty space marker at that location will be obscured by the item, preventing the image recognition of the cabinet door from obtaining the corresponding empty space marker. However, when the surface of an aisle in a cabinet door image between the nearest item to the cabinet door and the door itself is empty, the area from the nearest item to the door can be considered a front empty space within that aisle. The empty space marker at this front empty space will not be obscured by the item, and image recognition of the cabinet door image can obtain the corresponding empty space marker. Based on this empty space marker recognition result, the number of front empty spaces in that aisle can be determined. Furthermore, the specific location of each front empty space in that aisle can also be obtained.

[0158] In one embodiment of this disclosure, performing image recognition on at least one cabinet door image to obtain a vacancy mark recognition result can be understood as performing image recognition on at least one cabinet door image based on a pre-acquired vacancy mark recognition algorithm to obtain a vacancy mark recognition result, or it can be understood as acquiring a pre-trained vacancy mark recognition model, inputting at least one cabinet door image into the vacancy mark recognition model, and obtaining the vacancy mark recognition result output by the vacancy mark recognition model.

[0159] In one embodiment of this disclosure, the number of image recognition empty spaces in front of the target cargo channel does not match the number of empty spaces in front of the target cargo channel. This can be understood as the number of image recognition empty spaces in front of the target cargo channel being different from the number of empty spaces in front of the target cargo channel. It can also be understood as the difference between the number of image recognition empty spaces in front of the target cargo channel and the number of empty spaces in front of the target cargo channel being greater than or equal to the threshold of the difference in the number of empty spaces in front of the target cargo channel.

[0160] In one embodiment of this disclosure, the target depth information corresponding to the target cargo channel can be understood as the distance from the nearest item to the cabinet door in the target cargo channel to the cabinet door image acquisition device.

[0161] In one embodiment of this disclosure, obtaining target depth information corresponding to the target cargo channel based on the number of empty spaces in front of the image recognition can be understood as determining the position of the nearest item to the cabinet door in the target cargo channel based on the number of empty spaces in front of the image recognition, obtaining the distance from the nearest item to the cabinet door image acquisition device based on the position of the nearest item and the cabinet door angle value, and obtaining target depth information corresponding to the target cargo channel based on the distance from the nearest item to the cabinet door image acquisition device.

[0162] In the above scheme, the number of empty spaces in front of the aisles inside the cabinet is obtained based on the vacant display positions, and image recognition is performed on at least one cabinet door image to obtain the empty space marking recognition result. Based on the empty space marking recognition result, the number of empty spaces in front of at least one aisle inside the cabinet is obtained through image recognition. Since only the empty space markings in the cabinet door image need to be recognized when obtaining the empty space marking recognition result, the recognition difficulty is lower than obtaining the depth information of the items in the cabinet door image. Therefore, the accuracy of the number of empty spaces in front of the image recognition is higher than the accuracy of the number of empty spaces in front of the target aisle. When the number of empty spaces in front of the image recognition of the target aisle does not match the number of empty spaces in front of the target aisle, it can be considered that the error in the number of empty spaces in front of the target aisle is due to inaccurate depth information. To improve the accuracy of the subsequently acquired depth information, a second depth information estimation model is trained by responding to a mismatch between the number of empty spaces in front of the target cargo channel and the number of empty spaces in front of the target cargo channel. This model obtains the target depth information corresponding to the target cargo channel based on the number of empty spaces in front of the target cargo channel with higher accuracy. The cabinet door image is used as input, and the target depth information with higher accuracy is used as output. This allows the second depth information estimation model to learn the relationship between the depth information of the cabinet door image and the items in the cabinet door image that it could not learn before, ensuring that the accuracy of the subsequently acquired depth information is higher.

[0163] In one implementation of this disclosure, the cabinet door angle value and the cabinet door image corresponding to the cabinet door angle value are used as inputs to a second depth information estimation model corresponding to the cabinet door angle value, and the target depth information is used as the output. Before training each second depth information estimation model, the method further includes the following steps.

[0164] Receive the second update weight parameters corresponding to each second depth information estimation model sent by the second edge server, and update each second depth information estimation model according to the corresponding second update weight parameters;

[0165] The cabinet door angle value and the corresponding cabinet door image are used as inputs to the second depth information estimation model corresponding to the cabinet door angle value, and the target depth information is used as the output. The training of each second depth information estimation model can be achieved through the following steps:

[0166] The cabinet door angle value and the cabinet door image corresponding to the cabinet door angle value are used as inputs to the second depth information estimation model corresponding to the cabinet door angle value, and each updated second depth information estimation model is trained.

[0167] The method also includes the following steps:

[0168] In response to the convergence of the trained second depth information estimation model, the second gradient update vector corresponding to the trained second depth information estimation model is obtained according to the trained second depth information estimation model, and the second gradient update vector is sent to the first edge server.

[0169] Alternatively, in response to the convergence of the trained second depth information estimation model, the trained second depth information estimation model is stored as the target second depth information estimation model.

[0170] In one implementation of this disclosure, a second edge server is used to aggregate the second gradient update vector and update the weight parameters of the second depth estimation model on the second edge server according to the aggregated gradient update vector to obtain updated weight parameters. The second edge server can be a cloud server or a server provided by a display case control service provider. It should be noted that one second edge server can correspond to one or more display cases. For example, a display case control service provider can divide its managed area into multiple blocks, and multiple display cases in each block can correspond to one second edge server.

[0171] The second depth estimation model on the second edge server can be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.

[0172] In one implementation of this disclosure, the updated weight parameters sent by the second edge server received by the display cabinet are obtained by the second edge server aggregating the second gradient update vectors sent by multiple display cabinets and updating the weight parameters of the second depth estimation model on the second edge server according to the aggregated gradient update vectors. Therefore, the updated second depth estimation model on the display cabinet can reflect the common pattern between the depth information of the cabinet door image and the items in the cabinet door image that the second depth estimation model on the second edge server learned in the previous training. Then, using the cabinet door image as input and the target depth information as output, the updated second depth estimation model is trained. This allows the updated second depth estimation model on the display cabinet to learn not only common patterns but also personalized patterns between the depth information of the cabinet door image and the items within it. This enables the trained second depth estimation model on the display cabinet to learn the unique patterns between the cabinet door image and the items within it. If the trained second depth estimation model fails to converge, it indicates that further training is needed. The second depth estimation model on the display case obtains and sends the second gradient update vector, enabling the second edge server to continue obtaining the corresponding update weight parameters based on the second gradient update vectors uploaded by multiple display cases, thereby continuing to train the second depth estimation model on each display case. When the trained second depth estimation model on the display case converges, it can be considered that the converged second depth estimation model on the display case can accurately identify the depth information of the items in the cabinet door image it has acquired. The converged second depth estimation model on the display case can be stored as the target second depth estimation model, that is, a model with high accuracy in identifying the depth information of the items in the cabinet door image.

[0173] In the above technical solution, on the one hand, the final target second depth estimation model can be a model that learns both common rules and private rules, and its accuracy in recognizing the depth information of items in the cabinet door image is relatively high; on the other hand, since the process of continuing to train the second depth estimation model on each display cabinet is jointly executed by the display cabinet and the second edge server, compared with the display cabinet or server alone training the second depth estimation model, the required processing resources are less and the training speed is faster.

[0174] In one implementation of this disclosure, in step S101, obtaining the cabinet door angle value corresponding to each cabinet door image can be achieved through the following steps:

[0175] Obtain a pre-trained cabinet door angle estimation model, input each cabinet door image into the cabinet door angle estimation model, and obtain the cabinet door angle value output by the cabinet door angle estimation model corresponding to each cabinet door image;

[0176] Alternatively, obtain the cabinet door image acquisition time for each cabinet door image, and obtain the cabinet door angle value acquired by the angle sensor at the cabinet door image acquisition time for each cabinet door image. The angle sensor is used to acquire the angle value between the cabinet door and the cabinet body.

[0177] In one embodiment of this disclosure, the cabinet door angle estimation model can be pre-stored in the display cabinet or obtained from other devices or systems. The cabinet door angle estimation model can be a neural network (NN) model, a convolutional neural network (CNN) model, or a long short-term memory (LSTM) model, etc.

[0178] In one embodiment of this disclosure, the angle sensing device can be installed on the side wall of the cabinet door near the cabinet body, or on the side wall of the cabinet body near the cabinet door, or on the door hinge connecting the cabinet door and the cabinet body, wherein the cabinet door is rotatably connected to the cabinet body through the door hinge. This application does not impose specific restrictions on the specific installation position of the angle sensing device.

[0179] In one embodiment of this disclosure, the angle sensing device can be a photoelectric coded angle sensor or a magnetic coded angle sensor. Alternatively, the angle sensing device installed on the cabinet can measure the distance from itself to the cabinet door and obtain the included angle value of the cabinet door based on the distance from itself to the door hinge. Or, the angle sensing device installed on the cabinet door can measure the distance from itself to the cabinet and obtain the included angle value of the cabinet door based on the distance from itself to the door hinge. This application does not impose specific limitations on the method of angle measurement by the angle sensing device.

[0180] In the above scheme, by obtaining a pre-trained cabinet door angle estimation model, each cabinet door image is input into the cabinet door angle estimation model to obtain the cabinet door included angle value output by the cabinet door angle estimation model corresponding to each cabinet door image, or by obtaining the cabinet door image acquisition time of each cabinet door image and obtaining the cabinet door included angle value acquired by the angle sensor at the cabinet door image acquisition time of each cabinet door image, it can be ensured that the obtained cabinet door included angle value is relatively accurate.

[0181] In one implementation of this disclosure, the cabinet door image is a video frame image in the cabinet door video, and the cabinet door video is acquired by the cabinet door image acquisition device in response to the cabinet door unlocking to start recording and in response to the cabinet door locking to end recording.

[0182] In one embodiment of this disclosure, cabinet door unlocking can be understood as the cabinet door locking device on the display cabinet responding to an unlocking command to unlock the cabinet door, allowing the cabinet door to rotate or slide relative to the cabinet body of the display cabinet to open the item entrance / exit of the display cabinet; it can also be understood as the cabinet door locking device on the display cabinet being triggered by a corresponding unlocking operation, causing the cabinet door locking device itself to be set to an unlocked state, thereby allowing the cabinet door to rotate or slide relative to the cabinet body of the display cabinet to open the item entrance / exit of the display cabinet, through which users can move items out of the item display area or move items into the item display area.

[0183] In one embodiment of this disclosure, determining whether a cabinet door is locked can be achieved by receiving cabinet door locking status information sent by a cabinet door locking device on the display cabinet, and determining whether the cabinet door is locked based on the cabinet door locking status information. Alternatively, cabinet door locking status information sent by other devices or systems can be received, and the cabinet door can be determined based on the cabinet door locking status information. For example, an infrared detection device independent of the display cabinet can detect whether the cabinet door is locked, and send cabinet door locking status information to the display cabinet based on the detection result; alternatively, a camera independent of the display cabinet, such as a security camera, can acquire images including all or part of the display cabinet, perform image recognition based on the images, and send cabinet door locking status information to the display cabinet based on the image recognition.

[0184] In this embodiment, considering that items can only be moved out of or into the display case when the cabinet door is unlocked, by limiting the cabinet door image to video frame images in the cabinet door video, and the cabinet door video being acquired by the cabinet door image acquisition device in response to the cabinet door unlocking and in response to the cabinet door locking, the probability of the cabinet door image including the items moved out or into the display case can be increased, the number of images that need to be processed can be reduced, and data processing resources can be saved.

[0185] Figure 10 A flowchart illustrating the overall process of a display case control method according to an embodiment of the present disclosure is shown, as follows: Figure 10 As shown, the display case control methods include:

[0186] In step S201, at least one cabinet door image is acquired.

[0187] In step S202, a pre-trained cabinet door angle estimation model is obtained, and each cabinet door image is input into the cabinet door angle estimation model to obtain the cabinet door angle value output by the cabinet door angle estimation model corresponding to each cabinet door image.

[0188] In step S203, the cabinet door image acquisition time for each cabinet door image is obtained, and the cabinet door included angle value acquired by the angle sensor at the cabinet door image acquisition time for each cabinet door image is obtained.

[0189] In step S204, a pre-trained first depth information estimation model is obtained. The cabinet door images and the cabinet door angle values ​​corresponding to each cabinet door image are input into the first depth information estimation model to obtain the depth information of the items located in the target area output by the first depth information estimation model.

[0190] In step S205, a second depth information estimation model corresponding to each pre-trained cabinet door angle value is obtained. The cabinet door angle value and the cabinet door image corresponding to the cabinet door angle value are input into the second depth information estimation model corresponding to the cabinet door angle value to obtain the sub-depth information of the items located in the target area in the cabinet door image output by each second depth information estimation model.

[0191] In step S206, the depth information of the items located in the target area is obtained based on the sub-depth information output by each second depth information estimation model.

[0192] In step S207, the location information of at least one empty display position inside the cabinet that does not contain any items is obtained based on the depth information.

[0193] In step S208, the number of available spaces in front of the display aisles inside the cabinet is obtained based on the available display locations.

[0194] In step S209, image recognition is performed on at least one cabinet door image to obtain the empty space marking recognition result, and the number of empty spaces in the image recognition front of at least one cargo aisle inside the cabinet is obtained based on the empty space marking recognition result.

[0195] In step S210, in response to the mismatch between the number of image recognition front empty spaces of the target cargo channel and the number of front empty spaces of the target cargo channel, the target depth information corresponding to the target cargo channel is obtained based on the number of image recognition front empty spaces.

[0196] In step S211, the first update weight parameter sent by the first edge server is received, and the first depth information estimation model is updated according to the first update weight parameter.

[0197] In step S212, the cabinet door image is used as input and the target depth information is used as output to train the updated first depth information estimation model.

[0198] In step S213, in response to the convergence of the trained first depth information estimation model, the first gradient update vector is obtained based on the trained first depth information estimation model and sent to the first edge server; or, in response to the convergence of the trained first depth information estimation model, the trained first depth information estimation model is stored as the target first depth information estimation model.

[0199] In step S214, the second update weight parameters corresponding to each second depth information estimation model sent by the second edge server are received, and each second depth information estimation model is updated according to the corresponding second update weight parameters.

[0200] In step S215, the cabinet door angle value and the cabinet door image corresponding to the cabinet door angle value are used as inputs to the second depth information estimation model corresponding to the cabinet door angle value, and each updated second depth information estimation model is trained.

[0201] In step S216, in response to the convergence of the trained second depth information estimation model, the second gradient update vector is obtained based on the trained second depth information estimation model and sent to the second edge server; or, in response to the convergence of the trained second depth information estimation model, the trained second depth information estimation model is stored as the target second depth information estimation model.

[0202] This disclosure also discloses an electronic device, Figure 11 A schematic structural block diagram of an electronic device according to an embodiment of the present disclosure is shown, such as... Figure 11 As shown, the electronic device 700 includes a memory 701 and a processor 702; wherein the memory 701 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 702 to implement the above method steps.

[0203] Figure 12 This is a schematic diagram of the structure of a computer system suitable for implementing the display case control method according to an embodiment of the present disclosure. For example... Figure 12 As shown, the computer system 800 includes a processing unit 801, which can execute various processes described above based on a program stored in a read-only memory (ROM) 802 or a program loaded from a storage section 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0204] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed. The processing unit 801 can be implemented as a CPU, GPU, TPU, FPGA, NPU, etc.

[0205] In particular, according to embodiments of this disclosure, the methods described above with reference to the accompanying drawings can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a readable medium thereof, the computer program containing program code for performing the methods in the drawings. In such embodiments, the computer program can be downloaded and installed from a network via a communication section 809, and / or installed from a removable medium 811. For example, embodiments of this disclosure include a readable storage medium storing computer instructions that, when executed by a processor, implement program code for performing the methods in the drawings.

[0206] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0207] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The described units or modules can also be located in a processor, and the names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0208] In another aspect, this disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the apparatus described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs that are used by one or more processors to perform the methods described in this disclosure.

[0209] In addition, this disclosure also provides a computer program product storing a computer program that, when executed by a processor, enables the processor to at least implement the methods provided in the foregoing embodiments.

[0210] Figure 13 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown. Figure 14 A schematic top view of a display case according to one embodiment of the present disclosure is shown. Figure 13 as well as Figure 14 As shown, the display case 900 includes a cabinet body 901, a cabinet door 902, a first cabinet body image acquisition device 903, a second cabinet body image acquisition device 904, a cabinet door image acquisition device 905, and a processing device.

[0211] Cabinet door 902 is rotatably connected to cabinet body 901 and is used to open or close the item access entrance 911 of cabinet body 901;

[0212] The cabinet 901 includes an inner cavity 921, which is connected to the outside of the cabinet 901 through an item inlet / outlet 911. The inner cavity 921 is used to store items.

[0213] The first cabinet image acquisition device 903 and the second cabinet image acquisition device 904 are both connected to the top surface of the cabinet's inner cavity 921. The first cabinet image acquisition device 903 and the second cabinet image acquisition device 904 are used to acquire images of the item entrance / exit 911 from different directions.

[0214] The cabinet door image acquisition device 905 is connected to the side of the cabinet door 902 closest to the cabinet body 901, and the position of the cabinet door image acquisition device 905 matches the position of the door handle 912 of the cabinet door 902. The door handle 912 is connected to the side of the cabinet door 902 away from the cabinet body 901.

[0215] The processing device is communicatively connected to the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device. The processing device is used to execute the method provided in the foregoing embodiments.

[0216] In one embodiment of this disclosure, collecting images of the item entry / exit points can be understood as collecting images of all or part of the item entry / exit points. Through the collected images, it is possible to determine whether the user moves items out of the cabinet cavity from the item entry / exit points or whether the user moves items into the cabinet cavity from the item entry / exit points.

[0217] In one embodiment of this disclosure, the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device can be cameras or other devices with image acquisition functions.

[0218] In one embodiment of this disclosure, the processing apparatus may include one or more processing units, such as an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a memory, a video codec, a digital signal processor, a baseband processor, and / or a neural network processor. The different processing units may be independent devices or integrated into one or more processors.

[0219] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A display case control method, characterized in that, The method includes: Acquire at least one cabinet door image and the cabinet door angle value corresponding to each cabinet door image. The cabinet door image is acquired by a cabinet door image acquisition device, which is installed at the cabinet door of the display cabinet. The cabinet door angle value is the angle between the cabinet door and the cabinet body of the display cabinet when the cabinet door image is acquired. The depth information of the items located in the target area in the cabinet door image is obtained based on the cabinet door image and the included angle value of the cabinet door; Based on the depth information, obtain the location information of at least one empty display position within the cabinet that does not contain any items; The number of cabinet door images is greater than or equal to one; The step of obtaining the depth information of the item located in the target area in the cabinet door image based on the cabinet door image and the included angle value of the cabinet door includes: A pre-trained first depth information estimation model is obtained, and the cabinet door images and the cabinet door angle values ​​corresponding to each cabinet door image are input into the first depth information estimation model to obtain the depth information of the item located in the target area output by the first depth information estimation model. The method further includes: The number of empty spaces in front of the aisles inside the cabinet is obtained based on the available display locations. Image recognition is performed on at least one cabinet door image to obtain empty space marking recognition results. Based on the empty space marking recognition results, the number of empty spaces in front of the image recognition of at least one cargo aisle in the cabinet is obtained. The empty space marking recognition results are used to indicate the empty space markings on the surface of each cargo aisle in the cabinet that is used to carry items. In response to a mismatch between the number of empty spaces in front of the target cargo channel and the number of empty spaces in front of the target cargo channel, target depth information corresponding to the target cargo channel is obtained based on the number of empty spaces in front of the target cargo channel. The cabinet door image is used as input, and the target depth information is used as output to train the first depth information estimation model.

2. The display case control method according to claim 1, characterized in that, Before training the first depth information estimation model by taking the cabinet door image as input and the target depth information as output, the method further includes: Receive the first update weight parameters sent by the first edge server, and update the first depth information estimation model according to the first update weight parameters; The step of using the cabinet door image as input and the target depth information as output to train the first depth information estimation model includes: The cabinet door image is used as input, and the target depth information is used as output to train the updated first depth information estimation model. The method further includes: In response to the convergence of the first depth information estimation model after training, a first gradient update vector is obtained based on the first depth information estimation model after training, and the first gradient update vector is sent to the first edge server. Alternatively, in response to the convergence of the trained first depth information estimation model, the trained first depth information estimation model is stored as the target first depth information estimation model.

3. The display case control method according to claim 1, characterized in that, The number of cabinet door images is greater than or equal to two; The step of obtaining the depth information of the item located in the target area in the cabinet door image based on the cabinet door image and the included angle value of the cabinet door includes: Obtain the second depth information estimation model corresponding to each pre-trained cabinet door angle value, input the cabinet door angle value and the cabinet door image corresponding to the cabinet door angle value into the second depth information estimation model corresponding to the cabinet door angle value, so as to obtain the sub-depth information of the item located in the target area in the cabinet door image output by each second depth information estimation model; The depth information of the object located in the target area is obtained based on the sub-depth information output by the model for each second depth information estimation. The method further includes: The number of empty spaces in front of the aisles inside the cabinet is obtained based on the available display locations. Image recognition is performed on at least one cabinet door image to obtain empty space marking recognition results. Based on the empty space marking recognition results, the number of empty spaces in front of the image recognition of at least one cargo aisle in the cabinet is obtained. The empty space marking recognition results are used to indicate the empty space markings on the surface of each cargo aisle in the cabinet that is used to carry items. In response to a mismatch between the number of empty spaces in front of the target cargo channel and the number of empty spaces in front of the target cargo channel, target depth information corresponding to the target cargo channel is obtained based on the number of empty spaces in front of the target cargo channel. The cabinet door angle value and the cabinet door image corresponding to the cabinet door angle value are used as inputs to the second depth information estimation model corresponding to the cabinet door angle value, and the target depth information is used as output to train each second depth information estimation model.

4. The display case control method according to claim 3, characterized in that, The method further includes taking the cabinet door angle value and the corresponding cabinet door image as input to the second depth information estimation model corresponding to the cabinet door angle value, and taking the target depth information as output. Before training each second depth information estimation model, the method also includes... Receive the second update weight parameters corresponding to each second depth information estimation model sent by the second edge server, and update each second depth information estimation model according to the corresponding second update weight parameters; The process of using the cabinet door angle value and the corresponding cabinet door image as input to the second depth information estimation model corresponding to the cabinet door angle value, and using the target depth information as output, to train each second depth information estimation model includes: The cabinet door angle value and the cabinet door image corresponding to the cabinet door angle value are used as inputs to the second depth information estimation model corresponding to the cabinet door angle value, and each updated second depth information estimation model is trained. The method further includes: In response to the convergence of the trained second depth information estimation model, a first gradient update vector corresponding to the trained second depth information estimation model is obtained according to the trained second depth information estimation model, and the first gradient update vector is sent to the first edge server. Alternatively, in response to the convergence of the trained second depth information estimation model, the trained second depth information estimation model is stored as the target second depth information estimation model.

5. The display case control method according to any one of claims 1-4, characterized in that, Obtain the included angle value of the cabinet door corresponding to each cabinet door image, including: Obtain a pre-trained cabinet door angle estimation model, input each cabinet door image into the cabinet door angle estimation model, and obtain the cabinet door included angle value output by the cabinet door angle estimation model corresponding to each cabinet door image; Alternatively, the cabinet door image acquisition time for each cabinet door image can be obtained, and the cabinet door angle value acquired by the angle sensor at the cabinet door image acquisition time for each cabinet door image can be obtained. The angle sensor is used to acquire the angle value between the cabinet door and the cabinet body.

6. The display case control method according to any one of claims 1-4, characterized in that, The cabinet door image is a video frame image from the cabinet door video, and the cabinet door video is acquired by the cabinet door image acquisition device in response to the cabinet door unlocking to start recording and in response to the cabinet door locking to end recording.

7. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method of any one of claims 1-6.

8. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by a processor, the computer instructions implement the method of any one of claims 1-6.

9. A display case, characterized in that, The display case includes a cabinet body, cabinet doors, a first cabinet body image acquisition device, a second cabinet body image acquisition device, a cabinet door image acquisition device, and a processing device. The cabinet door is rotatably connected to the cabinet body and is used to open or close the item entrance / exit of the cabinet body. The cabinet includes an internal cavity, which is connected to the outside of the cabinet through the item inlet / outlet, and is used to store items. Both the first cabinet image acquisition device and the second cabinet image acquisition device are connected to the top surface of the inner cavity of the cabinet. The first cabinet image acquisition device and the second cabinet image acquisition device are used to acquire images of the item entrance and exit from different directions respectively. The cabinet door image acquisition device is connected to the side of the cabinet door closest to the cabinet body, and the position of the cabinet door image acquisition device matches the position of the door handle of the cabinet door. The door handle is connected to the side of the cabinet door furthest from the cabinet body. The processing device is communicatively connected to the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device, and the processing device is used to execute the method of any one of claims 1-6.

Images