Display cabinet control method, apparatus, medium, and display cabinet

Abstract

The embodiment of the present disclosure discloses a showcase control method, device, medium and showcase. The method comprises: acquiring at least two to-be-recognized images of a showcase; acquiring the position information of at least one article corresponding to each to-be-recognized image in the at least two to-be-recognized images, and acquiring the motion track information of the at least one article according to the position information; acquiring the article category information of the at least one article in the at least two to-be-recognized images; and in response to the article category indicated by the article category information of a target article in the at least one article matching an alarm article category and the target motion track indicated by the motion track information of the target article satisfying an alarm motion track condition, generating alarm information. The technical solution can generate alarm information to warn the user, ensure that no article is moved into the showcase by the user arbitrarily, and ensure that the user experience of the user taking the article from the showcase is not damaged.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of control, in particular to a showcase control method, device, medium and showcase. BACKGROUND

[0002] In recent years, when a merchant or an enterprise stores goods, in order to facilitate users to understand the information of the goods, the goods can be placed in a showcase to realize storage and display of the goods at the same time. When a user needs to use the goods in the showcase, the user can open the showcase by himself and take the corresponding goods from the showcase. In this scenario, the showcase can identify the goods removed from the showcase and upload the identification result, so that other devices or systems such as a server, a cloud, etc. can determine the user who removes the goods from the showcase or the goods removed from the showcase according to the identification result uploaded by the showcase, so as to perform corresponding statistics. SUMMARY

[0003] Embodiments of the present disclosure provide a showcase control method, device, medium and showcase, which are used to solve the problem that a showcase in the related art can be moved into by a user.

[0004] In a first aspect, a showcase control method is provided in embodiments of the present disclosure.

[0005] Specifically, the showcase control method comprises:

[0006] obtaining at least two to-be-identified images of a showcase;

[0007] obtaining item position information of at least one item corresponding to each to-be-identified image in the at least two to-be-identified images, and obtaining motion trajectory information of the at least one item according to the item position information;

[0008] obtaining item category information of the at least one item in the at least two to-be-identified images;

[0009] in response to the item category information of a target item in the at least one item indicating that an item category matches an alarm item category and the motion trajectory information of the target item indicating that a target motion trajectory satisfies an alarm motion trajectory condition, generating alarm information.

[0010] In an implementation manner of the present disclosure, the obtaining of the item position information of the at least one item corresponding to each to-be-identified image in the at least two to-be-identified images comprises:

[0011] obtaining a pre-trained item position recognition model, and inputting each to-be-identified image into the item position recognition model to obtain item position information output by the item position recognition model;

[0012] The method comprises the following steps:

[0013] According to the position information of the at least one item corresponding to each image to be identified, a to-be-identified region corresponding to each item in the at least one item is determined in each image to be identified.

[0014] An item category identification model is obtained, and the to-be-identified region is input into the item category identification model to obtain item category information output by the item category identification model.

[0015] In an implementation manner of the present disclosure, the method further comprises:

[0016] In response to the alarm information, scan code information collected by a scan code device of the display cabinet is obtained, and item category correction information is obtained according to the scan code information;

[0017] In response to the fact that the item category indicated by the item category correction information does not match the target item category, the to-be-identified region corresponding to the target item is taken as input, the item category correction information is taken as output, and the item category identification model is trained.

[0018] In an implementation manner of the present disclosure, before the step of training the item category identification model in response to the fact that the item category indicated by the item category correction information does not match the target item category, the to-be-identified region corresponding to the target item is taken as input, and the item category correction information is taken as output, the method further comprises:

[0019] The first updated weight parameter sent by the first edge server is received, and the item category identification model is updated according to the first updated weight parameter;

[0020] The step of training the item category identification model in response to the fact that the item category indicated by the item category correction information does not match the target item category, the to-be-identified region corresponding to the target item is taken as input, and the item category correction information is taken as output, comprises:

[0021] The step of training the updated item category identification model in response to the fact that the item category indicated by the item category correction information does not match the target item category, the to-be-identified region corresponding to the target item is taken as input, and the item category correction information is taken as output, comprises:

[0022] The method further comprises:

[0023] In response to the fact that the trained item category identification model does not converge, a first gradient update vector is obtained according to the trained item category identification model, and the first gradient update vector is sent to the first edge server;

[0024] Or, in response to the trained item category recognition model converging, storing the trained item category recognition model as a target item category recognition model.

[0025] In an implementation manner of the present disclosure, the method further includes:

[0026] In response to the alarm information, obtaining item trajectory correction information;

[0027] In response to the motion trajectory indicated by the item trajectory correction information not matching the target motion trajectory, obtaining, according to the item trajectory correction information, correction item position information of the target item corresponding to each to-be-recognized image;

[0028] Training the item position recognition model by taking each to-be-recognized image as input and taking the correction item position information of the target item corresponding to each to-be-recognized image as output.

[0029] In an implementation manner of the present disclosure, before the training of the item position recognition model by taking each to-be-recognized image as input and taking the correction item position information of the target item corresponding to each to-be-recognized image as output, the method further includes:

[0030] Receiving a second updated weight parameter sent by a second edge server, and updating the item position recognition model according to the second updated weight parameter;

[0031] The training of the item position recognition model by taking each to-be-recognized image as input and taking the correction item position information of the target item corresponding to each to-be-recognized image as output includes:

[0032] The training of the updated item position recognition model by taking each to-be-recognized image as input and taking the correction item position information of the target item corresponding to each to-be-recognized image as output;

[0033] The method further includes:

[0034] In response to the trained item position recognition model not converging, obtaining a second gradient update vector according to the trained item position recognition model, and sending the second gradient update vector to a second edge server;

[0035] Or, in response to the trained item position recognition model converging, storing the trained item position recognition model as a target item position recognition model.

[0036] In an implementation manner of the present disclosure, the target motion trajectory includes a plurality of sub-motion trajectories corresponding to different time periods respectively;

[0037] The target motion trajectory indicated by the motion trajectory information of the target item satisfies an alarm motion trajectory condition, including:

[0038] The number of sub-motion trajectories in the target motion trajectory that satisfy the alarm motion trajectory condition is greater than or equal to a target number threshold.

[0039] In an implementation manner of the present disclosure, the alarm motion trajectory condition comprises:

[0040] The motion direction of the motion trajectory at at least one time instant matches the alarm motion direction.

[0041] And / or, the first distance is greater than the second distance, the first distance being a distance between a trajectory point of the motion trajectory at a first time instant and the article display area of the display cabinet, the second distance being a distance between a trajectory point of the motion trajectory at a second time instant and the article display area, the first time instant being earlier than the second time instant.

[0042] And / or, a position of at least one trajectory point in the motion trajectory belongs to the article display area.

[0043] In an implementation manner of the present disclosure, the article category information is used to indicate at least two article categories and a category probability corresponding to each article category.

[0044] The article category indicated by the article category information of the target article in the at least one article matches the alarm article category, comprising:

[0045] The article category indicated by the article category information of the target article comprises the alarm article category, and a category probability corresponding to the alarm article category is the highest in the category probabilities indicated by the article category information of the target article.

[0046] In an implementation manner of the present disclosure, the to-be-recognized image is a plurality of video frames in a cabinet door video, the cabinet door video is collected by a to-be-recognized image collection device in response to the cabinet door being unlocked and ending recording in response to the cabinet door being locked, and the to-be-recognized image collection device is arranged at a cabinet door of the display cabinet.

[0047] Or, the to-be-recognized image is a plurality of video frames in a cabinet body video, the cabinet body video is collected by a cabinet body image collection device in response to the cabinet door being unlocked and ending recording in response to the cabinet door being locked, and the cabinet body image collection device is arranged at a cabinet body of the display cabinet.

[0048] In a second aspect, an electronic device is provided in the embodiments of the present disclosure, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method in the first aspect or any implementation manner of the first aspect.

[0049] In a third aspect, a computer readable storage medium is provided in the embodiments of the present disclosure, and the computer readable storage medium stores computer instructions, and the computer instructions are executed by a processor to implement the method in the first aspect or any implementation manner of the first aspect.

[0050] In a fourth aspect, the embodiments of the present disclosure provide a computer program product, which comprises computer instructions, and the computer instructions, when executed by a processor, implement the method in the first aspect or any implementation manner of the first aspect.

[0051] In a fifth aspect, the embodiments of the present disclosure provide a display cabinet, which comprises a cabinet body, a cabinet door, a first cabinet body image acquisition device, a second cabinet body image acquisition device, a cabinet door image acquisition device, and a processing device.

[0052] The cabinet door is rotationally connected with the cabinet body, and is used for opening or closing an article exit and entrance of the cabinet body.

[0053] The cabinet body comprises a cabinet body inner cavity, the cabinet body inner cavity is communicated with the outside of the cabinet body through the article exit and entrance, and the cabinet body inner cavity is used for storing articles.

[0054] The first cabinet body image acquisition device and the second cabinet body image acquisition device are both connected with a top surface of the cabinet body inner cavity, and are used for respectively acquiring images of the article exit and entrance from different directions.

[0055] The cabinet door image acquisition device is connected with a side of the cabinet door close to the cabinet body, and a position of the cabinet door image acquisition device matches a position of a door handle of the cabinet door, and the door handle is connected with a side of the cabinet door away from the cabinet body.

[0056] The processing device is in communication connection with the first cabinet body image acquisition device, the second cabinet body image acquisition device, and the cabinet door image acquisition device, and is used for executing the method in the first aspect or any implementation manner of the first aspect.

[0057] The technical scheme provided by the embodiments of the present disclosure can have the following beneficial effects:

[0058] The technical solution disclosed in the above embodiment can be used to obtain at least two to-be-identified images of a display cabinet, obtain the product position information of at least one product corresponding to each to-be-identified image in the at least two to-be-identified images, and obtain the motion track information of the at least one product according to the product position information. The motion trend of the product (i.e., the product near the display cabinet) in the to-be-identified image can be determined according to the motion track information. Then, the product category information of the at least one product in the at least two to-be-identified images is obtained, and the category of the product (i.e., the product near the display cabinet) in the to-be-identified image can be determined according to the product category information. When the product category indicated by the product category information of the target product in the at least one product matches the alarm product category, and the target motion track indicated by the motion track information of the target product satisfies the alarm motion track condition, that is, the motion trend of the product near the display cabinet and not stored in the display cabinet by the display cabinet operator is likely to move into the display cabinet, the alarm information is generated to warn at least one of the display cabinet operator and the user, so as to avoid the user from moving the product not used for storage in the display cabinet into the display cabinet, thereby ensuring that the display cabinet will not be moved into by the user, and the user experience of the user when taking the product from the display cabinet will not be damaged.

[0059] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0060] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description when read in conjunction with the accompanying drawings. In the drawings:

[0061] Figure 1 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown.

[0062] Figure 2 A schematic structural diagram of a mainboard according to an embodiment of the present disclosure is shown.

[0063] Figure 3 A schematic structural diagram of a control board according to an embodiment of the present disclosure is shown.

[0064] Figure 4 A schematic structural diagram of a power management module according to an embodiment of the present disclosure is shown.

[0065] Figure 5 A flowchart of a display cabinet control method according to an embodiment of the present disclosure is shown.

[0066] Figure 6 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown.

[0067] Figure 7A schematic top view of a display case according to an embodiment of the disclosure is shown.

[0068] Figure 8 A graphical user interface (GUI) diagram of a display case according to an embodiment of the disclosure is shown.

[0069] Figure 9 A graphical user interface (GUI) diagram of a display case according to an embodiment of the disclosure is shown.

[0070] Figure 10 A graphical user interface (GUI) diagram of a display case according to an embodiment of the disclosure is shown.

[0071] Figure 11 A general flowchart of a display case control method according to an embodiment of the disclosure is shown.

[0072] Figure 12 A schematic structural block diagram of an electronic device according to an embodiment of the disclosure is shown.

[0073] Figure 13 A structural diagram of a computer system suitable for implementing a display case control method according to an embodiment of the disclosure is shown.

[0074] Figure 14 A schematic structural diagram of a display case according to an embodiment of the disclosure is shown.

[0075] Figure 15 A schematic top view of a display case according to an embodiment of the disclosure is shown. DETAILED DESCRIPTION

[0076] Hereinafter, exemplary embodiments of the disclosure will be described in detail with reference to the accompanying drawings so as to be easily implemented by those skilled in the art. Also, portions unrelated to the description of the exemplary embodiments are omitted in the drawings for the sake of clarity.

[0077] In the disclosure, it should be understood that terms such as "include" or "have" are intended to indicate that there are features, numbers, steps, actions, components, parts or combinations thereof disclosed in the specification, and do not exclude the possibility that one or more other features, numbers, steps, actions, components, parts or combinations thereof exist or are added.

[0078] It is additionally noted that the embodiments and features of the disclosure can be combined with each other, if not contrary, and the disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0079] With the development of technology and the improvement of people's living standards, when a merchant or an enterprise stores goods, the goods are no longer simply placed on a shelf, but can be placed in a display cabinet to facilitate users to understand the information of the goods, so as to realize the storage and display of the goods at the same time. When a user needs to remove the goods from the display cabinet, the user can open the display cabinet by himself and perform corresponding operations.

[0080] In recent years, the number of display cabinets put into operation has gradually increased. In the process of using the display cabinet, the merchant or the enterprise generally needs to identify the goods in the display cabinet to determine the goods removed from the display cabinet, and count according to the above information. Based on the statistical result, the number and type of the remaining goods in the display cabinet can be known, and further settlement can also be made based on the statistical result.

[0081] In one embodiment, at least one camera arranged on the display cabinet can capture at least one image after the display cabinet is unlocked, and the image recognition is performed on the image, and the goods removed from the display cabinet are determined according to the image recognition result.

[0082] However, the applicant found in the implementation process that there are occasionally goods in some display cabinets that are not stored by the display cabinet operator. Through manual identification of the images collected by the display cabinet, it can be determined that this part of goods is removed into the display cabinet by the user using the display cabinet. Considering that the goods removed into the display cabinet by the user using the display cabinet may not meet the requirements of other users (for example, the quality of the goods does not meet the requirements, the type of the goods is greatly different from the types of other goods in the display cabinet), or even this part of goods may cause certain harm to the user using the goods (for example, the shelf life of the goods may have expired, the goods may have been opened and there may be foreign objects put into it, etc.), therefore, the goods removed into the display cabinet by the user may damage the user experience of other users taking goods from the display cabinet afterwards.

[0083] Therefore, how to ensure that goods are not removed into the display cabinet by the user is an increasingly urgent problem to be solved.

[0084] In view of the above defects, in an embodiment of the present disclosure, a display cabinet control method is provided.

[0085] The display cabinet control method provided by the embodiments of the present application can be applied to a display cabinet. The display cabinet can have a temperature control function. The temperature control function can be a refrigeration function, for example, the display cabinet can be a refrigerated display cabinet, a freezer display cabinet, a refrigeration cabinet, a freezer cabinet, a refrigerator, a wine cabinet, a cosmetic preservation cabinet, etc. The temperature control function can also be a heating function, for example, the display cabinet can be a warming cabinet, a heating display cabinet, a hot drink cabinet, etc. The embodiments of the present application do not make any limitation on the specific type of the display cabinet.

[0086] Exemplary, Figure 1 A schematic structural block diagram of a display cabinet according to an embodiment of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the display cabinet 100 can include a compressor 11, a condenser 12, a throttling element 13, and an evaporator 14, wherein the compressor 11, the condenser 12, the throttling element 13, and the evaporator 14 are connected by pipes filled with refrigerant to form a closed circuit, constituting a refrigeration system or a heating system capable of circulating refrigerant. Figure 1

[0087] In an embodiment of the present disclosure, the display cabinet includes a cabinet body and a cabinet door, wherein the cabinet body can be provided with a control board and a power management module, and the cabinet door can be provided with a main board.

[0088] In an embodiment of the present disclosure, Figure 2 A schematic structural block diagram of a main board according to an embodiment of the present disclosure is shown in FIG. 2. As shown in FIG. 2, the main board 200 can include a processor 201, a random access memory 202, a flash memory 203, a wireless local area network Bluetooth module 204, a gyroscope 205, a pressure sensor 206, a microphone 207, a loudspeaker 208, a camera 209, and a cellular communication module 210. Figure 2

[0089] In an embodiment of the present disclosure, Figure 3 A schematic structural block diagram of a control board according to an embodiment of the present disclosure is shown in FIG. 3. As shown in FIG. 3, the control board 300 can include a power input interface 301, a power output interface 302, a metering chip 303, a micro control unit chip 304, a real-time clock chip, a light switch interface 305, a temperature control switch interface 306, an evaporative fan interface 307, a compressor interface 308, a condenser fan interface 309, a temperature sensor interface 310, a communication interface 311, and a power interface 312. Figure 3

[0090] In an embodiment of the present disclosure, Figure 4 A schematic structural block diagram of a power management module according to an embodiment of the present disclosure is shown in FIG. 4. As shown in FIG. 4, the power management module 400 can include an alternating current to direct current conversion module 401, a charging management module 402, and a battery 403. The power management module 400 is used to power the main board and the control board, and to manage the charging and discharging of the battery. The power management module 400 can also be used to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), etc. In other embodiments, the power management module 400 can also be provided in the processor. Figure 4

[0091] ​​​​In an embodiment of the present application, the display cabinet further comprises a display screen. The display cabinet realizes the display function through a graphic processor, the display screen, and an application processor, etc. The graphic processor is a microprocessor for image processing, and is connected to the display screen and the application processor. The graphic processor is used to perform mathematical and geometric calculations for graphic rendering. The processor can include one or more graphic processors that execute program instructions to generate or change display information.

[0092] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the display cabinet. In other embodiments of the present application, the display cabinet can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware. For example, by combining different components, the display cabinet in the embodiments of the present application can be any one of a retail cabinet, a heating cabinet, a refrigeration cabinet, a freezer, a combination cabinet, or a showcase.

[0093] Figure 5 A flowchart of a display cabinet control method according to an embodiment of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the display cabinet control method includes the following steps S101-S104: Figure 5

[0094] In step S101, at least two images to be identified of the display cabinet are obtained.

[0095] In an embodiment of the present disclosure, obtaining the images to be identified can be receiving the images to be identified sent by the image to be identified acquisition device, or reading the images to be identified previously stored in the display cabinet, or receiving the images to be identified sent by other devices or systems. The images to be identified can be static images, or dynamic images or videos. It should be noted that the at least two images to be identified can be any two images in the plurality of cabinet body images, or can be two images determined in the plurality of cabinet body images according to a preset sampling time interval threshold value, which satisfies that the sampling time interval is greater than or equal to the preset sampling time interval threshold value. The images to be identified can be understood as including images of all or part of the display cabinet, or as including images of all or part of the goods in the display cabinet.

[0096] In an embodiment of the present disclosure, the image to be identified acquisition device can be arranged on the cabinet body or the cabinet door of the display cabinet. For example, the image to be identified acquisition device can be arranged on the top or bottom of the cabinet door, or on the side wall near the cabinet body side of the cabinet door, or on the side wall away from the cabinet body side of the cabinet door, or on the door handle of the cabinet door. The present application does not limit the specific arrangement position of the image to be identified acquisition device. The display cabinet can include one or more image to be identified acquisition devices, and the present application does not limit the specific number of the image to be identified acquisition device.

[0097] ​Exemplary, Figure 6 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown, Figure 7 A schematic top view of a display cabinet according to an embodiment of the present disclosure is shown, as Figure 6 And Figure 7 As shown, the display cabinet includes a cabinet body 501, a cabinet door 502, and a to-be-identified image acquisition device 503, wherein the to-be-identified image acquisition device 503 is connected to the inner wall top surface of the cabinet body 501.

[0098] The cabinet body 501 includes a display area 511 and an article access opening 521, the display area 511 is in communication with the outside of the cabinet body 501 through the article access opening 521, the display area 511 is used to accommodate articles stored in the display cabinet, and the articles in the display area 511 can be removed from the display cabinet through the article access opening 521, or articles can be moved into the display area 511 through the article access opening 521.

[0099] The cabinet door of the display cabinet can be rotatably connected to the cabinet body, or can be slidably connected to the cabinet body, or the cabinet door can be connected to the cabinet body by folding. It should be noted that the present application does not limit the connection mode of the cabinet door and the cabinet body, and for the convenience of understanding, Figure 6 And Figure 7 Taking the rotatable connection of the cabinet door 502 and the cabinet body 501 as an example for description. The cabinet door 502 is used to open or close the article access opening 521.

[0100] The to-be-identified image acquisition device 503 can be used to acquire at least one of the images of the entire display area 511, the images of part of the display area 511, and the images of the article access opening 521 to obtain the to-be-identified image. It should be noted that the display cabinet can include only one to-be-identified image acquisition device 503, or can include multiple to-be-identified image acquisition devices 503 for acquiring to-be-identified images from different directions respectively. The present disclosure does not limit the number of to-be-identified image acquisition devices 503.

[0101] In step S102, the article position information of at least one article corresponding to each of the at least two to-be-identified images is obtained, and the motion trajectory information of the at least one article is obtained according to the article position.

[0102] In an embodiment of the present disclosure, obtaining the article position information of at least one article corresponding to each of the at least two to-be-identified images can be understood as performing article position recognition on each of the at least two to-be-identified images according to a pre-obtained algorithm to obtain the article position information, or sending the at least two to-be-identified images and receiving the article position information obtained by other devices or systems such as a cloud server by performing article position recognition on each of the at least two to-be-identified images.

[0103] In an embodiment of the present disclosure, obtaining the motion trajectory information of the at least one item according to the item position can be understood as comparing the item position information to determine the items with different positions in the at least two to-be-identified images, and then obtaining the motion trajectory information of the items with different positions in the at least two to-be-identified images according to the positions of the items with different positions in the at least two to-be-identified images.

[0104] In an embodiment of the present disclosure, the motion trajectory information can be understood as information indicating the positions of the corresponding item at at least two time points, and the positions at the at least two time points can be understood as the starting position and the ending position of the movement of the item. The position of the item can be understood as the relative position of the item to the display cabinet, for example, the distance between the item and the display cabinet, or can be understood as the coordinates of the item in a coordinate system, for example, a three-dimensional coordinate system, a polar coordinate system, etc., and the position of the display cabinet can be understood as the pole point in the three-dimensional coordinate system.

[0105] In step S103, item category information of at least one item in the at least two to-be-identified images is obtained.

[0106] In an embodiment of the present disclosure, obtaining the item category information of the at least one item in the at least two to-be-identified images can be understood as performing item category identification on the taking area image according to a pre-obtained algorithm, or can be understood as sending the at least two to-be-identified images and receiving the item category information obtained by performing item category identification on the at least two to-be-identified images by other devices or systems, for example, a cloud server.

[0107] In step S104, in response to the item category indicated by the item category information of the target item in the at least one item matching the alarm item category and the target motion trajectory indicated by the motion trajectory information of the target item satisfying the alarm motion trajectory condition, alarm information is generated.

[0108] In an embodiment of the present disclosure, the item category indicated by the item category information matching the alarm item category can be understood as the alarm item category including the item category indicated by the item category information, wherein the alarm item category can be understood as the item category of the item stored in the display cabinet by the display cabinet operator; the item category indicated by the item category information matching the alarm item category can also be understood as the item category indicated by the item category information being included in the item category other than the alarm item category (i.e., the item category of the item stored in the display cabinet by the display cabinet operator). For example, when the item category indicated by the item category information is chocolate, and the non-alarm item category is tea beverage, soda, carbonated soda, and fruit juice, since the item category other than tea beverage, soda, carbonated soda, and fruit juice includes chocolate, it can be determined that the item category indicated by the item category information matches the alarm item category.

[0109] In an embodiment of the present disclosure, the motion trajectory indicated by the motion trajectory information satisfies the alarm motion trajectory condition, which can be understood as the similarity between the motion trajectory information and the alarm motion trajectory information being greater than or equal to a similarity threshold; or can be understood as the value of at least one of the speed, acceleration, motion direction, and distance from the display cabinet of at least one trajectory point in the motion trajectory obtained according to the motion trajectory information belonging to the value alarm range of the corresponding parameter.

[0110] In an embodiment of the present disclosure, the alarm information can be understood as information for prompting that the items in at least two images to be recognized include items being moved into the display cabinet, and the items do not belong to the items stored in the display cabinet by the display cabinet operator. The alarm information can be used to be displayed through a human-computer interaction device on the display cabinet, such as a display screen, a loudspeaker, etc., to prompt the user using the display cabinet to stop moving the items not belonging to the items stored in the display cabinet by the display cabinet operator into the display cabinet; the alarm information can also be sent to other devices or systems, such as a cloud authorization server or a mobile terminal corresponding to the display cabinet, to prompt the display cabinet operator of the display cabinet that a user is moving the items not belonging to the items stored in the display cabinet by the display cabinet operator into the display cabinet, so that the display cabinet operator of the display cabinet can check the items existing in the display cabinet in a timely manner in response to the alarm information.

[0111] Exemplary, take the display cabinet used to provide self-service as an example. When the customer needs to buy the goods in the display cabinet, the customer can use the mobile communication terminal to scan the two-dimensional code on the surface of the display cabinet to access the cloud authorization server corresponding to the display cabinet and send a statistical authorization request to the cloud authorization server, that is, to request to buy the goods in the display cabinet. After the cloud authorization server authorizes the statistical authorization request, the cloud authorization server sends statistical authorization information to the display cabinet. The display cabinet receives the statistical authorization information sent by the cloud server and, in response to the statistical authorization information, unlocks the door of the display cabinet. After the door is unlocked, the customer can open the door and take the goods in the display cabinet through the goods access port. From the time the door is opened to the time the door is closed, the image acquisition device at the cabinet body of the display cabinet can continuously acquire images of the goods access port and the display area of the display cabinet to obtain at least two to-be-identified images. Then the display cabinet obtains the goods position information of at least one goods corresponding to each to-be-identified image in the at least two to-be-identified images, and obtains the motion trajectory information of the at least one goods according to the goods position. The display cabinet obtains the goods category information of the at least one goods in the at least two to-be-identified images, generates an alarm information in response to the fact that the goods category indicated by the goods category information of the target goods in the at least one goods matches the alarm goods category and the target motion trajectory indicated by the motion trajectory information of the target goods satisfies the alarm motion trajectory condition, and displays the alarm information through the man-machine interaction device on the display cabinet, such as a loudspeaker, a display screen, etc., to prompt the user to stop moving the goods that are not stored in the display cabinet by the display cabinet operator into the display cabinet, so as to ensure that the display cabinet will not be moved into the goods by the user, so that other users will not take the goods that are not stored in the display cabinet by the display cabinet operator from the display cabinet, and so that the user experience of other users will not be damaged.

[0112] For example, the same is illustrated by taking the example of a display cabinet used to provide self-service. When the maintenance personnel of the display cabinet, such as a convenience store clerk, needs to replenish the display cabinet with goods, the maintenance personnel can use the mobile communication terminal to scan the two-dimensional code on the surface of the display cabinet to access the cloud authorization server corresponding to the display cabinet and send a goods replenishment request to the cloud authorization server, i.e., a request to replenish the display cabinet with goods. After the cloud authorization server processes the statistical authorization request, the cloud authorization server sends statistical authorization information to the display cabinet, i.e., permission for the maintenance personnel to replenish the display cabinet with goods. The display cabinet receives the statistical authorization information sent by the cloud server and, in response to the statistical authorization information, unlocks the door of the display cabinet. After the door is unlocked, the maintenance personnel can open the door and move the scanned goods into the display cabinet through the goods access port. After the maintenance personnel finishes replenishing the goods, the door can be closed. From the time the door is opened to the time the door is closed, the image acquisition device at the cabinet body of the display cabinet can continuously acquire images of the goods access port and the display area of the display cabinet to obtain at least two images to be identified. Then the display cabinet obtains the position information of at least one item corresponding to each of the at least two images to be identified, and obtains the motion trajectory information of the at least one item according to the position information of the at least one item. The display cabinet obtains the item category information of the at least one item in the at least two images to be identified, and in response to the item category information of the target item indicating that the item category matches the alarm item category and the motion trajectory information of the target item indicating that the target motion trajectory satisfies the alarm motion trajectory condition, generates an alarm information and displays the alarm information through the man-machine interaction device on the display cabinet, such as a loudspeaker, a display screen, etc., to prompt the maintenance personnel to stop moving items that are not stored in the display cabinet by the display cabinet operator into the display cabinet, thereby ensuring that the maintenance personnel can move the items stored in the display cabinet by the display cabinet operator into the display cabinet, and ensuring that after the maintenance personnel finishes maintaining the display cabinet, i.e., closes the door, the user will not take the items that are not stored in the display cabinet by the display cabinet operator from the display cabinet, so that the user experience will not be damaged.

[0113] Exemplary, with the display cabinet is used to provide goods access service to the target user as an example, wherein the target user can be understood as the user belonging to a certain designated unit or department. When the user needs to buy the goods in the display cabinet, the mobile communication terminal can be used to scan the two-dimensional code on the surface of the display cabinet to access the server corresponding to the display cabinet and send an identity authentication request to the server, i.e. request to authenticate the identity of the user; or the corresponding identity label (such as a badge, a certificate, etc.) can also be scanned through the scanning device on the display cabinet, and the identity authentication request is sent by the display cabinet to the corresponding server. When the display cabinet receives the goods access authorization information returned by the server, it can be understood that the server confirms that the user is the target user, and the display cabinet can unlock the cabinet door in response to the goods access authorization information. After the cabinet door is unlocked, the user can open the cabinet door and take the goods in the display cabinet through the goods access port, and the user can close the cabinet door after finishing taking the goods. From the cabinet door being opened to the cabinet door being closed, the image acquisition device at the cabinet body of the display cabinet can continuously acquire images of the goods access port and the display area of the display cabinet to obtain at least two images to be identified. Then the display cabinet obtains the goods position information of at least one goods corresponding to each of the at least two images to be identified, and acquires the motion trajectory information of the at least one goods according to the goods position; the goods category information of the at least one goods in the at least two images to be identified is obtained, and the alarm information is generated in response to the fact that the goods category indicated by the goods category information of the target goods in the at least one goods matches the alarm goods category and the target motion trajectory indicated by the motion trajectory information of the target goods meets the alarm motion trajectory condition, and the alarm information is displayed through the man-machine interaction device on the display cabinet, such as a loudspeaker, a display screen, etc. to prompt the user to stop moving the goods which are not stored in the display cabinet by the display cabinet operator into the display cabinet, so as to ensure that the goods are not moved into the display cabinet by the user, so that other users will not take the goods which are not stored in the display cabinet by the display cabinet operator from the display cabinet, and the user experience of other users will not be damaged.

[0114] The technical scheme above, by acquiring at least two to-be-recognized images of the display cabinet, acquiring the article position information of at least one article corresponding to each to-be-recognized image in the at least two to-be-recognized images, and acquiring the motion track information of the at least one article according to the article position, the moving trend of the article (i.e. the article near the display cabinet) in the to-be-recognized image can be determined according to the motion track information, and then the article category information of the at least one article in the at least two to-be-recognized images is acquired, and the category of the article (i.e. the article near the display cabinet) in the to-be-recognized image can be determined according to the article category information; in response to the article category indicated by the article category information of the target article in the at least one article matching the alarm article category and the target motion track indicated by the motion track information of the target article satisfying the alarm motion track condition, i.e. the moving trend of the article near the display cabinet and not stored in the display cabinet by the display cabinet operator is likely to move into the display cabinet, the alarm information is generated to warn at least one of the display cabinet operator and the user, so as to avoid the user moving the article not for storage in the display cabinet into the display cabinet, thereby ensuring that the display cabinet will not be moved into by the user, and the user experience of the user taking the article from the display cabinet will not be damaged.

[0115] In an implementation manner of the present disclosure, the article position information of at least one article corresponding to each to-be-recognized image in the at least two to-be-recognized images can be acquired through the following steps:

[0116] The pre-trained article position recognition model is acquired, and each to-be-recognized image is input into the article position recognition model to acquire the article position information output by the article position recognition model;

[0117] The article category information of at least one article in the at least two to-be-recognized images can be acquired through the following steps:

[0118] The to-be-recognized region corresponding to each article in the at least one article in each to-be-recognized image is determined according to the article position information of the at least one article corresponding to each to-be-recognized image;

[0119] The pre-trained article category recognition model is acquired, and the to-be-recognized region is input into the article category recognition model to acquire the article category information output by the article category recognition model.

[0120] In an embodiment of the present disclosure, the article position recognition model and the article category recognition model can be pre-stored in the display cabinet or acquired from other devices or systems. The article position recognition model and the article category recognition model can be a neural network (NN) model, a convolutional neural network (CNN) model, a long short-term memory (LSTM) model, or the like.

[0121] In this embodiment, by acquiring the pre-trained article position recognition model and inputting each to-be-recognized image into the article position recognition model to obtain the article position information output by the article position recognition model, it can be ensured that the obtained article position information can accurately indicate the position of the corresponding article. By determining the to-be-recognized region corresponding to each article of the at least one article in each to-be-recognized image according to the article position information of the at least one article corresponding to each to-be-recognized image, the area of the region in the to-be-recognized image that needs to be recognized can be reduced, the amount of data to be processed can be reduced, and then the pre-trained article category recognition model is acquired and the to-be-recognized region is input into the article category recognition model to obtain the article category information output by the article category recognition model, which can ensure that the accuracy of the article category indicated by the article category information is high.

[0122] In an implementation of the present disclosure, the method further comprises the following steps:

[0123] In response to the alarm information, the scanning code information collected by the scanning code device of the display cabinet is acquired, and the article category correction information is acquired according to the scanning code information;

[0124] In response to the fact that the article category indicated by the article category correction information does not match the target article category, the to-be-recognized region corresponding to the target article is taken as input, the article category correction information is taken as output, and the article category recognition model is trained.

[0125] In an embodiment of the present disclosure, the scanning code information collected by the scanning code device can be a to-be-recognized image received by the scanning code device, or pre-stored scanning code information in the display cabinet, or scanning code information received from other devices or systems. The scanning code information can be used to indicate the article information of the scanned article, which can include at least one of the quantity, category, and price of the article.

[0126] In an embodiment of the present disclosure, the item category correction information is obtained according to the code scanning information. The item category correction information can be obtained by querying the item category database according to the code scanning information to obtain the item category of the code scanning item, and generating the item category correction information including the item category of the code scanning item, wherein the item category database is used to indicate the correspondence between the item category and the code scanning information. The item category correction information can also be obtained by sending the code scanning information to other devices or systems, such as a cloud server, and receiving the item category correction information.

[0127] In this embodiment, by responding to the alarm information, the code scanning information collected by the code scanning device of the display cabinet is obtained, and the item category correction information is obtained according to the code scanning information. In response to the fact that the item category indicated by the item category correction information does not match the target item category, the to-be-recognized region corresponding to the target item is taken as the input, the item category correction information is taken as the output, and the item category recognition model is trained. This can ensure that when the item category recognition model fails to accurately recognize the category of the item in the to-be-recognized image, the trained item category recognition model can learn the rules between the to-be-recognized image that has failed to be successfully recognized and the category of the item in the to-be-recognized image, and ensure that the success rate of recognizing the category of the item in the to-be-recognized image based on the trained item category recognition model is high.

[0128] In an implementation of the present disclosure, before the item category recognition model is trained in response to the fact that the item category indicated by the item category correction information does not match the target item category, the to-be-recognized region corresponding to the target item is taken as the input, and the item category correction information is taken as the output, the method further includes the following steps:

[0129] receiving the first updated weight parameter sent by the first edge server, and updating the item category recognition model according to the first updated weight parameter;

[0130] In response to the fact that the item category indicated by the item category correction information does not match the target item category, the to-be-recognized region corresponding to the target item is taken as the input, and the item category correction information is taken as the output, the item category recognition model is trained, which can be achieved by the following steps:

[0131] In response to the fact that the item category indicated by the item category correction information does not match the target item category, the to-be-recognized region corresponding to the target item is taken as the input, and the item category correction information is taken as the output, the updated item category recognition model is trained;

[0132] The method further includes the following steps:

[0133] In response to the fact that the trained item category recognition model does not converge, a first gradient update vector is obtained according to the trained item category recognition model, and the first gradient update vector is sent to the first edge server.

[0134] Or, in response to the trained item category recognition model converging, the trained item category recognition model is stored as a target item category recognition model.

[0135] In an implementation form of the present disclosure, the first edge server is configured to aggregate the gradient update vectors, and update the weight parameters of the item category recognition model on the first edge server according to the aggregated gradient update vectors to obtain updated weight parameters. The first edge server can be a cloud server or a server provided by the display cabinet operator. It should be noted that one first edge server can correspond to one or more display cabinets. For example, the display cabinet operator can divide the area under its jurisdiction into multiple blocks, and the display cabinets in each block can correspond to one first edge server.

[0136] The item category recognition model on the first edge server can be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.

[0137] In an implementation form of the present disclosure, the updated weight parameter sent by the first edge server and received by the display cabinet is obtained by the first edge server aggregating the gradient update vectors sent by the plurality of display cabinets, and updating the weight parameter of the item category recognition model on the first edge server according to the aggregated gradient update vectors, so that the updated item category recognition model on the display cabinet can reflect the common rule between the to-be-recognized image and the category of the item in the to-be-recognized image learned by the item category recognition model on the first edge server in the last round of training. Then, in response to the fact that the item category indicated by the item category correction information does not match the target item category, the to-be-recognized region corresponding to the target item is taken as input, the item category correction information is taken as output, and the updated item category recognition model is trained, so that the updated item category recognition model on the display cabinet can learn the rule between the to-be-recognized image and the category of the item in the to-be-recognized image acquired by the display cabinet itself on the basis of learning the common rule, so that the trained item category recognition model on the display cabinet can learn the private rule between the to-be-recognized image and the category of the item in the to-be-recognized image acquired by the display cabinet itself. When the trained item category recognition model on the display cabinet does not converge, it indicates that the trained item category recognition model on the display cabinet still needs to be trained. By obtaining the gradient update vector according to the trained item category recognition model on the display cabinet and sending the gradient update vector, the first edge server can continue to obtain the corresponding updated weight parameter based on the gradient update vectors uploaded by the plurality of display cabinets, so as to continue to train the item category recognition model on each display cabinet. When the trained item category recognition model on the display cabinet converges, it can be considered that the converged item category recognition model on the display cabinet can accurately recognize the hand in the to-be-recognized image acquired by the display cabinet itself. The converged item category recognition model on the display cabinet can be stored as a target item category recognition model, that is, a model with high accuracy in recognizing the hand in the to-be-recognized image.

[0138] In the above technical solution, on the one hand, the finally obtained target item category recognition model can be a model that learns both common rules and private rules, and has high accuracy in recognizing the category of the item in the to-be-recognized image. On the other hand, since the process of continuously training the item category recognition model on each display cabinet is performed by the display cabinet and the first edge server together, compared with further training the item category recognition model by the display cabinet or the server alone, the required processing resources are less and the training speed is faster.

[0139] In an implementation form of the present disclosure, the method further comprises the following steps:

[0140] In response to the alarm information, obtaining item trajectory correction information;

[0141] In response to the motion trajectory indicated by the item trajectory correction information not matching the target motion trajectory, correction item position information of the target item corresponding to each of the to-be-identified images is obtained according to the item trajectory correction information;

[0142] Each of the to-be-identified images is taken as input, and the correction item position information of the target item corresponding to each of the to-be-identified images is taken as output, and the item position identification model is trained.

[0143] In an embodiment of the present disclosure, the item trajectory correction information can be sent by a server or input through a man-machine interaction device of the display cabinet.

[0144] In an implementation of the present disclosure, before the item trajectory correction information input through the man-machine interaction device of the display cabinet is obtained, at least two to-be-identified images can also be displayed. The at least two to-be-identified images can be displayed through the man-machine interaction device such as a display screen on the display cabinet, or can be displayed through other devices or systems.

[0145] In an embodiment of the present disclosure, the item trajectory correction information input through the man-machine interaction device of the display cabinet can be item trajectory correction information obtained by input of a user through at least one of a keyboard, a touch screen, and a touch pad on the display cabinet; can be voice information input by a user through a microphone on the display cabinet, and the item trajectory correction information is obtained by voice recognition on the voice information; or can be image information of an action of the user obtained through a camera on the display cabinet, and the item trajectory correction information is obtained by image recognition on the image information.

[0146] For example, the display case can present at least two images to be identified in a graphical user interface (GUI) displayed by the touch screen, and obtain an input action of the user inputting item trajectory correction information corresponding to the at least two images to be identified presented in the graphical user interface (GUI) through the touch screen, and obtain the item trajectory correction information input by the user according to the input action of the item trajectory correction information. The graphical user interface (GUI) is an interactive interface of an application (APP) running on the display case. Further, the graphical user interface (GUI) can also display one or more visual prompts on the touch screen for the user to perform the input action of the item trajectory correction information, and the visual prompts can provide the user with a hint or a reminder of the input action of the item trajectory correction information. The visual prompts can be text, graphics, or any combination thereof. The input action of the item trajectory correction information can include contact with the touch screen. In some embodiments, in addition to the visual prompts, the display case can also provide non-visual prompts to indicate the progress of the input action of the item trajectory correction information. The non-visual prompts can include audio prompts (e.g., sound) or physical prompts (e.g., vibration). In some embodiments, the input action of the item trajectory correction information is a predetermined gesture performed on the touch screen. The gesture used herein is the movement of an object / accessory in contact with the touch screen. For example, the predetermined gesture can include contacting the touch screen at a position where the item moved by the user is located in any of the at least two images to be identified presented on the touch screen (initiating the gesture), and maintaining contact with the touch screen at the position where the item moved by the user is located during the process of the touch screen sequentially displaying the at least two images to be identified, and maintaining contact with the touch screen for more than a predetermined contact time threshold after the predetermined contact time threshold (completing the gesture).

[0147] For ease of illustration, in the process of obtaining the item trajectory correction information input by the user and the first annotation information of the item scanned by the code, and other embodiments described below, the contact on the touch screen will be described as being performed by the user using at least one hand and using one or more fingers. However, it should be understood that the contact can also be performed using any suitable object or accessory, such as a stylus, a finger, and the like. The contact can include one or more taps on the touch screen, maintaining continuous contact with the touch screen, moving the contact point while maintaining continuous contact, interrupting the contact, or any combination thereof.

[0148] If the contact does not correspond to an attempt to perform the item trajectory correction information input action, or if the contact corresponds to an attempt by the user to fail or abandon the item trajectory correction information input action, the display case will not obtain the corresponding item trajectory correction information. For example, if the item trajectory correction information input action is to touch the touch screen at the location of the item moved by the user in the image displayed by the touch screen and to discontinue the contact after maintaining the contact with the touch screen for a time exceeding a preset contact time threshold, and the recognized contact is a series of random taps on the touch screen, the display case will not obtain the corresponding item trajectory correction information because the contact does not correspond to the item trajectory correction information input action.

[0149] If the contact corresponds to a successful performance of the item trajectory correction information input action, i.e., the user successfully performs the item trajectory correction information input action, the display case can obtain the corresponding item trajectory correction information based on the item trajectory correction information input action.

[0150] Figure 8 FIG. 1 shows a schematic diagram of a graphical user interface (GUI) of a display case according to an embodiment of the present disclosure, Figure 9 FIG. 1 shows a schematic diagram of a graphical user interface (GUI) of a display case according to an embodiment of the present disclosure, Figure 10 FIG. 1 shows a schematic diagram of a graphical user interface (GUI) of a display case according to an embodiment of the present disclosure, Figures 8-10 FIG. 1 shows a schematic diagram of a graphical user interface (GUI) of a display case according to an embodiment of the present disclosure,

[0151] In Figure 8 In the case shown in FIG. 1, the user uses his finger 61 to touch the touch screen 62 of the display case, and the display case will not obtain the corresponding item trajectory correction information because the contact does not correspond to an attempt to perform the item trajectory correction information input action.

[0152] In Figure 9 In the case shown in FIG. 1, the user uses his finger 61 to touch the touch screen 62 of the display case, and the display case will not obtain the corresponding item trajectory correction information because the contact does not correspond to an attempt to perform the item trajectory correction information input action.

[0153] In Figure 10In the example, the user ends the item trajectory correction information input action by releasing their finger 61 from the display case's touch screen 62. Specifically, in response to finger 61 being out of contact with touch screen 62 for a period exceeding a preset time difference, the display case determines the image position of the item 64 moved by the user in at least two images to be identified displayed on the graphical user interface GUI 63 based on the recorded finger movement trajectory 65. The display case then determines the object position of the item 64 moved by the user corresponding to each image to be identified based on the image position. Based on the obtained object position of the item 64 moved by the user, the display case obtains item trajectory correction information indicating the movement trajectory of the item 64 moved by the user.

[0154] In this embodiment, in response to an alarm message, object trajectory correction information is obtained; in response to a mismatch between the motion trajectory indicated by the object trajectory correction information and the target motion trajectory, corrected object position information of the target object corresponding to each image to be identified is obtained based on the object trajectory correction information; and an object position recognition model is trained using each image to be identified as input and the corrected object position information of the target object corresponding to each image to be identified as output. This ensures that if the object position recognition model fails to successfully identify the object position of an object in an image to be identified, the trained object position recognition model can learn the relationship between the previously unsuccessful image to be identified and the object position of the object in the image to be identified, thereby ensuring a high success rate for identifying the object position of the object in the image to be identified based on the trained object position recognition model.

[0155] In one implementation of the present disclosure, each image to be identified is used as input, and the corrected object position information of the target object corresponding to each image to be identified is used as output. Before training the object position recognition model, the method further includes the following steps:

[0156] receiving a second updated weight parameter sent by the second edge server, and updating the object location recognition model according to the second updated weight parameter;

[0157] Taking each image to be recognized as input and the corrected object position information of the target object corresponding to each image to be recognized as output, the object position recognition model is trained by the following steps:

[0158] Taking each image to be recognized as input and the corrected object position information of the target object corresponding to each image to be recognized as output, the updated object position recognition model is trained;

[0159] The method further comprises the steps of:

[0160] In response to the trained item position recognition model not converging, a second gradient update vector is obtained according to the trained item position recognition model, and the second gradient update vector is sent to the second edge server;

[0161] Alternatively, in response to the trained item position recognition model converging, the trained item position recognition model is stored as a target item position recognition model.

[0162] In an implementation manner of the present disclosure, the second edge server is configured to aggregate the gradient update vectors, and update the weight parameters of the item position recognition model on the second edge server according to the aggregated gradient update vectors to obtain updated weight parameters. The second edge server can be a cloud server or a server provided by the display cabinet operator. It should be noted that one second edge server can correspond to one or more display cabinets. For example, the display cabinet operator can divide the region under its jurisdiction into multiple blocks, and the display cabinets in each block can correspond to one second edge server.

[0163] The item position recognition model on the second edge server can be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.

[0164] In an implementation form of the present disclosure, the updated weight parameter sent by the second edge server and received by the display cabinet is obtained by the second edge server aggregating the gradient update vectors sent by the plurality of display cabinets, and updating the weight parameter of the item position recognition model on the second edge server according to the aggregated gradient update vectors. Therefore, the updated item position recognition model on the display cabinet can reflect the common rule between the to-be-recognized image and the position of the item in the to-be-recognized image learned by the item position recognition model on the second edge server in the last round of training. Then, each to-be-recognized image is taken as input, and the correction item position information of the target item corresponding to each to-be-recognized image is taken as output, and the updated item position recognition model is trained, so that the updated item position recognition model on the display cabinet can learn the common rule and the rule between the to-be-recognized image and the position of the item in the to-be-recognized image obtained by the display cabinet itself, and the trained item position recognition model on the display cabinet can learn the private rule between the to-be-recognized image and the position of the item in the to-be-recognized image obtained by the display cabinet itself. When the trained item position recognition model on the display cabinet does not converge, it indicates that the trained item position recognition model on the display cabinet still needs to be trained. By obtaining the gradient update vector according to the trained item position recognition model on the display cabinet and sending the gradient update vector, the second edge server can continue to obtain the corresponding updated weight parameter based on the gradient update vectors uploaded by the plurality of display cabinets, so as to continue to train the item position recognition model on each display cabinet. When the trained item position recognition model on the display cabinet converges, it can be considered that the converged item position recognition model on the display cabinet can accurately recognize the position of the item in the to-be-recognized image obtained by the display cabinet itself, and the converged item position recognition model on the display cabinet can be stored as the target item position recognition model, that is, the model with high accuracy in recognizing the position of the item in the to-be-recognized image.

[0165] In the above technical solution, on the one hand, the finally obtained target item position recognition model can be a model that learns both the common rule and the private rule, and has high accuracy in recognizing the position of the item in the to-be-recognized image. On the other hand, since the process of continuing to train the item position recognition model on each display cabinet is performed by the display cabinet and the second edge server together, compared with further training the item position recognition model by the display cabinet or the server alone, the required processing resources are less, and the training speed is faster.

[0166] In an implementation form of the present disclosure, the target motion trajectory includes a plurality of sub-motion trajectories corresponding to different time periods respectively;

[0167] The target motion trajectory indicated by the motion trajectory information of the target item satisfies the alarm motion trajectory condition, which includes:

[0168] The number of sub-motion trajectories in the target motion trajectory that satisfy the alarm motion trajectory condition is greater than or equal to a target number threshold.

[0169] In the above technical solution, considering that the motion trajectory of the article is often relatively complex, it is difficult to directly determine whether the target motion trajectory as a whole satisfies the alarm motion trajectory condition, which may require a large amount of data operation, thereby increasing the operation cost. However, it is relatively easy to determine whether each sub-motion trajectory included in the target motion trajectory satisfies the alarm motion trajectory condition, and the required amount of data operation is relatively small, thereby reducing the operation cost. By limiting the target motion trajectory indicated by the motion trajectory information of the target article to satisfy the alarm motion trajectory condition to the number of sub-motion trajectories in the target motion trajectory that satisfy the alarm motion trajectory condition being greater than or equal to a target number threshold, the purpose of determining whether the target motion trajectory as a whole satisfies the alarm motion trajectory condition can be achieved with a relatively small amount of data operation, thereby reducing the overall operation cost.

[0170] In an implementation manner of the present disclosure, the alarm motion trajectory condition comprises:

[0171] The motion direction of the motion trajectory at at least one time instant matches the alarm motion direction.

[0172] And / or, the first distance is greater than the second distance, the first distance being the distance between the trajectory point of the motion trajectory at the first time instant and the article display area of the display cabinet, and the second distance being the distance between the trajectory point of the motion trajectory at the second time instant and the article display area, the first time instant being earlier than the second time instant.

[0173] And / or, the position of at least one trajectory point in the motion trajectory belongs to the article display area.

[0174] In the above technical solution, the alarm motion direction can be understood as a direction pointing to the article display area, and if the article continues to move in the alarm motion direction, the article is likely to be moved into the article display area. By limiting the alarm motion trajectory condition to include the motion direction of the motion trajectory at at least one time instant matching the alarm motion direction, it can be ensured that the article of the article category matching the alarm article category generates an alarm information when it is likely to be moved into the article display area.

[0175] In the technical solution, the first distance is the distance between the track point of the motion track at the first time and the article display area of the display cabinet, the second distance is the distance between the track point of the motion track at the second time and the article display area, the first time is earlier than the second time, and therefore the first distance being greater than the second distance can be understood as that the article is getting closer and closer to the article display area, and if the article continues to move, the article can be moved into the article display area. By limiting the alarm motion track condition to include that the first distance is greater than the second distance, it can be ensured that the article whose article category matches the alarm article category generates the alarm information when the article is possibly moved into the article display area.

[0176] In the technical solution, when the position of the at least one track point in the motion track belongs to the article display area, it can be understood that the article has been moved into the article display area. By limiting the alarm motion track condition to include that the position of the at least one track point in the motion track belongs to the article display area, it can be ensured that the article whose article category matches the alarm article category generates the alarm information when the article has been moved into the article display area.

[0177] In an implementation manner of the present disclosure, the article category information is used to indicate at least two article categories and a category probability corresponding to each article category.

[0178] The article category indicated by the article category information of the target article in the at least one article matches the alarm article category, and the article category information of the target article includes:

[0179] The article category indicated by the article category information of the target article includes the alarm article category, and the category probability corresponding to the alarm article category is the highest in the category probability indicated by the article category information of the target article.

[0180] In an implementation manner of the present disclosure, the category probability corresponding to the article category can be understood as being used to indicate the possibility that the article category of the to-be-recognized image is determined to be the corresponding article category. The higher the category probability corresponding to the article category is, the higher the possibility that the to-be-recognized image includes the article of the article category is.

[0181] In the technical solution, the speed of the user moving the article can be relatively fast, so that the collected part of the region of the to-be-identified image can be blurred, so that the article category of the article in the to-be-identified image cannot be accurately determined. By limiting the article category information to indicate at least two article categories and the category probability corresponding to each article category, it can be ensured that the article category information cannot be obtained due to identification difficulties. By limiting the matching of the article category indicated by the article category information of the target article in the at least one article and the alarm article category to include the article category indicated by the article category information of the target article, the article category information of the target article includes the alarm article category, and the category probability corresponding to the alarm article category is the highest in the category probability indicated by the article category information of the target article, the possibility of the highest article category existing in the to-be-identified image can be determined based on the possibility of the highest article category existing in the to-be-identified image, whether the alarm information needs to be generated, and the stability of generating the alarm information is improved.

[0182] In an implementation manner of the present disclosure, the to-be-identified image is a plurality of video frames in a cabinet door video, the cabinet door video is collected by a to-be-identified image collection device in response to the cabinet door being unlocked and in response to the cabinet door being locked, and the to-be-identified image collection device is arranged at the cabinet door of the display cabinet.

[0183] Or, the to-be-identified image is a plurality of video frames in a cabinet body video, the cabinet body video is collected by a cabinet body image collection device in response to the cabinet door being unlocked and in response to the cabinet door being locked, and the cabinet body image collection device is arranged at the cabinet body of the display cabinet.

[0184] In an implementation manner of the present disclosure, the cabinet door is unlocked, which can be understood as that the cabinet door locking device on the display cabinet is unlocked in response to an unlocking instruction, so that the cabinet door can rotate or slide relative to the cabinet body of the display cabinet to open the article access port of the display cabinet. It can also be understood that the cabinet door locking device on the display cabinet is triggered by a corresponding unlocking operation, so that the cabinet door locking device itself is set to an unlocked state, so that the cabinet door can rotate or slide relative to the cabinet body of the display cabinet to open the article access port of the display cabinet. Through the article access port, the user can move the article from the article display area or move the article to the article display area.

[0185] In an embodiment of the present disclosure, whether the cabinet door is locked can be determined by receiving the cabinet door locking state information sent by the cabinet door locking device on the display cabinet, and determining whether the cabinet door is locked according to the cabinet door locking state information. The cabinet door locking state information can also be received from other devices or systems, and whether the cabinet door is locked can be determined according to the cabinet door locking state information. For example, whether the cabinet door is locked can be identified by an infrared identification device independent of the display cabinet, and the cabinet door locking state information can be sent to the display cabinet according to the identification result. A camera independent of the display cabinet, such as a security camera, can also be used to obtain an image including all or part of the display cabinet, perform image recognition according to the image, and send the cabinet door locking state information to the display cabinet according to the image recognition.

[0186] In this embodiment, considering that the items in the display cabinet can only be removed or moved into the display cabinet when the cabinet door of the display cabinet is unlocked, by limiting the to-be-identified image to be a plurality of video frames in the cabinet door video or a plurality of video frames in the cabinet body video, the probability that the to-be-identified image includes an object that can be moved into the display cabinet can be improved, the number of images to be processed can be reduced, and data processing resources can be saved.

[0187] Figure 11 The overall flowchart of the display cabinet control method according to an embodiment of the present disclosure is shown in FIG. 1. Figure 11 As shown in FIG. 1, the display cabinet control method includes the following steps.

[0188] In step S201, at least two to-be-identified images of the display cabinet are obtained.

[0189] In step S202, a pre-trained item position recognition model is obtained, and each to-be-identified image is input into the item position recognition model to obtain item position information output by the item position recognition model.

[0190] In step S203, motion trajectory information of at least one item is obtained according to the item position.

[0191] In step S204, a to-be-identified region corresponding to each item in the at least one item is determined in each to-be-identified image according to the item position information of the at least one item corresponding to each to-be-identified image.

[0192] In step S205, a pre-trained item category recognition model is obtained, and the to-be-identified region is input into the item category recognition model to obtain item category information output by the item category recognition model.

[0193] In step S206, in response to the fact that the item category indicated by the item category information of the target item includes the alarm item category, the category probability corresponding to the alarm item category is the highest among the category probabilities indicated by the item category information of the target item, and the number of sub-motion trajectories satisfying the alarm motion trajectory condition in the target motion trajectory indicated by the motion trajectory information of the target item is greater than or equal to the target number threshold, alarm information is generated.

[0194] In step S207, in response to the alarm information, the code scanning information collected by the code scanning device of the display cabinet is acquired, and item category correction information is acquired according to the code scanning information.

[0195] In step S208, the first updated weight parameter sent by the first edge server is received, and the item category recognition model is updated according to the first updated weight parameter.

[0196] In step S209, in response to the fact that the item category indicated by the item category correction information does not match the target item category, the target item corresponding to the to-be-recognized region is taken as input, the item category correction information is taken as output, and the updated item category recognition model is trained.

[0197] In step S210, in response to the fact that the trained item category recognition model does not converge, a first gradient update vector is acquired according to the trained item category recognition model, and the first gradient update vector is sent to the first edge server, or, in response to the fact that the trained item category recognition model converges, the trained item category recognition model is stored as a target item category recognition model.

[0198] In step S211, in response to the alarm information, item trajectory correction information is acquired.

[0199] In step S212, in response to the fact that the motion trajectory indicated by the item trajectory correction information does not match the target motion trajectory, the correction item position information of the target item corresponding to each to-be-recognized image is acquired according to the item trajectory correction information.

[0200] In step S213, the second updated weight parameter sent by the second edge server is received, and the item position recognition model is updated according to the second updated weight parameter.

[0201] In step S214, each to-be-recognized image is taken as input, and the correction item position information of the target item corresponding to each to-be-recognized image is taken as output, and the updated item position recognition model is trained.

[0202] In step S215, in response to the trained item location recognition model not converging, a second gradient update vector is obtained based on the trained item location recognition model, and the second gradient update vector is sent to the second edge server, or, in response to the trained item location recognition model converging, the trained item location recognition model is stored as the target item location recognition model.

[0203] The present disclosure also discloses an electronic device, Figure 12 A schematic structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 12 As shown, the electronic device 600 includes a memory 601 and a processor 602; wherein the memory 601 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 602 to implement the above method steps.

[0204] Figure 13 FIG. 1 is a schematic diagram of a computer system suitable for implementing a display cabinet control method according to an embodiment of the present disclosure. Figure 13 As shown, the computer system 700 includes a processing unit 701, which can execute various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the system 700 are also stored in the RAM 703. The processing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0205] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom can be installed into the storage section 708 as needed. Among them, the processing unit 701 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0206] In particular, the method described above with reference to the accompanying drawings can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a non-transitory computer readable medium, the computer program containing program code for executing the methods of the accompanying drawings. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. For example, embodiments of the present disclosure include a readable storage medium having stored thereon a computer instruction which, when executed by a processor, implements program code for executing the methods of the accompanying drawings.

[0207] The flow diagrams and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0208] The units or modules described in the embodiments of the present disclosure can be implemented by software, or by hardware. The described units or modules can also be provided in a processor, and the names of the units or modules do not constitute a limitation on the units or modules themselves in some cases.

[0209] As another aspect, the present disclosure also provides a computer readable storage medium, which can be the computer readable storage medium included in the apparatus described in the above embodiments, or can exist separately from the apparatus and not be assembled into the apparatus. The computer readable storage medium stores one or more programs for execution by one or more processors to perform the methods described in the present disclosure.

[0210] In addition, the present disclosure also provides a computer program product, which stores a computer program that, when executed by a processor, causes the processor to implement at least the methods provided in the above embodiments.

[0211] Figure 14 shows a schematic structural diagram of a display cabinet according to an embodiment of the present disclosure, Figure 15 FIG. 1 shows a schematic top view of a display cabinet according to an embodiment of the present disclosure. Figure 14 as well as Figure 15 As shown, the display cabinet 800 includes a cabinet body 801, a cabinet door 802, a first cabinet body image acquisition device 803, a second cabinet body image acquisition device 804, a cabinet door image acquisition device 805 and a processing device;

[0212] The cabinet door 802 is rotatably connected to the cabinet body 801 and is used to open or close the article access entrance 811 of the cabinet body 801;

[0213] The cabinet 801 includes an inner cavity 821 , which is connected to the outside of the cabinet 801 through an article entrance and exit 811 . The inner cavity 821 is used to store articles.

[0214] The first cabinet image acquisition device 803 and the second cabinet image acquisition device 804 are both connected to the top surface of the cabinet inner cavity 821. The first cabinet image acquisition device 803 and the second cabinet image acquisition device 804 are used to respectively acquire images of the item entrance and exit 811 from different directions;

[0215] The cabinet door image acquisition device 805 is connected to the side of the cabinet door 802 close to the cabinet body 801, and the position of the cabinet door image acquisition device 805 matches the position of the door handle 812 of the cabinet door 802, and the door handle 812 is connected to the side of the cabinet door 802 away from the cabinet body 801;

[0216] The processing device is communicatively connected to the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device, and is used to execute the method provided in the above embodiment.

[0217] In one embodiment of the present disclosure, collecting images of the item entrance and exit can be understood as collecting images of all or part of the item entrance and exit. Through the collected images, it is possible to determine the items that the user moves out of the cabinet cavity from the item entrance and exit, or to determine the items that the user moves into the cabinet cavity from the item entrance and exit.

[0218] In one embodiment of the present disclosure, the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device may be cameras, or other devices with image acquisition functions.

[0219] In an embodiment of the present disclosure, the processing device can include one or more processing units, for example: the processing device can include one or more of an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a memory, a video codec, a digital signal processor, a baseband processor, and / or a neural network processor. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0220] The above description is merely preferred embodiments of the present disclosure and a description of the principles of the technology used. Those skilled in the art should understand that the scope of the application involved in the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with each other to form a technical solution with similar functions disclosed in the present disclosure (but not limited to).

Claims

1. A display cabinet control method, characterized in that: The method comprises: Obtain at least two images of the display case to be identified; Obtaining object position information of at least one object corresponding to each of the at least two images to be identified, and obtaining motion trajectory information of the at least one object based on the object position; Obtaining item category information of at least one item in the at least two images to be identified; In response to the item category indicated by the item category information of the target item among the at least one item matching the alarm item category and the target motion trajectory indicated by the motion trajectory information of the target item satisfying the alarm motion trajectory condition, an alarm message is generated; wherein the alarm item category refers to the item category of items not stored in the display case by the display case operator; the alarm message is used to be displayed through a human-computer interaction device on the display case to prompt a user of the display case to stop moving items that are not stored in the display case by the display case operator into the display case; The alarm motion trajectory conditions include: The direction of motion of the motion trajectory at at least one moment matches the alarm motion direction; and / or, the first distance is greater than the second distance, the first distance being the distance between a track point of the motion trajectory at a first moment and the item display area of ​​the display case, the second distance being the distance between a track point of the motion trajectory at a second moment and the item display area, and the first moment being earlier than the second moment; And / or, the position of at least one track point in the motion track belongs to the item display area.

2. The display cabinet control method according to claim 1, characterized in that: The acquiring of the object location information of at least one object corresponding to each of the at least two images to be identified includes: Obtaining a pre-trained object location recognition model, and inputting each image to be recognized into the object location recognition model to obtain object location information output by the object location recognition model; The obtaining of item category information of at least one item in the at least two images to be identified comprises: determining, in each image to be identified, a region to be identified corresponding to each of the at least one object according to the object position information of the at least one object corresponding to each image to be identified; A pre-trained item category recognition model is obtained, and the area to be recognized is input into the item category recognition model to obtain item category information output by the item category recognition model.

3. The display cabinet control method according to claim 2, characterized in that: The method further comprises: In response to the warning information, obtaining scanning information collected by a scanning device of the display case, and obtaining item category correction information based on the scanning information; In response to the item category indicated by the item category correction information not matching the target item category, the item category recognition model is trained using the to-be-recognized area corresponding to the target item as input and the item category correction information as output.

4. The display cabinet control method according to claim 3, characterized in that: In response to the mismatch between the item category indicated by the item category correction information and the target item category, the method further comprises: receiving a first updated weight parameter sent by a first edge server, and updating the item category recognition model according to the first updated weight parameter; In response to the mismatch between the item category indicated by the item category correction information and the target item category, training the item category recognition model using the to-be-recognized area corresponding to the target item as input and the item category correction information as output includes: In response to the item category indicated by the item category correction information not matching the target item category, training an updated item category recognition model using the to-be-recognized area corresponding to the target item as input and the item category correction information as output; The method further comprises: In response to the trained item category recognition model not converging, obtaining a first gradient update vector according to the trained item category recognition model, and sending the first gradient update vector to the first edge server; Alternatively, in response to the trained item category recognition model converging, the trained item category recognition model is stored as the target item category recognition model.

5. The display cabinet control method according to claim 2, characterized in that: The method further comprises: In response to the warning information, obtaining object trajectory correction information; In response to a mismatch between the motion trajectory indicated by the object trajectory correction information and the target motion trajectory, obtaining corrected object position information of the target object corresponding to each image to be identified based on the object trajectory correction information; Each image to be identified is used as input, and the corrected object position information of the target object corresponding to each image to be identified is used as output to train the object position recognition model.

6. The display cabinet control method according to claim 5, characterized in that: Before training the object position recognition model, the method further comprises: taking each image to be recognized as input and outputting the corrected object position information of the target object corresponding to each image to be recognized as output; receiving a second updated weight parameter sent by the second edge server, and updating the object location recognition model according to the second updated weight parameter; The method of taking each image to be identified as input and taking the corrected object position information of the target object corresponding to each image to be identified as output, and training the object position recognition model includes: Taking each image to be identified as input and the corrected object position information of the target object corresponding to each image to be identified as output, training an updated object position recognition model; The method further comprises: In response to the trained object location recognition model not converging, obtaining a second gradient update vector according to the trained object location recognition model, and sending the second gradient update vector to the second edge server; Alternatively, in response to the trained object location recognition model converging, the trained object location recognition model is stored as the target object location recognition model.

7. The display cabinet control method according to any one of claims 1 to 6, characterized in that: The target motion trajectory includes a plurality of sub-motion trajectories corresponding to different time periods; The target motion trajectory indicated by the motion trajectory information of the target object meets the alarm motion trajectory condition, including: The number of sub-motion trajectories in the target motion trajectory that meet the warning motion trajectory condition is greater than or equal to a target number threshold.

8. The display cabinet control method according to any one of claims 1 to 6, characterized in that: The item category information is used to indicate at least two item categories and a category probability corresponding to each item category; The item category indicated by the item category information of the target item in the at least one item matches the alarm item category, comprising: The item category indicated by the item category information of the target item includes the alarm item category, and the category probability corresponding to the alarm item category is the highest among the category probabilities indicated by the item category information of the target item.

9. The display cabinet control method according to any one of claims 1 to 6, characterized in that: The images to be identified are multiple video frames in a cabinet door video, the cabinet door video is captured by the image acquisition device to be identified starting recording in response to the cabinet door being unlocked and ending recording in response to the cabinet door being locked, and the image acquisition device to be identified is disposed at the cabinet door of the display cabinet; Alternatively, the image to be identified is a plurality of video frames in a cabinet video, and the cabinet video is captured by the cabinet image acquisition device starting recording in response to unlocking the cabinet door and ending recording in response to locking the cabinet door, and the cabinet image acquisition device is arranged on the cabinet of the display cabinet.

10. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 9.

11. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. A display cabinet, characterized in that: The display cabinet includes a cabinet body, a cabinet door, a first cabinet body image acquisition device, a second cabinet body image acquisition device, a cabinet door image acquisition device and a processing device; The cabinet door is rotatably connected to the cabinet body and is used to open or close the article access entrance of the cabinet body; The cabinet body includes an inner cavity of the cabinet body, the inner cavity of the cabinet body is connected to the outer cavity of the cabinet body through the article access port, and the inner cavity of the cabinet body is used to store articles; The first cabinet image acquisition device and the second cabinet image acquisition device are both connected to the top surface of the cabinet inner cavity, and the first cabinet image acquisition device and the second cabinet image acquisition device are used to respectively acquire images of the article entrance and exit from different directions; The cabinet door image acquisition device is connected to a side of the cabinet door close to the cabinet body, and the position of the cabinet door image acquisition device matches the position of the cabinet door handle, and the door handle is connected to a side of the cabinet door away from the cabinet body; The processing device is communicatively connected to the first cabinet image acquisition device, the second cabinet image acquisition device, and the cabinet door image acquisition device, and the processing device is used to execute the method according to any one of claims 1 to 9.

Images