Display cabinet control method, device and medium
By identifying and processing personal information in the display cabinet image, ensuring that the image does not contain user personal information, the problem of privacy leakage during display cabinet image transmission is solved and safe image transmission is achieved.
Patent Information
- Application Number
- CN202111629238.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-28
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-12-28
AI Technical Summary
In the prior art, images sent by display cabinets may leak user's personal information, resulting in privacy and security violations.
By obtaining the set of images to be identified, it is determined whether it contains personal information images. If not, the image collection will be sent. The pre-trained personal information target recognition model is used to perform image recognition and label information processing, the model is updated to ensure the accuracy of recognition, and if necessary When removing or covering the personal information area, ensure that the image does not contain personal information.
It effectively prevents the leakage of user personal information, reduces privacy and security risks, and ensures the security of image transmission.
Smart Images

Figure CN114332930B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of control technology, and in particular to a display cabinet control method, device and medium. Background Art
[0002] In recent years, when merchants or enterprises store items, in order to facilitate users to understand the information of items, they can place items in display cabinets to store and display goods at the same time. When the user needs to move items out of the display cabinet or put items into the display cabinet, the user can open the display cabinet by himself and perform the corresponding operation. In this scenario, the display cabinet can capture images of the items in the display cabinet and upload the collected data so that other devices or systems such as servers and clouds can determine the users who move items out of or into the display cabinet, or the items moved out of or into the display cabinet, etc. based on the images uploaded by the display cabinet, so as to perform statistics. Summary of the invention
[0003] The embodiments of the present disclosure provide a display cabinet control method, device and medium, which are used to solve the problem in the related art that images sent by the display cabinet may infringe upon the personal information security of the user.
[0004] In a first aspect, a display cabinet control method is provided in an embodiment of the present disclosure.
[0005] Specifically, the display cabinet control method includes:
[0006] Acquire a set of images to be identified, including at least one image;
[0007] Determine whether the set of images to be identified includes a personal information image, where the personal information image is an image including at least one personal information target, and the personal information target is used to obtain the personal information of at least one person;
[0008] In response to the set of images to be identified not including the personal information image, the set of images to be identified is sent.
[0009] In one implementation of the present disclosure, determining whether a set of images to be identified includes personal information images includes:
[0010] Obtain a pre-trained personal information target recognition model, and input an image in the set of images to be recognized into the personal information target recognition model to obtain personal information target recognition information output by the personal information target recognition model;
[0011] Determining whether the set of images to be identified includes personal information images according to the personal information target identification information;
[0012] The method further comprises:
[0013] Obtaining personal information target annotation information, and determining an annotated image in the set of images to be identified based on the personal information target annotation information, wherein the personal information target annotation information is used to indicate the personal information target in the annotated image;
[0014] In response to the personal information image not including the annotated image, the annotated image is used as input and the personal information target annotated information is used as output to train the personal information target recognition model.
[0015] In one implementation of the present disclosure, in response to the personal information image not including the annotated image, taking the annotated image as input and the personal information target annotated information as output, before training the personal information target recognition model, the method further includes:
[0016] Receive the updated weight parameters sent by the edge server, and update the personal information target recognition model according to the updated weight parameters;
[0017] In response to the personal information image not including the annotated image, taking the annotated image as input and the personal information target annotated information as output, training a personal information target recognition model, including:
[0018] In response to the personal information image not including the annotated image, taking the annotated image as input and the personal information target annotated information as output, training the updated personal information target recognition model;
[0019] The method further comprises:
[0020] In response to the trained personal information target recognition model not converging, obtaining a gradient update vector according to the trained personal information target recognition model, and sending the gradient update vector to the edge server;
[0021] Or, in response to the convergence of the trained personal information target recognition model, the trained personal information target recognition model is stored as the target personal information target recognition model.
[0022] In one implementation of the present disclosure, the personal information target labeling information is sent by a server, or the personal information target labeling information is input through a human-computer interaction device of the display cabinet.
[0023] In one implementation of the present disclosure, determining whether a set of images to be identified includes personal information images includes:
[0024] Perform image recognition on the images in the image set to be identified, and determine whether the image set to be identified includes personal information images based on the image recognition results.
[0025] In an implementation of the present disclosure, the images in the set of images to be recognized are captured in response to a door of a display cabinet being unlocked.
[0026] In one implementation of the present disclosure, the method further includes:
[0027] In response to the set of images to be identified including the personal information image, the personal information image is removed from the set of images to be identified to obtain a filtered set of images to be identified, and the filtered set of images to be identified is sent; and / or
[0028] In response to the set of images to be identified including personal information images, the area where the personal information target is located in the personal information image is covered according to a preset pattern, or the area where the personal information target is located is removed from the personal information image to obtain a modified set of images to be identified, and the modified set of images to be identified is sent.
[0029] In a second aspect, a display cabinet control device is provided in an embodiment of the present disclosure.
[0030] Specifically, the display cabinet control device includes:
[0031] An image acquisition module is configured to acquire a set of images to be identified including at least one image, wherein the images in the set of images to be identified are acquired in response to a door of the display cabinet being unlocked;
[0032] An image recognition module is configured to determine whether the set of images to be recognized includes a personal information image, where the personal information image is an image including at least one personal information target, and the personal information target is used to obtain personal information of at least one person;
[0033] The image sending module is configured to send the set of images to be identified in response to the set of images to be identified not including the personal information image.
[0034] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the method described in the first aspect or any one of the implementations of the first aspect.
[0035] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the method described in the first aspect or any one of the implementations of the first aspect is implemented.
[0036] In a fifth aspect, a computer program product is provided in an embodiment of the present disclosure, which includes computer instructions, and when the computer instructions are executed by a processor, the method described in the first aspect or any implementation manner of the first aspect is implemented.
[0037] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:
[0038] The above technical solution obtains a set of images to be identified including at least one image, and determines whether the set of images to be identified includes a personal information image. In response to the set of images to be identified not including the personal information image, the set of images to be identified is sent, wherein the personal information image is an image including at least one personal information target, and the personal information target is used to obtain the personal information of at least one person. In the above solution, since any image in the set of images to be identified sent by the display cabinet does not include the personal information target, other devices or systems receiving the set of images to be identified cannot obtain the personal information of any user based on the images in the set of images to be identified, thereby ensuring that the images sent by the display cabinet will not infringe on the personal information security of any user, and reducing the risk of user privacy being infringed.
[0039] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:
[0041] Figure 1 A schematic structural block diagram of a display cabinet according to an embodiment of the present disclosure is shown;
[0042] Figure 2 A schematic structural block diagram of a mainboard according to an embodiment of the present disclosure is shown;
[0043] Figure 3 A schematic structural block diagram of a control panel according to an embodiment of the present disclosure is shown;
[0044] Figure 4 A schematic structural block diagram of a power management module according to an embodiment of the present disclosure is shown;
[0045] Figure 5 A flow chart showing a display cabinet control method according to an embodiment of the present disclosure;
[0046] Figure 6 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown;
[0047] Figure 7 A schematic top view of a display cabinet according to an embodiment of the present disclosure is shown;
[0048] Figure 8 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown;
[0049] Fig. 9 A schematic diagram of a graphical user interface GUI of a display cabinet according to an embodiment of the present disclosure is shown;
[0050] Fig.10 A schematic diagram of a graphical user interface GUI of a display cabinet according to an embodiment of the present disclosure is shown;
[0051] Fig.11 A schematic diagram of a graphical user interface GUI of a display cabinet according to an embodiment of the present disclosure is shown;
[0052] Fig.12 An overall flow chart of a display cabinet control method according to an embodiment of the present disclosure is shown;
[0053] Fig.13 A schematic structural block diagram of a display cabinet control device according to an embodiment of the present disclosure is shown;
[0054] Fig.14 A schematic structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;
[0055] Fig.15 It is a structural schematic diagram of a computer system suitable for implementing a display cabinet control method according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0056] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.
[0057] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the specification, and do not exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.
[0058] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0059] As mentioned above, with the development of technology and the improvement of people's living standards, when merchants or enterprises store items, they no longer simply place them on shelves, but in order to facilitate users to understand the information of the items, they can place the items in display cabinets to store and display the goods at the same time. When the user needs to move an item out of the display cabinet or put an item into the display cabinet, the user can open the display cabinet by himself and perform the corresponding operation.
[0060] In recent years, the number of display cabinets put into operation has gradually increased. In the process of using the display cabinets, merchants or enterprises generally need to monitor the items in the display cabinets to determine the users who move items out of or into the display cabinets, as well as the items that are moved out of or into the display cabinets, and make statistics based on the above information. Based on the statistical results, the number and type of remaining items in the display cabinet can be known, and further settlement can also be made based on the statistical results.
[0061] In one embodiment, a radio frequency identification system (RFID) antenna can be installed at the item entrance and exit of the display cabinet, and RFID tags can be attached to the items. When the items are removed from the display cabinet through the item entrance or moved into the display cabinet through the item entrance, the RFID tags attached to the items can be detected by the RFID antenna, thereby determining the items removed from or moved into the display cabinet. However, in this solution, since each item needs to be attached with an RFID tag, more RFID tags are required, which increases the cost of statistics.
[0062] In another embodiment, after the display cabinet is unlocked, an image corresponding to the display cabinet may be collected, and the object moved out of or into the display cabinet may be determined based on the image. However, in this solution, the collected image usually needs to be sent by the display cabinet to the cloud for identification, so that the display cabinet can determine the object moved out of or into the display cabinet based on the result returned by the cloud. However, sometimes, the collected image may include the user's face or the user's work badge, etc., and the cloud may obtain the user's personal information based on these contents in the image, which may infringe the user's personal privacy.
[0063] Therefore, how to ensure that the images sent by the display cabinet will not infringe on the user's personal information security is an increasingly urgent problem that needs to be solved.
[0064] In view of the above-mentioned defects, in one embodiment of the present disclosure, a display cabinet control method is proposed, which obtains a set of images to be identified including at least one image, and determines whether the set of images to be identified includes a personal information image. In response to the set of images to be identified not including the personal information image, the set of images to be identified is sent, wherein the personal information image is an image including at least one personal information target, and the personal information target is used to obtain the personal information of at least one person. In the above scheme, since any image in the set of images to be identified sent by the display cabinet does not include the personal information target, other devices or systems that receive the set of images to be identified cannot obtain the personal information of any user based on the images in the set of images to be identified, thereby ensuring that the images sent by the display cabinet will not infringe on the personal information security of any user and reducing the risk of user privacy being violated.
[0065] The display cabinet control method provided in the embodiment of the present application can be applied to the display cabinet, and the display cabinet can have a temperature control function, and the temperature control function can be a cooling function, for example, the display cabinet can be a refrigerated display cabinet, a frozen display cabinet, a refrigerator, a freezer, a refrigerator, a wine cabinet, a cosmetics preservation cabinet, etc.; the temperature control function can also be a heating function, for example, the display cabinet can be a warming cabinet, a heated display cabinet, a hot drink cabinet, etc. The embodiment of the present application does not impose any restrictions on the specific type of the display cabinet.
[0066] For example, Figure 1 A schematic structural block diagram of a display cabinet according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the display cabinet 100 may include a compressor 11, a condenser 12, a throttling element 13 and an evaporator 14, wherein the compressor 11, the condenser 12, the throttling element 13 and the evaporator 14 are connected by pipes filled with refrigerant to form a closed pipeline, constituting a refrigeration system or heating system capable of circulating refrigerant.
[0067] Among them, the compressor refers to a driven fluid machinery used to increase the low-pressure refrigerant to a high-pressure refrigerant. The compressor can inhale low-temperature and low-pressure gaseous refrigerant, drive the piston to compress the refrigerant through the operation of the motor, and then discharge the high-temperature and high-pressure gaseous refrigerant to provide power for the refrigeration cycle. The compressor can include reciprocating compressors, screw compressors, rotary compressors, scroll compressors and centrifugal compressors, etc. The embodiments of the present application do not impose any restrictions on the specific types of compressors.
[0068] The condenser refers to a heat exchanger used to exchange heat between the refrigerant in the condenser and the air outside the condenser to achieve heat release. Specifically, the condenser may include a long pipe for accommodating the refrigerant, which may usually be made of a metal material with strong thermal conductivity such as copper, and the pipe may usually be coiled into a spiral shape. In addition, in order to improve the heat exchange efficiency of the condenser, a heat sink with excellent thermal conductivity may be provided on the pipe to increase the heat dissipation area, thereby accelerating the speed of heat exchange and improving the heat exchange efficiency. It is also possible to speed up the flow of air around the condenser by providing a blower or fan matching the condenser, thereby accelerating the speed of heat exchange and improving the heat exchange efficiency.
[0069] The throttling element is used to throttle the liquid refrigerant at room temperature and high pressure through the throttling element to become a low-temperature and low-pressure gas refrigerant, wherein the throttling element can also be called a throttling element or a regulating valve, and the throttling element can include an expansion valve, a capillary tube, etc. In addition, the throttling element can also control the flow of the refrigerant flowing through the throttling element to avoid the flow of the refrigerant flowing through the throttling element being too large or too small. Among them, if the flow of the refrigerant flowing through the throttling element is too large, the refrigerant flowing out of the throttling element will still include liquid refrigerant, and the liquid refrigerant will enter the compressor to produce liquid hammer, causing damage to the compressor; if the flow of the refrigerant flowing through the throttling element is too small, it will cause too little refrigerant to enter the compressor, reducing the working efficiency of the compressor.
[0070] The evaporator refers to a heat exchanger used to exchange heat between the refrigerant in the evaporator and the air outside the condenser to achieve heat absorption. Specifically, the evaporator may include a long pipe for accommodating the refrigerant, which may usually be made of a metal material with strong thermal conductivity such as copper, and the pipe may usually be coiled into a spiral shape. In addition, in order to improve the heat exchange efficiency of the condenser, a heat sink with excellent thermal conductivity may be provided on the pipe to increase the heat dissipation area, thereby accelerating the speed of heat exchange and improving the heat exchange efficiency. It is also possible to provide a blower or fan matching the evaporator to speed up the flow of air around the evaporator, thereby accelerating the speed of heat exchange and improving the heat exchange efficiency.
[0071] Refrigerant can also be called refrigerant, refrigerant or snow seed, which refers to the medium substance that completes energy conversion in the refrigeration system or heating system. Refrigerant is usually a substance that is easy to undergo reversible phase change (such as absorbing heat to become gas, releasing heat to become liquid). Through reversible phase change, refrigerant can transfer heat. Specifically, when the gaseous refrigerant is under pressure, it releases heat and becomes liquid. When the high-pressure liquid is decompressed and becomes gas, it absorbs heat. Refrigerants can include ammonia, air, water, salt water, Freon (also known as chlorofluorocarbon, chlorofluorocarbon), etc., among which Freon can include trichloromonofluoromethane, dichloromonofluoromethane, trifluoromethane, tetrafluoroethane, trichlorodifluoroethane, etc.
[0072] When the display cabinet is a display cabinet with refrigeration function, low-temperature and low-pressure vapor refrigerant flows from the evaporator into the compressor, and the compressor compresses the low-temperature and low-pressure vapor refrigerant and makes the high-temperature and high-pressure gaseous refrigerant flow into the condenser; the high-temperature and high-pressure gaseous refrigerant exchanges heat with the air outside the condenser through the condenser, so that the high-temperature and high-pressure gaseous refrigerant is cooled into a liquid refrigerant at normal temperature and high pressure in the condenser, and then the liquid refrigerant at normal temperature and high pressure flows into the throttling element, and the throttling element throttles the liquid refrigerant at normal temperature and high pressure, so that the refrigerant flowing out of the throttling element is converted into a liquid refrigerant at low temperature and low pressure; the liquid refrigerant at low temperature and low pressure flows into the evaporator, and the liquid refrigerant at low temperature and low pressure exchanges heat with the air outside the evaporator through the evaporator, and the liquid refrigerant at low temperature and low pressure evaporates and gasifies into a gaseous refrigerant at low temperature and low pressure to absorb heat. The air outside the evaporator can be introduced into the storage area of the display cabinet, and the air outside the condenser can be introduced into the outside of the display cabinet, thereby transferring the heat in the storage area of the display cabinet to the outside of the display cabinet to cool the storage area of the display cabinet.
[0073] When the display cabinet is a display cabinet with a heating function, the low-temperature and low-pressure vapor refrigerant flows from the condenser into the compressor, and the compressor compresses the low-temperature and low-pressure vapor refrigerant and allows the high-temperature and high-pressure gaseous refrigerant to flow into the evaporator; the high-temperature and high-pressure gaseous refrigerant exchanges heat with the air outside the evaporator through the evaporator, so that the high-temperature and high-pressure gaseous refrigerant is cooled into a normal-temperature and high-pressure liquid refrigerant in the evaporator, and then the normal-temperature and high-pressure liquid refrigerant flows into the throttling element, and the throttling element throttles the normal-temperature and high-pressure liquid refrigerant, so that the refrigerant flowing out of the throttling element is converted into a low-temperature and low-pressure liquid refrigerant; the low-temperature and low-pressure liquid refrigerant flows into the condenser, and the low-temperature and low-pressure liquid refrigerant exchanges heat with the air outside the condenser through the condenser, and the low-temperature and low-pressure liquid refrigerant evaporates and gasifies into a low-temperature and low-pressure gaseous refrigerant to absorb heat. The air outside the evaporator can be introduced into the storage area of the display cabinet, and the air outside the condenser can be introduced into the outside of the display cabinet, thereby transferring the heat outside the display cabinet to the storage area of the display cabinet to heat the storage area of the display cabinet.
[0074] In one embodiment of the present application, a display cabinet includes a cabinet body and a cabinet door, wherein a control board and a power management module may be disposed in the cabinet body, and a main board may be disposed in the cabinet door.
[0075] In one embodiment of the present application, Figure 2 A schematic structural block diagram of a mainboard according to an embodiment of the present disclosure is shown. Figure 2 As shown, the mainboard 200 includes a processor 201 , a random access memory 202 , a flash memory 203 , a wireless local area network Bluetooth module 204 , a gyroscope 205 , a pressure sensor 206 , a microphone 207 , a speaker 208 , a camera 209 and a cellular communication module 210 .
[0076] The processor may include one or more processing units, for example, the processor may include one or more of an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a memory, a video codec, a digital signal processor, a baseband processor, and / or a neural network processor. Different processing units may be independent devices or integrated into one or more processors.
[0077] Among them, the image signal processor is used to process the data fed back by the camera. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the image signal processor for processing and converts it into an image visible to the naked eye. The image signal processor can also perform algorithm optimization on the noise, brightness, and skin color of the image. The image signal processor can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the image signal processor can be set in the camera.
[0078] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, digital signal processors can be used to perform Fourier transform on frequency point energy.
[0079] Video codecs are used to compress or decompress digital video. A showcase can support one or more video codecs. In this way, the showcase can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0080] Neural network computing processors can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. Neural network computing processors can realize applications such as intelligent cognition of display cabinets, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0081] In some embodiments, the processor may include one or more interfaces. The interface may include an integrated circuit interface, an integrated circuit built-in audio interface, a pulse code modulation interface, a universal asynchronous receiver and transmitter interface, a mobile industry processor interface, a general purpose input and output interface, a subscriber identity module interface, and / or a universal serial bus interface, etc.
[0082] The random access memory 202 can be used to store computer executable program codes, which include instructions and data. The processor 201 executes various functional applications and data processing of the showcase by running the instructions stored in the random access memory 202. The random access memory 202 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data (such as audio data, image data, etc.) created during the use of the showcase, etc.
[0083] The flash memory 203 can be used to realize the storage capacity of the extended display cabinet. The flash memory 203 can communicate with the processor 201 through the flash memory interface to realize the data storage function. For example, files such as music and video can be stored in the flash memory.
[0084] The processor 201 , the random access memory 202 and the flash memory 203 can form a minimum system to provide a system operating environment.
[0085] The wireless LAN Bluetooth module 204 can provide wireless communication solutions including wireless LAN, Bluetooth, global navigation satellite system, frequency modulation, short-range wireless communication technology, infrared technology, etc. applied to the display cabinet. The wireless LAN Bluetooth module 204 can be one or more devices integrating at least one communication processing module. The wireless LAN Bluetooth module 204 receives electromagnetic waves via an antenna, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 201. The wireless LAN Bluetooth module 204 can also receive a signal to be sent from the processor 201, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna. In one embodiment of the present application, communication with the user's terminal can be performed through the wireless LAN Bluetooth module.
[0086] The cellular communication module 210 can provide a solution for wireless communications including 2G / 3G / 4G / 5G applied to the display cabinet. The cellular communication module 210 may include at least one filter, a switch, a power amplifier, a low noise amplifier, etc. The cellular communication module 210 can receive electromagnetic waves by an antenna, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The cellular communication module 210 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna. In some embodiments, at least some of the functional modules of the cellular communication module 210 can be set in the processor 201. In some embodiments, at least some of the functional modules of the cellular communication module 210 can be set in the same device as at least some of the modules of the processor 201. In one embodiment of the present application, the cellular communication module 210 can communicate with the cloud server of the display cabinet control service provider.
[0087] Through the wireless LAN Bluetooth module 204 and the cellular communication module 210, the display cabinet can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications, general packet radio service, code division multiple access, wideband code division multiple access, time division code division multiple access, long term evolution, etc.
[0088] The gyroscope 205 can be used to determine the real-time attitude of the display case door.
[0089] The pressure sensor 206 is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 206 can be set on the display screen. There are many types of pressure sensors 206, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 206, the capacitance between the electrodes changes, and the intensity of the pressure is determined according to the change in capacitance. When a touch operation acts on the display screen, the touch operation intensity is detected by the pressure sensor 206, and the touch position can also be calculated according to the detection signal of the pressure sensor 206. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the beverage selection application icon, an instruction to view the specific information of the beverage is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the beverage selection application icon, an instruction to purchase a beverage is executed.
[0090] Microphone 207, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 207 with their mouth to input the sound signal into the microphone 207. The display cabinet can be provided with at least one microphone 207. In other embodiments, the display cabinet can be provided with two microphones 207, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the display cabinet can also be provided with three, four or more microphones 207 to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc. In one embodiment of the present application, the sound of the display cabinet when it is running can be collected by the microphone 207.
[0091] The speaker 208, also called a "loudspeaker", is used to convert the audio electrical signal into a sound signal. The display cabinet can play music or a prompt voice through the speaker 208.
[0092] The camera 209 is used to capture images, which include static images and dynamic images (i.e., videos). The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device or a complementary metal oxide semiconductor phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the image signal processor to convert it into a digital image signal. The image signal processor outputs the digital image signal to the digital signal processor for processing. The digital signal processor converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the display cabinet may include one or more cameras 209. In one embodiment of the present application, the camera 209 may have the function of heating itself to ensure that its lens does not fog up.
[0093] In one embodiment of the present application, Figure 3 A schematic structural block diagram of a control panel according to an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the control board 300 includes a power input interface 301, a power output interface 302, a metering chip 303, a micro control unit chip 304, a real-time clock chip, a light switch interface 305, a temperature control switch interface 306, an evaporating fan interface 307, a compressor interface 308, a condensing fan interface 309, a temperature sensor interface 310, a communication interface 311 and a power interface 312.
[0094] Among them, the metering chip 303 is an electric quantity sensor, and voltage data, current data, real-time power data and average power data can be obtained through the metering chip 303. The time of the microcontrol unit chip 304 can be maintained through the real-time clock chip. The control signal of the light switch of the display cabinet can be received through the light switch interface 305. The control signal of the temperature control switch of the display cabinet can be received through the temperature control switch interface 306. The evaporation fan control signal can be sent to the evaporation fan of the display cabinet through the evaporation fan interface 307 to control the operation of the evaporation fan. The compressor control signal can be sent to the compressor of the display cabinet through the compressor interface 308 to control the operation of the compressor. The condensing fan control signal can be sent to the condensing fan of the display cabinet through the condensing fan interface 309 to control the operation of the condensing fan. The temperature sensor data collected by one or more temperature sensors can be received through the temperature sensor interface 310 to determine the temperature value of one or more positions of the display cabinet.
[0095] In one embodiment of the present application, Figure 4 A schematic structural block diagram of a power management module according to an embodiment of the present disclosure is shown as follows: Figure 4As shown, the power management module 400 includes an AC to DC conversion module 401, a charging management module 402 and a battery 403. The power management module 400 is used to power the mainboard and the control board and manage the charging and discharging of the battery. The power management module 400 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 400 can also be set in the processor.
[0096] In one embodiment of the present application, the display cabinet further comprises a display screen. The display cabinet realizes the display function through a graphics processor, a display screen, and an application processor. The graphics processor is a microprocessor for image processing, connected to the display screen and the application processor. The graphics processor is used to perform mathematical and geometric calculations for graphics rendering. The processor may include one or more graphics processors, which execute program instructions to generate or change display information.
[0097] The display screen is used to display static images, videos, etc. The display screen includes a display panel. The display panel can be a liquid crystal display, an organic light emitting diode, an active matrix organic light emitting diode or an active matrix organic light emitting diode, a flexible light emitting diode, a quantum dot light emitting diode, etc. In some embodiments, the display cabinet may include one or more display screens.
[0098] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the display cabinet. In other embodiments of the present application, the display cabinet may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware. Exemplarily, by combining different components, the display cabinet in the embodiment of the present application can be any one of a retail cabinet, a heating cabinet, a refrigerated cabinet, a freezer, a combination cabinet, and a display cabinet.
[0099] Figure 5 A flow chart of a display cabinet control method according to an embodiment of the present disclosure is shown. Figure 5 As shown, the display cabinet control method includes the following steps S101-S103:
[0100] In step S101, a set of images to be recognized including at least one image is obtained.
[0101] In one embodiment of the present disclosure, obtaining a set of images to be identified including at least one image may be receiving a set of images to be identified sent by other devices or systems, or reading a set of images to be identified stored in a display cabinet in advance. The images in the set of images to be identified may be acquired by an image acquisition device in the display cabinet, such as a camera, or may be acquired by other image acquisition devices or image acquisition systems corresponding to the display cabinet, such as a security camera. The set of images to be identified may include one or more static images, or may include one or more dynamic images or videos. The images in the set of images to be identified may be understood as including images of all or part of the display cabinet, or may be understood as including images of all or part of the items in the display cabinet. Exemplarily, the images in the set of images to be identified may include the entrance and exit of the display cabinet. Based on the images in the set of images to be identified, the items removed from or moved into the display cabinet may be determined for statistics. Based on the statistical results, the corresponding account may be settled, or at least one of the quantity, type, and position of the items stored in the display cabinet after the items are removed or moved in may be determined based on the statistical results.
[0102] For example, Figure 6 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown. Figure 7 A schematic top view of a display cabinet according to an embodiment of the present disclosure is shown. Figure 6 as well as Figure 7 As shown, the display cabinet includes a cabinet body 501 , a cabinet door 502 and a camera 503 , wherein the cabinet body 501 includes a display area 511 and an article entrance and exit 521 , and the display area 511 is connected to the outside of the cabinet body 501 through the article entrance and exit 521 .
[0103] The cabinet door of the display cabinet may be connected to the cabinet body in a rotatable manner or in a slidable manner, or the cabinet door may be connected to the cabinet body in a folding manner. It should be noted that the embodiment of the present application does not specifically limit the manner in which the cabinet door is connected to the cabinet body. For ease of understanding, Figure 6 as well as Figure 7 Take the rotating connection between the cabinet door and the cabinet body as an example. Figure 6 as well as Figure 7 As shown, the cabinet door 502 is used to open or close the object entrance and exit 521. The camera 503 is set on the top of the cabinet body and is used to collect images of the object entrance and exit 521. According to the multiple images collected by the camera 503, a set of images to be identified can be obtained. It should be noted that the display cabinet can also include multiple cameras that collect images of the object entrance and exit from different directions. For example, Figure 8 A schematic structural diagram of a display cabinet according to an embodiment of the present disclosure is shown. Figure 8As shown, the display cabinet includes a first camera 516, a second camera 526, a third camera 536, a fourth camera 546 and a fifth camera 556. The first camera 516 collects images of the cargo entrance and exit 521 from a first direction 5161, the second camera 526 collects images of the cargo entrance and exit 521 from a second direction 5162, the third camera 536 collects images of the cargo entrance and exit 521 from a third direction 5163, the fourth camera 546 collects images of the cargo entrance and exit 521 from a fourth direction 5164, and the fifth camera 556 collects images of the cargo entrance and exit 521 from a fifth direction 5165. The first direction 5161, the second direction 5162, the third direction 5163, the fourth direction 5164 and the fifth direction 5165 are all different. According to multiple images collected by at least one of the first camera 516, the second camera 526, the third camera 536, the fourth camera 546 and the fifth camera 556, a set of images to be identified can be obtained.
[0104] In step S102, it is determined whether the set of images to be identified includes personal information images.
[0105] Among them, the personal information image is an image including at least one personal information target, and the personal information target is used to obtain the personal information of at least one person.
[0106] In one embodiment of the present disclosure, personal information can be understood as various information recorded electronically or otherwise that can identify a specific natural person alone or in combination with other information. For example, personal information can include the natural person's name, date of birth, ID number, biometric information, address, telephone number, email address, health information, whereabouts, etc. A personal information target can be understood as a target in an image that can be used to obtain at least one type of personal information. The target can exist in the image in the form of graphics, text, or a combination of graphics and text.
[0107] Specifically, the personal information target may include at least one of a face, a fingerprint, an identity document, a transportation ticket, and a medical record. Exemplarily, when any image in the set of images to be identified includes a graphic corresponding to the entire or partial face of at least one person, the image may be determined to be a personal information image, and the set of images to be identified includes personal information images.
[0108] In step S103, in response to the image set to be identified not including the personal information image, the image set to be identified is sent.
[0109] In one embodiment of the present disclosure, sending the set of images to be identified may be sending the set of images to be identified to other devices or systems, such as a cloud where a statistical platform is located, via a wired or wireless communication network. Sending the set of images to be identified may be periodically sending the set of images to be identified acquired within a previous time interval at a certain time interval, or may be sending the set of images to be identified in response to an instruction to send the set of images to be identified.
[0110] For example, a display cabinet is used to provide unmanned vending services. When a customer needs to purchase items in the display cabinet, the customer can use a mobile communication terminal to scan the QR code on the surface of the display cabinet to access the cloud authorization server corresponding to the display cabinet, and send a statistical authorization request to the cloud authorization server, that is, request to purchase the items in the display cabinet. After the cloud authorization server passes the statistical authorization request, it sends statistical authorization information to the display cabinet. The display cabinet receives the statistical authorization information sent by the cloud server, and unlocks the door of the display cabinet in response to the statistical authorization information. After the door is unlocked, the customer can open the door and take the items in the display cabinet through the item entrance and exit. After the customer finishes taking the items, the door can be closed. The camera on the display cabinet can continuously collect images of the item entrance and exit, so that the collected images include images of the items taken by the customer through the item entrance and exit, and a set of images to be identified is generated based on the collected images. The display cabinet determines whether the set of images to be identified includes personal information images, and in response to the set of images to be identified not including personal information images, sends the set of images to be identified to the cloud statistics server, so that the cloud statistics server deducts the customer's account according to the set of images to be identified. Among them, since the set of images to be identified does not include personal information images, the cloud statistics server cannot obtain any personal information of the customer based on the set of images to be identified, thereby reducing the risk of leakage of the customer's personal privacy.
[0111] For example, the display cabinet is used to provide unmanned vending service. When the maintenance personnel of the display cabinet, such as the convenience store clerk, need to replenish items to the display cabinet, they can use the mobile communication terminal to scan the QR code on the surface of the display cabinet to access the cloud authorization server corresponding to the display cabinet, and send a goods replenishment request to the cloud authorization server, that is, request to replenish items to the display cabinet. After the cloud service authorization server passes the statistical authorization request, it sends statistical authorization information to the display cabinet, that is, allows the maintenance personnel to replenish items to the display cabinet. The display cabinet receives the statistical authorization information sent by the cloud server, and unlocks the door of the display cabinet in response to the statistical authorization information. After the door is unlocked, the maintenance personnel can open the door and move the scanned items into the display cabinet through the item entrance and exit. After the maintenance personnel finishes replenishing the items, they can close the door. The camera on the display cabinet can continuously collect images of the item entrance and exit, so that the collected images include images of the maintenance personnel moving items into the display cabinet through the item entrance and exit, and generate a set of images to be identified based on the collected images. The display cabinet determines whether the image set to be identified includes the personal information image, and in response to the image set to be identified not including the personal information image, sends the image set to be identified to the cloud statistics server, so that the cloud statistics server determines the number, type and location of the items that the maintenance personnel have added to the display cabinet according to the image set to be identified, so as to determine the number, type, location, etc. of the items stored in the display cabinet after the items are removed or moved in. Among them, since the image set to be identified does not include the personal information image, the cloud statistics server cannot obtain any personal information of the maintenance personnel according to the image set to be identified, thereby reducing the risk of leakage of the personal privacy of the maintenance personnel.
[0112] For example, a display cabinet is used to provide an item retrieval service to a target user, where the target user can be understood as a user belonging to a designated unit or department. When a user needs to take an item from the display cabinet, the user can use a mobile communication terminal to scan the QR code on the surface of the display cabinet to access the server corresponding to the display cabinet, and send an identity authentication request to the server, that is, request the identity of the user to be authenticated; or the corresponding identity mark (such as a work badge, certificate, etc.) can be scanned by a scanning device on the display cabinet, and the display cabinet sends an identity authentication request to the corresponding server. When the display cabinet receives the item retrieval authorization information returned by the server, it can be understood that the server confirms that the user is the target user, and the display cabinet can unlock the door of the display cabinet in response to the item retrieval authorization information. After the door is unlocked, the user can open the door and take the items in the display cabinet through the item entrance and exit. After the user finishes taking the items, the door can be closed, and the camera on the display cabinet can continue to collect images of the item entrance and exit, so that the collected images include images of the items taken by the customer through the item entrance and exit, and a set of images to be identified is generated based on the collected images. The display cabinet determines whether the image set to be identified includes the personal information image, and in response to the image set to be identified not including the personal information image, sends the image set to be identified to the cloud statistics server, so that the cloud statistics server obtains the items taken from the display cabinet by the user according to the image set to be identified, and can further determine the types and quantities of the items remaining in the display cabinet based on the items taken from the display cabinet by the user. Among them, since the image set to be identified does not include the personal information image, the cloud statistics server cannot obtain any personal information of the customer according to the image set to be identified, thereby reducing the risk of leakage of the customer's personal privacy.
[0113] The above technical solution obtains a set of images to be identified including at least one image, and determines whether the set of images to be identified includes a personal information image. In response to the set of images to be identified not including the personal information image, the set of images to be identified is sent, wherein the personal information image is an image including at least one personal information target, and the personal information target is used to obtain the personal information of at least one person. In the above solution, since any image in the set of images to be identified sent by the display cabinet does not include the personal information target, other devices or systems receiving the set of images to be identified cannot obtain the personal information of any user based on the images in the set of images to be identified, thereby ensuring that the images sent by the display cabinet will not infringe on the personal information security of any user, and reducing the risk of user privacy being infringed.
[0114] In one implementation of the present disclosure, determining whether the set of images to be identified includes personal information images can be implemented through the following steps:
[0115] Obtain a pre-trained personal information target recognition model, and input an image in the set of images to be recognized into the personal information target recognition model to obtain personal information target recognition information output by the personal information target recognition model;
[0116] Determining whether the set of images to be identified includes personal information images according to the personal information target identification information;
[0117] The method further comprises the steps of:
[0118] Obtaining personal information target annotation information, and determining an annotated image in the set of images to be identified based on the personal information target annotation information, wherein the personal information target annotation information is used to indicate the personal information target in the annotated image;
[0119] In response to the personal information image not including the annotated image, the annotated image is used as input and the personal information target annotated information is used as output to train the personal information target recognition model.
[0120] In one embodiment of the present disclosure, the personal information target recognition model may be pre-stored in the display cabinet or obtained from other devices or systems. The personal information target recognition model may be a neural network (NN) model, a convolutional neural network (CNN) model, or a long short-term memory network (LSTM) model, etc.
[0121] In one embodiment of the present disclosure, a labeled image may be understood as an image including at least one personal information target that has not been successfully identified by a personal information target recognition model.
[0122] In this embodiment, by obtaining a pre-trained personal information target recognition model, inputting images in a set of images to be recognized into the personal information target recognition model, and judging whether the set of images to be recognized includes personal information images based on the personal information target recognition information, it is possible to ensure that the accuracy of judging whether the set of images to be recognized includes personal information images is high. Then, the personal information target annotation information is obtained, and the annotated image in the set of images to be recognized is determined based on the personal information target annotation information. In response to the personal information image not including the annotated image, the annotated image is used as input and the personal information target annotation information is used as output to train the personal information target recognition model, so that the trained personal information target recognition model can learn the rules between personal information targets and images that were not successfully recognized before, thereby ensuring that the success rate of identifying personal information targets in images based on the trained personal information target recognition model is high.
[0123] In one implementation of the present disclosure, in response to the personal information image not including the annotated image, taking the annotated image as input and the personal information target annotated information as output, before training the personal information target recognition model, the method further includes the following steps:
[0124] Receive the updated weight parameters sent by the edge server, and update the personal information target recognition model according to the updated weight parameters;
[0125] In response to the personal information image not including the annotated image, taking the annotated image as input and the personal information target annotated information as output, training a personal information target recognition model, including:
[0126] In response to the personal information image not including the annotated image, taking the annotated image as input and the personal information target annotated information as output, training the updated personal information target recognition model;
[0127] The method further comprises the steps of:
[0128] In response to the trained personal information target recognition model not converging, obtaining a gradient update vector according to the trained personal information target recognition model, and sending the gradient update vector to the edge server;
[0129] Or, in response to the convergence of the trained personal information target recognition model, the trained personal information target recognition model is stored as the target personal information target recognition model.
[0130] In one implementation of the present disclosure, the edge server is used to aggregate the gradient update vectors, and update the weight parameters of the personal information target recognition model on the edge server according to the aggregated gradient update vectors to obtain updated weight parameters. The edge server can be a cloud server or a server provided by a display cabinet control service provider. It should be noted that one edge server can correspond to one or more display cabinets. For example, the display cabinet control service provider can divide the area under its jurisdiction into multiple blocks, and multiple display cabinets in each area can correspond to one edge server.
[0131] The personal information target recognition model on the edge server can be a neural network model, a convolutional neural network model, or a long short-term memory network model, etc.
[0132] In one implementation of the present disclosure, the updated weight parameter sent by the edge server and received by the display cabinet is obtained by the edge server aggregating the gradient update vectors sent by multiple display cabinets, and updating the weight parameter of the personal information target recognition model on the edge server according to the aggregated gradient update vectors. Therefore, the updated personal information target recognition model on the display cabinet can reflect the common rules between the images in the image set to be recognized and the personal information targets learned by the personal information target recognition model on the edge server in the previous round of training. Then, the annotated image is used as input, and the personal information target annotation information is used as output to train the personal information target recognition model, so that the updated personal information target recognition model on the display cabinet can not only learn the common rules, but also learn the annotated images obtained by the display cabinet itself and the personal information targets in the annotated images, so that the trained personal information target recognition model on the display cabinet can learn the private rules between the annotated images obtained by the display cabinet itself and the personal information targets in the annotated images; when the trained personal information target recognition model does not converge, it means that the trained personal information target recognition model on the display cabinet still needs to be trained. By training the personal information target recognition model based on the trained personal information target recognition model on the display cabinet, the personal information target recognition model on the edge server can be trained according to the trained personal information target recognition model on the display cabinet. The personal information target recognition model obtains the gradient update vector and sends the gradient update vector, so that the edge server can continue to obtain the corresponding update weight parameters based on the gradient update vectors uploaded by multiple display cabinets, so as to continue to train the personal information target recognition model on each display cabinet; when the trained personal information target recognition model converges, it can be considered that the converged personal information target recognition model on the display cabinet can more accurately recognize the personal information target in the image of the image set to be recognized obtained by itself, and the converged personal information target recognition model on the display cabinet can be stored as the target personal information target recognition model, that is, a model with a higher accuracy rate for recognizing the personal information target in the image.
[0133] In the above technical solution, on the one hand, the target personal information target recognition model finally obtained can be a model that has learned both common rules and private rules, and its accuracy in identifying personal information targets in images is relatively high; on the other hand, since the process of continuing to train the personal information target recognition model on each display cabinet is jointly executed by the display cabinet and the edge server, compared with the training of the personal information target recognition model by only the display cabinet or the server alone, less processing resources are required and the training speed is faster.
[0134] In one implementation of the present disclosure, the personal information target labeling information is sent by a server, or the personal information target labeling information is input through a human-computer interaction device of the display cabinet.
[0135] In an implementation of the present disclosure, before obtaining the personal information target labeling information input through the human-computer interaction device of the display cabinet, an image in the set of images to be identified may also be displayed.
[0136] In one embodiment of the present disclosure, displaying images in the image set to be recognized may be by displaying the images in the image set to be recognized through a human-computer interaction device on a display cabinet, such as a display screen, or by displaying the images in the image set to be recognized through other devices or systems.
[0137] In one embodiment of the present disclosure, the personal information target labeling information input through the human-computer interaction device of the display cabinet can be obtained by obtaining the personal information target labeling information input by the user through at least one of a keyboard, a touch screen, and a touchpad on the display cabinet; it can also be obtained by obtaining voice information input by the user through a microphone on the display cabinet, and performing voice recognition on the voice information to obtain the personal information target labeling information; it can also be obtained by obtaining image information of the action performed by the user through a camera on the display cabinet, and performing image recognition on the image information to obtain the personal information target labeling information.
[0138] Exemplarily, the showcase can display images in the image set to be identified in a graphical user interface (GUI) displayed by the touch screen, and obtain the personal information target annotation information input action corresponding to the image in the image set to be identified displayed in the graphical user interface GUI input by the user through the touch screen, and obtain the personal information target annotation information input by the user according to the personal information target annotation information input action, wherein the graphical user interface GUI is an interactive interface of an application (APP) running on the showcase. Further, the graphical user interface GUI can also display one or more visual prompts on the touch screen for the user to perform the input action of inputting the personal information target annotation information, and the visual prompts will provide the user with a hint or reminder to input the personal information target annotation information input action. These visual prompts can be text, graphics or any combination thereof. The personal information target annotation information input action includes contacting the touch screen. In some embodiments, in addition to the visual prompts, the showcase can also provide non-visual prompts to indicate the completion progress of the personal information target annotation information input action. The non-visual prompts can include audio prompts (such as sounds) or physical prompts (such as vibrations). In some embodiments, the personal information target annotation information input action is a predetermined gesture performed on the touch screen. The gesture used herein is the movement of an object / accessory in contact with the touch screen. For example, the predetermined gesture may include contacting the touch screen at a location where the personal information target is located in an image in the set of images to be identified displayed on the touch screen (initialization gesture), and breaking contact after maintaining continuous contact with the touch screen for a time exceeding a preset contact time threshold (completing the gesture).
[0139] For ease of explanation, in the process of obtaining the scan code correction request information input by the user, and in other embodiments described below, the contact on the touch screen will be described as being performed by the user using at least one hand and one or more fingers. However, it should be understood that the contact can also be performed using any appropriate object or accessory, such as a stylus, a finger, etc. The contact may include: one or more taps on the touch screen, maintaining continuous contact with the touch screen, moving the contact point while maintaining continuous contact, interrupting contact, or any combination thereof.
[0140] The showcase detects contact on the touch screen, and if the contact does not correspond to an attempt to perform a personal information target annotation information input action, or if the contact corresponds to a failed or abandoned attempt by the user to perform a personal information target annotation information input action, then the showcase will not obtain the corresponding personal information target annotation information. For example, if the personal information target annotation information input action is to contact the touch screen at the location where the personal information target is located in the image displayed on the touch screen, and the contact is interrupted after the continuous contact with the touch screen exceeds a preset contact time threshold, and the detected contact is a series of random taps on the touch screen, then, since the contact does not correspond to the personal information target annotation information input action.
[0141] If the contact corresponds to the successful execution of the personal information target labeling information input action, that is, the user successfully executes the personal information target labeling information input action, then the display cabinet can obtain the corresponding personal information target labeling information based on the personal information target labeling information input action.
[0142] Fig. 9 A schematic diagram of a graphical user interface GUI of a display cabinet according to an embodiment of the present disclosure is shown. Fig.10 A schematic diagram of a graphical user interface GUI of a display cabinet according to an embodiment of the present disclosure is shown. Fig.11 A schematic diagram of a graphical user interface GUI of a display cabinet according to an embodiment of the present disclosure is shown. Figure 9-11 A schematic diagram of a graphical user interface GUI of a display cabinet is shown at different times during the execution of an action of inputting personal information target labeling information.
[0143] exist Fig. 9 In the example, the user indicates with a finger 51 that the finger 51 has not yet touched the touch screen 52 of the showcase, and at this time the showcase will not obtain the corresponding personal information target labeling information.
[0144] exist Fig.10 In the example, the user starts to input personal information target labeling information by touching the touch screen 52 of the showcase with his finger 51. Specifically, the user keeps touching the touch screen 52 with the finger 51 and moves the finger 51 along the outline of the personal information target 54 in the image displayed by the graphical user interface GUI 53, and the showcase starts to record the movement track 55 of the finger 51. It should be noted that when the finger 51 is not out of contact with the touch screen 52, the showcase will not obtain the corresponding personal information target labeling information based on the recorded movement track.
[0145] exist Fig.11In the example, the user ends the personal information target annotation information input action by releasing the finger 51 from the touch screen 52 of the showcase. Specifically, in response to the finger 51 being released from the touch screen 52 for more than a preset time difference, the showcase determines at least one personal information target 54 in the image displayed by the graphical user interface GUI 53 based on the recorded movement trajectory, and obtains personal information target annotation information for indicating the personal information target 54 in the image, i.e., the annotated image.
[0146] In one implementation of the present disclosure, receiving the personal information target labeling information sent by the server can be understood as receiving the personal information target labeling information sent by the server in a wired or wireless manner.
[0147] In this embodiment, since the computing and processing capabilities of the server are often stronger than those of the display cabinet, and the user's ability to understand his or her own images is often stronger than that of the display cabinet, the accuracy of both the server and the user in identifying the personal information target in the image set to be identified is much higher than that of the display cabinet itself. Therefore, by receiving the personal information target annotation information sent by the server, or obtaining the personal information target annotation information input by the user through the human-computer interaction device of the display cabinet, the personal information target recognition model that is further trained based on the personal information target annotation information can learn the rules between the personal information target and the image that was not successfully identified before, thereby ensuring that the success rate of identifying the personal information target in the image based on the trained personal information target recognition model is high.
[0148] In one implementation of the present disclosure, determining whether the set of images to be identified includes personal information images can be implemented through the following steps:
[0149] Perform image recognition on the images in the image set to be identified, and determine whether the image set to be identified includes personal information images based on the image recognition results.
[0150] In one embodiment of the present disclosure, performing image recognition on images in the set of images to be recognized can be understood as recognizing the targets included in the images in the set of images to be recognized, and determining whether the targets included in the images in the set of images to be recognized include personal information targets based on the recognition results. Exemplarily, facial image recognition can be performed on the images in the set of images to be recognized. When it is determined based on facial image recognition that at least one image in the set of images to be recognized includes all or part of the face of at least one person, it can be determined that the set of images to be recognized includes personal information images. Character recognition can also be performed on the images in the set of images to be recognized. When it is determined based on the character recognition results that at least one image in the set of images to be recognized includes one or more characters, and at least one of the name, date of birth, identity document number, address, telephone number, and email address of at least one person can be obtained based on the one or more characters, it can be determined that the set of images to be recognized includes personal information images.
[0151] In this embodiment, by performing image recognition on images in the image set to be identified and judging whether the image set to be identified includes personal information images based on the image recognition results, the accuracy of judging whether the image set to be identified includes personal information images can be improved.
[0152] In an implementation of the present disclosure, the images in the set of images to be recognized are captured in response to a door of a display cabinet being unlocked.
[0153] In one embodiment of the present disclosure, the cabinet door of the display cabinet is unlocked, which can be understood as the cabinet door locking device on the display cabinet unlocking the cabinet door in response to the unlocking instruction, so that the cabinet door can rotate or slide relative to the cabinet body of the display cabinet to open the object exit and entrance of the display cabinet; it can also be understood as the cabinet door locking device on the display cabinet is triggered by the corresponding unlocking operation, so that the cabinet door locking device itself is set to an unlocked state, thereby allowing the cabinet door to rotate or slide relative to the cabinet body of the display cabinet to open the object exit and entrance of the display cabinet.
[0154] In one embodiment of the present disclosure, to determine whether a cabinet door is locked, cabinet door locking status information sent by a cabinet door locking device on a display cabinet can be received, and whether the cabinet door is locked can be determined based on the cabinet door locking status information. Cabinet door locking status information sent by other devices or systems can also be received, and whether the cabinet door is locked can be determined based on the cabinet door locking status information. For example, an infrared detection device independent of the display cabinet can detect whether the cabinet door is locked, and the cabinet door locking status information can be sent to the display cabinet based on the detection result; or a camera independent of the display cabinet, such as a security camera, can obtain an image including all or part of the display cabinet, perform image recognition based on the image, and send the cabinet door locking status information to the display cabinet based on the image recognition.
[0155] In this embodiment, by limiting the images in the image set to be identified to those collected in response to the door of the display cabinet being unlocked, the number of images in the image set to be identified can be reduced. Since items in the display cabinet can be moved out or into the display cabinet only when the door of the display cabinet is unlocked, the above scheme can reduce the amount of calculation required to determine whether the image set to be identified includes personal information images without reducing the accuracy of the detection results of objects being moved out or into the display cabinet determined based on the images in the image set to be identified, thereby saving data processing resources.
[0156] In one implementation of the present disclosure, the method further includes the following steps:
[0157] In response to the set of images to be identified including the personal information image, the personal information image is removed from the set of images to be identified to obtain a filtered set of images to be identified, and the filtered set of images to be identified is sent; and / or
[0158] In response to the set of images to be identified including personal information images, the area where the personal information target is located in the personal information image is covered according to a preset pattern, or the area where the personal information target is located is removed from the personal information image to obtain a modified set of images to be identified, and the modified set of images to be identified is sent.
[0159] In one embodiment of the present disclosure, the preset pattern may be obtained based on a preset picture pre-stored in the display cabinet. For example, the preset pattern may be obtained by cropping the preset picture pre-stored in the display cabinet based on the size of the area where the personal information target is located. The preset pattern may also be generated according to a preset algorithm in the display cabinet. For example, the preset pattern may be a mosaic pattern generated based on a preset algorithm.
[0160] In this embodiment, in response to the image set to be identified including the personal information image, the personal information image is removed from the image set to be identified to obtain a filtered image set to be identified, and the filtered image set to be identified is sent, so as to ensure that any image in the image set to be identified does not include the personal information target; and in response to the image set to be identified including the personal information image, the area where the personal information target is located in the personal information image is covered according to a preset pattern, or the area where the personal information target is located is removed from the personal information image to obtain a modified image set to be identified, so as to ensure that any image in the image set to be identified does not include the personal information target, or any personal information target cannot be successfully identified based on any image in the image set to be identified, so that other devices or systems receiving the image set to be identified cannot obtain the personal information of any user based on the images in the image set to be identified, thereby ensuring that the images sent by the display cabinet will not infringe on the personal information security of any user, thereby reducing the risk of user privacy infringement.
[0161] The trained personal information target recognition model did not converge. The trained personal information target recognition model converged. The trained personal information target recognition model did not converge. The trained personal information target recognition model converged.
[0162] Fig.12 The overall flow chart of the display cabinet control method according to an embodiment of the present disclosure is shown as follows: Fig.12 As shown, the display cabinet control method includes:
[0163] In step S201, a set of images to be recognized is obtained, including at least one image captured in response to a door of a display cabinet being unlocked.
[0164] In step S202, image recognition is performed on the images in the image set to be recognized, and it is determined whether the image set to be recognized includes personal information images based on the image recognition results.
[0165] In step S203, a pre-trained personal information target recognition model is obtained, and an image in the image set to be recognized is input into the personal information target recognition model to obtain personal information target recognition information output by the personal information target recognition model.
[0166] In step S204, it is determined whether the set of images to be identified includes a personal information image according to the personal information target identification information.
[0167] In step S205, in response to the image set to be identified not including the personal information image, the image set to be identified is sent.
[0168] In step S206, images in the set of images to be identified are displayed.
[0169] In step S207, the personal information target annotation information input through the human-computer interaction device of the showcase is obtained, and the annotated image in the set of images to be identified is determined according to the personal information target annotation information.
[0170] In step S208, it is determined whether the personal information image includes a marked image.
[0171] In step S209, the updated weight parameters sent by the edge server are received, and the personal information target recognition model is updated according to the updated weight parameters.
[0172] In step S210, in response to the personal information image not including the annotated image, the annotated image is used as input and the personal information target annotated information is used as output to train the updated personal information target recognition model.
[0173] In step S211, in response to the trained personal information target recognition model not converging, a gradient update vector is obtained according to the trained personal information target recognition model, and the gradient update vector is sent to the edge server.
[0174] In step S212, in response to the trained personal information target recognition model converging, the trained personal information target recognition model is stored as a target personal information target recognition model.
[0175] In step S213, misrecognition indication information is received.
[0176] In step S214, the misidentified image is used as input and the misidentification indication information is used as output to train the personal information target recognition model.
[0177] In step S215, in response to the image set to be identified including the personal information image, the personal information image is removed from the image set to be identified to obtain a filtered image set to be identified, and the filtered image set to be identified is sent.
[0178] In step S216, in response to the set of images to be identified including personal information images, the area where the personal information target is located in the personal information image is covered according to a preset pattern, or the area where the personal information target is located is removed from the personal information image to obtain a modified set of images to be identified, and the modified set of images to be identified is sent.
[0179] The following are embodiments of the apparatus of the present disclosure, which can be used to execute embodiments of the method of the present disclosure.
[0180] Fig.13 FIG. 1 is a schematic structural block diagram of a display cabinet control device according to an embodiment of the present disclosure. The display cabinet control device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Fig.13 As shown, the display cabinet control device includes:
[0181] The image acquisition module 601 is configured to acquire a set of images to be identified including at least one image, wherein the images in the set of images to be identified are acquired in response to the door of the display cabinet being unlocked;
[0182] An image recognition module 602 is configured to determine whether the set of images to be recognized includes a personal information image, where the personal information image is an image including at least one personal information object, and the personal information object is used to obtain personal information of at least one person;
[0183] The image sending module 603 is configured to send the set of images to be identified in response to the set of images to be identified not including the personal information image.
[0184] The above technical solution obtains a set of images to be identified including at least one image, and determines whether the set of images to be identified includes a personal information image. In response to the set of images to be identified not including the personal information image, the set of images to be identified is sent, wherein the personal information image is an image including at least one personal information target, and the personal information target is used to obtain the personal information of at least one person. In the above solution, since any image in the set of images to be identified sent by the display cabinet does not include the personal information target, other devices or systems receiving the set of images to be identified cannot obtain the personal information of any user based on the images in the set of images to be identified, thereby ensuring that the images sent by the display cabinet will not infringe on the personal information security of any user, and reducing the risk of user privacy being infringed.
[0185] The present disclosure also discloses an electronic device, Fig.14 A schematic structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Fig.14 As shown, the electronic device 700 includes a memory 701 and a processor 702; wherein the memory 701 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 702 to implement the above method steps.
[0186] Fig.15 Schematic diagram of a computer system suitable for implementing a display cabinet control method according to an embodiment of the present disclosure. Fig.15 As shown, the computer system 800 includes a processing unit 801, which can perform various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The processing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0187] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. The drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that the computer program read therefrom is installed into the storage section 808 as needed. Among them, the processing unit 801 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.
[0188] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the road map or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0189] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.
[0190] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.
[0191] In addition, the present disclosure also provides a computer program product, in which a computer program is stored. When the computer program is executed by a processor, the processor can at least implement the method provided in the aforementioned embodiment.
[0192] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.
Claims
1. A display cabinet control method, characterized in that: The method comprises: Acquire a set of images to be identified including at least one image; the images in the set of images to be identified are collected in response to the door of the display cabinet being unlocked, and are images including all or part of the items in the display cabinet, and are used to determine the items moved out of the display cabinet or moved into the display cabinet; Determining whether the set of images to be identified includes a personal information image, wherein the personal information image is an image including at least one personal information target, and the personal information target is used to obtain personal information of at least one person; In response to the set of images to be identified not including the personal information image, the set of images to be identified is sent.
2. The display cabinet control method according to claim 1, characterized in that: The step of judging whether the set of images to be identified includes personal information images includes: Obtain a pre-trained personal information target recognition model, and input the images in the set of images to be recognized into the personal information target recognition model to obtain personal information target recognition information output by the personal information target recognition model; Determining whether the set of images to be identified includes a personal information image according to the personal information target identification information; The method further comprises: Acquire personal information target annotation information, and determine an annotated image in the set of images to be identified according to the personal information target annotation information, wherein the personal information target annotation information is used to indicate the personal information target in the annotated image; In response to the personal information image not including the annotated image, the annotated image is used as input, and the personal information target annotated information is used as output to train the personal information target recognition model.
3. The display cabinet control method according to claim 2, characterized in that: In response to the personal information image not including the annotated image, taking the annotated image as input and the personal information target annotated information as output, before training the personal information target recognition model, the method further includes: Receiving the updated weight parameters sent by the edge server, and updating the personal information target recognition model according to the updated weight parameters; In response to the personal information image not including the annotated image, taking the annotated image as input and the personal information target annotated information as output, training the personal information target recognition model, comprising: In response to the personal information image not including the annotated image, taking the annotated image as input and the personal information target annotated information as output, training an updated personal information target recognition model; The method further comprises: In response to the trained personal information target recognition model not converging, obtaining a gradient update vector according to the trained personal information target recognition model, and sending the gradient update vector to the edge server; Or, in response to the convergence of the trained personal information target recognition model, the trained personal information target recognition model is stored as the target personal information target recognition model.
4. The display cabinet control method according to claim 2, characterized in that: The personal information target marking information is sent by a server, or the personal information target marking information is input through a human-computer interaction device of the display cabinet.
5. The display cabinet control method according to claim 1, characterized in that: The step of judging whether the set of images to be identified includes personal information images includes: Perform image recognition on the images in the set of images to be recognized, and determine whether the set of images to be recognized includes personal information images based on the image recognition results.
6. The display cabinet control method according to any one of claims 1 to 5, characterized in that: The method further comprises: In response to the set of images to be identified including the personal information image, removing the personal information image from the set of images to be identified to obtain a filtered set of images to be identified, and sending the filtered set of images to be identified; and / or In response to the set of images to be identified including the personal information image, the area where the personal information target in the personal information image is located is covered according to a preset pattern, or the area where the personal information target is located is removed from the personal information image to obtain a modified set of images to be identified, and the modified set of images to be identified is sent.
7. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 6.
8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.
