Control method, module, electronic device and readable medium for refrigerator display screen

Abstract

The present application relates to a control method, module, electronic device and readable medium for a refrigerator display screen, wherein the method includes: collecting image data of a target area in a preset direction of the refrigerator and point cloud data of the space where the refrigerator is located; identifying image data, and determining the user's target sight direction when a user is identified in the image data, and determining the user's target walking direction based on the point cloud data; judging whether the target walking direction and the target sight direction are both the directions where the refrigerator is located, and controlling the display screen to light up when the target walking direction and the target sight direction are both the directions where the refrigerator is located. By predicting whether the user intends to view the display screen based on the user's motion trajectory and sight direction, and then determining whether to turn on the display screen based on the user's intention, the problem of accidentally turning on the display screen due to the user's walking is solved.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a control method, module, electronic device and readable medium for a refrigerator display screen. Background Art

[0002] With the development of smart homes and the improvement of living standards, users have put forward more functional requirements for refrigerators, such as smart networking and audio and video playback. As an important part of human-computer interaction, the structural design of the display screen is also facing continuous improvement and upgrading. The function of the display screen has also gradually developed from the initial graphic information display and touch buttons to an intelligent operation display interface. In order to facilitate users to view or operate the display screen, the display screen is generally set to always on or directly control whether to light up the display screen based on the distance between the user and the refrigerator. However, there are often users near the refrigerator. It is easy to turn on the display screen by mistake only based on the distance between the user and the refrigerator, which will generate additional power consumption.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The present application provides a control method, module, electronic device and readable medium for a refrigerator display screen to solve the technical problem of the display screen being accidentally turned on due to the user walking around.

[0005] According to one aspect of an embodiment of the present application, the present application provides a method for controlling a refrigerator display screen, including: collecting image data of a target area in a preset direction of the refrigerator and point cloud data of the space where the refrigerator is located; identifying the image data, and when a user is identified in the image data, determining the user's target sight direction, and determining the user's target walking direction based on the point cloud data; judging whether the target walking direction and the target sight direction are both the direction where the refrigerator is located, and controlling the display screen to light up when the target walking direction and the target sight direction are both the direction where the refrigerator is located.

[0006] Optionally, determining the target walking direction of the user based on the point cloud data includes: generating a motion trajectory of the user using the point cloud data; and inputting the motion trajectory into a direction prediction model to determine the target walking direction of the user through the direction prediction model.

[0007] Optionally, using point cloud data to generate the user's movement trajectory includes: dividing the point cloud data according to time to obtain multiple frames of point cloud data; reconstructing the space in three dimensions based on each frame of point cloud data to obtain a three-dimensional model of the space at multiple moments; tracing the user's location in multiple three-dimensional models in chronological order to obtain the user's movement trajectory in the space.

[0008] Optionally, generating the user's movement trajectory using point cloud data also includes: extracting first human point cloud data and spatial point cloud data from the point cloud data; constructing a three-dimensional human body model using the first human point cloud data, and constructing a three-dimensional spatial model using the spatial point cloud data; extracting multiple second human point cloud data from the point cloud data, and using the second human point cloud data to reconstruct the path of the three-dimensional human body model in the three-dimensional spatial model to obtain the user's movement trajectory, wherein the second human point cloud data is point cloud data collected at multiple moments during the user's movement.

[0009] Optionally, determining the target sight line direction of the user includes: obtaining a face image and an eye image from image data; extracting facial features from the face image, and obtaining the initial sight line direction of the user based on the facial features; determining the eye features of the user using the eye image and the face features; fusing the facial features and the eye features to obtain fused features, and determining sight line residual information through the fused features, wherein the sight line residual information is used to characterize the sight line difference between the initial sight line direction and the target sight line direction; and adjusting the initial sight line direction using the sight line residual information to obtain the target sight line direction of the user.

[0010] Optionally, the method also includes obtaining a direction prediction model in the following manner: obtaining a historical trajectory segment of the user in space, wherein the historical trajectory segment is generated based on the user's walking record between a first moment and a second moment, and the historical trajectory segment carries a direction label, and the direction label is used to characterize the actual direction of the user at the next moment after the second moment; using the historical trajectory segment as a training set to train an initial model to output a training direction through the initial model; determining a direction error between the training direction and the actual direction of the direction label; and when the direction error is less than or equal to an error threshold, determining the initial model as a direction prediction model.

[0011] Optionally, after determining the direction error between the training direction and the actual direction of the direction label, the method further includes: when the direction error is greater than an error threshold, adjusting the model parameters of the initial model; training the adjusted initial model using the training set until the direction error between the training direction output by the initial model and the actual direction is greater than the error threshold, and determining the initial model as a direction prediction model.

[0012] According to another aspect of the embodiments of the present application, the present application provides a control module for a refrigerator display screen, including: a data acquisition module, used to collect image data of a target area in a preset direction of the refrigerator and point cloud data of the space where the refrigerator is located; a direction determination module, used to identify image data, and when a user is identified in the image data, determine the user's target line of sight direction, and determine the user's target walking direction based on the point cloud data; a direction judgment module, used to determine whether the target walking direction and the target line of sight direction are both the direction where the refrigerator is located, and when the target walking direction and the target line of sight direction are both the direction where the refrigerator is located, control the display screen to light up.

[0013] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0014] According to another aspect of an embodiment of the present application, the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, and the program code enables the processor to execute the above method.

[0015] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0016] The present application provides a method for controlling a refrigerator display screen, including: collecting image data of a target area in a preset direction of the refrigerator and point cloud data of the space where the refrigerator is located; identifying image data, and determining the user's target sight direction when a user is identified in the image data, and determining the user's target walking direction based on the point cloud data; judging whether the target walking direction and the target sight direction are both the directions where the refrigerator is located, and controlling the display screen to light up when the target walking direction and the target sight direction are both the directions where the refrigerator is located. By predicting whether the user intends to view the display screen based on the user's motion trajectory and sight direction, and then determining whether to turn on the display screen based on the user's intention, the problem of accidentally turning on the display screen due to the user's walking is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 A flowchart of an optional method for controlling a refrigerator display screen provided according to an embodiment of the present application;

[0020] Figure 2 A block diagram of an optional control module of a refrigerator display screen provided according to an embodiment of the present application;

[0021] Figure 3 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0023] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0024] With the development of smart homes and the improvement of living standards, users have put forward more functional requirements for refrigerators, such as smart networking and audio and video playback. As an important part of human-computer interaction, the structural design of the display screen is also facing continuous improvement and upgrading. The function of the display screen has also gradually developed from the initial graphic information display and touch buttons to an intelligent operation display interface. In order to facilitate users to view or operate the display screen, the display screen is generally set to always on or directly control whether to light up the display screen based on the distance between the user and the refrigerator. However, there are often users near the refrigerator. It is easy to turn on the display screen by mistake only based on the distance between the user and the refrigerator, which will generate additional power consumption.

[0025] After half a century of development, refrigerator display technology has gradually evolved from the original single and rough panels to the current exquisite, beautiful and multifunctional ones. In recent years, with the development and maturity of liquid crystal display technology, LCD (Liquid Crystal Display) has become an important part of refrigerator panel display. LCD modules have the advantages of low power consumption, low operating voltage, large amount of information, and long life. They play an increasingly important role in scientific research, production, product design and other fields, making the display screen of refrigerator panels gradually perfect.

[0026] However, the display screen on the refrigerator door is currently either always on or can only display information after manual operation (for example, opening the refrigerator door, or manually clicking the display screen, etc.). After opening the refrigerator door, the user needs to turn around and look at the display screen to understand the status of the refrigerator, which is very inconvenient.

[0027] But for users, they hope that the display screen on the ice will be more intelligent and user-friendly, rather than simply turning on the display screen after operation.

[0028] Although some can light up the display screen through induction, human body induction is usually used. As long as a person passes by, the refrigerator display screen will be lit up. This indiscriminate electric screen often lights up the display screen due to accidental operation.

[0029] In order to solve the problems mentioned in the background technology, according to one aspect of an embodiment of the present application, a method for controlling a refrigerator display screen is provided, such as Figure 1 As shown, including:

[0030] Step 101, collecting image data of a target area in a preset direction of the refrigerator and point cloud data of the space where the refrigerator is located;

[0031] Step 103, identifying the image data, and if a user is identified in the image data, determining the target sight direction of the user, and determining the target walking direction of the user according to the point cloud data;

[0032] Step 105, determining whether the target walking direction and the target sight direction are both the directions where the refrigerator is located, and controlling the display screen to light up when the target walking direction and the target sight direction are both the directions where the refrigerator is located.

[0033] The present application is applied to a smart refrigerator, and in particular to display screen control of a smart refrigerator.

[0034] Optionally, image data of a target area in a preset direction of the refrigerator is collected in order to detect whether a user enters the target area in the preset direction of the refrigerator. The preset direction may be the front of the refrigerator door (other directions may also be set as preset directions according to the placement of the refrigerator). Then the target area is the front area of ​​the refrigerator door. The shape of the area is not fixed and may be a sector, a rectangle or other shapes.

[0035] Optionally, collecting image data of a target area in a preset direction of the refrigerator is another way to determine the user's target sight direction based on the image data.

[0036] Specifically, the millimeter wave radar device can be used to collect point cloud data. The point cloud data of the space where the refrigerator is located is collected in order to construct a three-dimensional space model of the space where the refrigerator is located based on the point cloud data and trace the user's movement trajectory, thereby determining the user's target movement direction.

[0037] This application comprehensively judges whether the user wants to go to the refrigerator to use the display screen based on the target line of sight direction and the target movement direction, and then controls whether the display screen is turned on based on the judgment result. By predicting the user's intention, the display screen is turned on in advance and the refrigerator status information is displayed, thereby improving the user's experience.

[0038] As an optional embodiment, determining the user's target walking direction based on point cloud data includes: generating the user's motion trajectory using the point cloud data; and inputting the motion trajectory into a direction prediction model to determine the user's target walking direction through the direction prediction model.

[0039] Optionally, the direction prediction model is pre-trained so that the direction prediction model can predict the walking direction at the next moment according to the input action trajectory.

[0040] It should be noted that if there are multiple users, multiple direction prediction models are provided for the accuracy of prediction, that is, each user has a corresponding direction prediction model (the training set is the historical action trajectory of each corresponding user). Therefore, after collecting the user image, the user identity can be further identified and the direction prediction model matching the user identity can be called.

[0041] Next, the steps of generating the user's movement trajectory using point cloud data are described.

[0042] As an optional embodiment, generating a user's movement trajectory using point cloud data includes: dividing the point cloud data according to time to obtain multiple frames of point cloud data; reconstructing the space in three dimensions based on each frame of point cloud data to obtain a three-dimensional model of the space at multiple moments; tracing the user's location in multiple three-dimensional models in chronological order to obtain the user's movement trajectory in the space.

[0043] The point cloud data is divided according to time to obtain multi-frame point cloud data. The time division gradient can be 0.1 second or 0.5 second. The specific gradient value can be adjusted according to the size of the space or other factors. If the space is larger, the division gradient can be larger. If the space is smaller, the division gradient can be smaller. This application does not limit this.

[0044] It can be simply understood as first determining the user's location at each time, and then connecting each point to generate a movement trajectory.

[0045] As an optional embodiment, generating a user's movement trajectory using point cloud data also includes: extracting first human point cloud data and spatial point cloud data from the point cloud data; constructing a three-dimensional human body model using the first human point cloud data, and constructing a three-dimensional space model using the spatial point cloud data; extracting multiple second human point cloud data from the point cloud data, and using the second human point cloud data to reconstruct the path of the three-dimensional human body model in the three-dimensional space model to obtain the user's movement trajectory, wherein the second human point cloud data is point cloud data collected at multiple moments during the user's movement.

[0046] The present application also provides another embodiment of generating a motion trajectory using point cloud data. First, a three-dimensional space model and a three-dimensional human body model are constructed separately, and then the position of the three-dimensional human body model is traced, and finally the user's motion trajectory is obtained.

[0047] The first human body point cloud data and the second human body point cloud data are both the user's human body point cloud data. The first human body point cloud data is the point cloud data when the user first enters the space where the refrigerator is located, and is used to capture the second human body point cloud data. The second human body point cloud data is the point cloud data collected after the user enters the space where the refrigerator is located.

[0048] Specifically, first human point cloud data is extracted from the point cloud data to generate a three-dimensional human model, and spatial point cloud data is extracted from the point cloud data to generate a three-dimensional spatial model, and then second human point cloud data is extracted from the point cloud data to capture the movement trajectory of the three-dimensional human model in the three-dimensional spatial model, that is, the movement trajectory of the user in the space where the refrigerator is located.

[0049] As an optional embodiment, determining the target sight direction of the user includes: obtaining a face image and an eye image from image data; extracting facial features from the face image, and obtaining the initial sight direction of the user based on the facial features; determining the eye features of the user using the eye image and the face features; fusing the facial features and the eye features to obtain fused features, and determining sight residual information through the fused features, wherein the sight residual information is used to characterize the sight difference between the initial sight direction and the target sight direction; and adjusting the initial sight direction using the sight residual information to obtain the target sight direction of the user.

[0050] Optionally, extracting facial features from a face image, determining eye features of a user using an eye image and facial features, and obtaining fusion features of facial features and eye features can all be achieved through a neural network model.

[0051] Specifically, the weight of each feature point in the facial feature is determined, and the facial feature is adjusted based on the weight of each feature point in the facial feature; and the initial sight line direction of the user is determined according to the adjusted facial feature.

[0052] Specifically, the facial features and eye features are fused to obtain fused features, and the line of sight residual information is determined through the fused features, including: determining intermediate features according to the adjusted facial features, eye features, and weights of each feature point in the adjusted facial features; performing weighted summation of the intermediate features and the adjusted facial features based on the intermediate features, the adjusted facial features, and the weights corresponding to the intermediate features and the adjusted facial features, to obtain fused features.

[0053] Likewise, intermediate features can be determined by a pre-trained neural network.

[0054] For example, if the user's actual line of sight direction is 15 degrees (positive 15 degrees is above the horizontal direction, negative 15 degrees is below the horizontal direction) and 45 degrees east of south, and the initial line of sight direction of the user predicted by the user's facial features is 18 degrees from the horizontal direction and 49 degrees east of south, and the line of sight residual information obtained by fusing the facial features and eye features has a deviation of 3 degrees, then the initial line of sight direction is corrected according to the line of sight residual information, and the predicted user line of sight direction can be obtained as 15 degrees (i.e. (18-3) degrees) from the horizontal direction and 46 degrees east of south (i.e. (49-3) degrees). Obviously, the corrected angle is closer to the user's actual line of sight direction.

[0055] Since the target point is often where the user's gaze is, it is possible to determine whether the user's intended point is the refrigerator based on the line of sight. After determining the user's intention, the display screen will be turned on, which can provide services to the user in a more intelligent way.

[0056] As an optional embodiment, the method also includes obtaining a direction prediction model in the following manner: obtaining a historical trajectory segment of the user in space, wherein the historical trajectory segment is generated based on the user's walking record between a first moment and a second moment, and the historical trajectory segment carries a direction label, and the direction label is used to characterize the actual direction of the user at the next moment after the second moment; using the historical trajectory segment as a training set to train an initial model to output a training direction through the initial model; determining a direction error between the training direction and the actual direction of the direction label; and when the direction error is less than or equal to an error threshold, determining the initial model as a direction prediction model.

[0057] Optionally, the error threshold is any value close to zero, and the specific value can be set according to actual conditions, which is not limited in this application.

[0058] Specifically, the training set of the direction prediction model is a large number of walking trajectory fragments of users in the space (i.e., the walking trajectory of users between the first moment and the second moment), where the first moment and the second moment are only used to represent the time period and are not limited to specific moments.

[0059] As an optional embodiment, after determining the direction error between the training direction and the actual direction of the direction label, the method also includes: when the direction error is greater than an error threshold, adjusting the model parameters of the initial model; training the adjusted initial model using the training set until the direction error between the training direction output by the initial model and the actual direction is greater than the error threshold, and determining the initial model as a direction prediction model.

[0060] The present application also provides an optional embodiment, which can determine the user's intention based solely on the user's target sight direction or target walking direction. When the user's target sight direction is the direction of the refrigerator, it is determined that the user wants to open the refrigerator, that is, there is a need to view the display screen, and then the display screen is controlled to light up; or when the user's target walking direction is the direction of the refrigerator, it is determined that the user wants to operate the refrigerator, that is, there is a need to view the display screen, and then the display screen is controlled to light up.

[0061] The present application provides a method for controlling a refrigerator display screen, including: collecting image data of a target area in a preset direction of the refrigerator and point cloud data of the space where the refrigerator is located; identifying image data, and determining the user's target sight direction when a user is identified in the image data, and determining the user's target walking direction based on the point cloud data; judging whether the target walking direction and the target sight direction are both the directions where the refrigerator is located, and controlling the display screen to light up when the target walking direction and the target sight direction are both the directions where the refrigerator is located. By predicting whether the user intends to view the display screen based on the user's motion trajectory and sight direction, and then determining whether to turn on the display screen based on the user's intention, the problem of accidentally turning on the display screen due to the user's walking is solved.

[0062] According to another aspect of the embodiment of the present application, the present application provides a control module for a refrigerator display screen, such as Figure 2 As shown, including:

[0063] The data acquisition module 202 is used to acquire image data of a target area in a preset direction of the refrigerator and point cloud data of the space where the refrigerator is located;

[0064] A direction determination module 204 is used to identify the image data, and when a user is identified in the image data, determine the user's target sight direction, and determine the user's target walking direction based on the point cloud data;

[0065] The direction determination module 206 is used to determine whether the target walking direction and the target sight direction are both the directions where the refrigerator is located, and control the display screen to light up when the target walking direction and the target sight direction are both the directions where the refrigerator is located.

[0066] It should be noted that the data acquisition module 202 in this embodiment can be used to execute step 101 in the embodiment of the present application, the direction determination module 204 in this embodiment can be used to execute step 103 in the embodiment of the present application, and the direction determination module 206 in this embodiment can be used to execute step 105 in the embodiment of the present application.

[0067] Optionally, the direction determination module 204 includes:

[0068] The generation submodule is used to generate the user's action trajectory using point cloud data;

[0069] The determination submodule is used to input the action trajectory into the direction prediction model to determine the user's target walking direction through the direction prediction model.

[0070] Optionally, the generation submodule is also used to divide the point cloud data according to time to obtain multiple frames of point cloud data; reconstruct the space in three dimensions based on each frame of point cloud data to obtain a three-dimensional model of the space at multiple moments; trace the user's location in multiple three-dimensional models in chronological order to obtain the user's movement trajectory in the space.

[0071] Optionally, the generation submodule is also used to extract first human body point cloud data and spatial point cloud data from the point cloud data; use the first human body point cloud data to construct a three-dimensional human body model, and use the spatial point cloud data to construct a three-dimensional spatial model; extract multiple second human body point cloud data from the point cloud data, and use the second human body point cloud data to reconstruct the path of the three-dimensional human body model in the three-dimensional spatial model to obtain the user's movement trajectory, wherein the second human body point cloud data is point cloud data collected at multiple times during the user's movement process.

[0072] Optionally, the direction determination module 204 is also used to obtain a face image and an eye image in the image data; extract facial features from the face image, and obtain the user's initial line of sight direction based on the facial features; determine the user's eye features using the eye image and the face features; fuse the facial features and the eye features to obtain a fused feature, and determine the line of sight residual information through the fused feature, wherein the line of sight residual information is used to characterize the line of sight difference between the initial line of sight direction and the target line of sight direction; and adjust the initial line of sight direction using the line of sight residual information to obtain the user's target line of sight direction.

[0073] Optionally, the module also includes a training module, which is used to obtain a direction prediction model in the following manner: obtain a historical trajectory segment of the user in space, wherein the historical trajectory segment is generated based on the user's walking record between a first moment and a second moment, and the historical trajectory segment carries a direction label, and the direction label is used to characterize the actual direction of the user at the next moment after the second moment; use the historical trajectory segment as a training set to train the initial model to output a training direction through the initial model; determine the direction error between the training direction and the actual direction of the direction label; when the direction error is less than or equal to an error threshold, determine the initial model as a direction prediction model.

[0074] Optionally, the training module is also used to adjust the model parameters of the initial model after determining the direction error between the training direction and the actual direction of the direction label, when the direction error is greater than an error threshold; the adjusted initial model is trained using the training set until the direction error between the training direction output by the initial model and the actual direction is greater than the error threshold, and the initial model is determined as a direction prediction model.

[0075] It should be noted here that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0076] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 3 As shown, it includes a memory 301, a processor 303, a communication interface 305 and a communication bus 307. The memory 301 stores a computer program that can be run on the processor 303. The memory 301 and the processor 303 communicate through the communication interface 305 and the communication bus 307. When the processor 303 executes the computer program, the steps of the above method are implemented.

[0077] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0078] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0079] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0080] According to another aspect of the embodiments of the present application, a computer-readable medium having a non-volatile program code executable by a processor is provided.

[0081] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0082] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0083] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0084] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0085] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0087] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the module embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0090] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0091] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for controlling a refrigerator display screen, characterized in that: include: Collecting image data of a target area in a preset direction of the refrigerator and point cloud data of a space where the refrigerator is located; Identify the image data, and if a user is identified in the image data, determine the target sight direction of the user, and determine the target walking direction of the user based on the point cloud data; Determine whether the target walking direction and the target sight direction are both the direction where the refrigerator is located, and control the display screen to light up when the target walking direction and the target sight direction are both the direction where the refrigerator is located.

2. The method according to claim 1, characterized in that: Determining the target walking direction of the user according to the point cloud data includes: Generating a movement trajectory of the user using the point cloud data; The motion trajectory is input into a direction prediction model to determine the target walking direction of the user through the direction prediction model.

3. The method according to claim 2, characterized in that The generating the user's movement trajectory by using the point cloud data comprises: Dividing the point cloud data according to time to obtain multiple frames of point cloud data; Reconstructing the space in three dimensions based on each frame of point cloud data to obtain a three-dimensional model of the space at multiple moments; The location of the user is traced back in the multiple three-dimensional models in chronological order to obtain the movement trajectory of the user in the space.

4. The method according to claim 2, characterized in that: The generating the user's movement trajectory by using the point cloud data further comprises: Extracting first human body point cloud data and space point cloud data from the point cloud data; constructing a three-dimensional human body model using the first human body point cloud data, and constructing a three-dimensional space model using the space point cloud data; A plurality of second human body point cloud data are extracted from the point cloud data, and the second human body point cloud data are used to reconstruct the path of the three-dimensional human body model in the three-dimensional space model to obtain the movement trajectory of the user, wherein the second human body point cloud data are point cloud data collected at multiple moments during the movement of the user.

5. The method according to claim 1, characterized in that Determining the target sight direction of the user includes: Acquire a face image and an eye image from the image data; Extracting facial features from the facial image, and acquiring an initial sight line direction of the user based on the facial features; Determining eye features of the user using the human eye image and the facial features; The facial feature and the eye feature are fused to obtain a fused feature, so as to determine sight line residual information through the fused feature, wherein the sight line residual information is used to characterize the sight line difference between the initial sight line direction and the target sight line direction; The initial sight line direction is adjusted using the sight line residual information to obtain the target sight line direction of the user.

6. The method according to claim 2, characterized in that The method further comprises obtaining the direction prediction model in the following manner: Acquire a historical track segment of the user in the space, wherein the historical track segment is generated based on the walking record of the user between a first moment and a second moment, and the historical track segment carries a direction tag, and the direction tag is used to represent the actual direction of the user at a next moment after the second moment; Training an initial model using the historical trajectory segments as a training set, so as to output a training direction through the initial model; Determining a direction error between the training direction and the actual direction of the direction label; When the direction error is less than or equal to an error threshold, the initial model is determined as the direction prediction model.

7. The method according to claim 6, characterized in that After determining the direction error between the training direction and the actual direction of the direction label, the method further includes: When the direction error is greater than the error threshold, adjusting the model parameters of the initial model; The adjusted initial model is trained using the training set until the direction error between the training direction output by the initial model and the actual direction is greater than the error threshold, and the initial model is determined as the direction prediction model.

8. A control module for a refrigerator display screen, characterized in that: include: A data acquisition module, used to acquire image data of a target area in a preset direction of the refrigerator and point cloud data of a space where the refrigerator is located; a direction determination module, configured to identify the image data, and when a user is identified as being present in the image data, determine the target sight direction of the user, and determine the target walking direction of the user based on the point cloud data; The direction determination module is used to determine whether the target walking direction and the target sight direction are both the directions where the refrigerator is located, and control the display screen to light up when the target walking direction and the target sight direction are both the directions where the refrigerator is located.

9. An electronic device, comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate through the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the method according to any one of claims 1 to 7.

Images