Digital twin light adjustment method, device and system
By acquiring indoor images and recognizing user behavior through voice, combined with a light intensity prediction model and a keyword database, the system automatically controls lighting devices to adjust light intensity, solving the problem of insufficient intelligence in light adjustment in existing technologies and improving the user experience.
Patent Information
- Application Number
- CN202210605106.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-05-30
AI Technical Summary
Existing lighting adjustment methods cannot meet users' actual needs, have low levels of intelligence, poor user experience, and cannot automatically adjust to adapt to user behavior and ambient light.
By acquiring images and voice recordings of the current indoor scene, identifying user behavior and light intensity, and utilizing a light intensity prediction model and a preset keyword library, the system automatically controls light adjustment devices to adjust light intensity, including the use of shading and lighting equipment.
It enables automatic adjustment of light intensity based on user behavior and ambient light, improving the intelligence level of light adjustment and enhancing the comfort of users' living and working environments.
Smart Images

Figure CN115175415B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of smart home / smart family technology, and specifically, to a digital twin light adjustment method, device and system. Background Art
[0002] With the rapid development of science and technology, more and more smart devices are beginning to be used in family life, providing users with a comfortable living environment.
[0003] At present, users have a growing demand for automatic adjustment of indoor lighting. In existing family scenarios, when family members are watching TV in the living room at noon, the indoor light is strong and the TV is not clear. Users need to wake up the voice assistant to close the curtains or close them manually. When children are reading under a desk lamp at night, the light intensity needs to be adjusted manually to protect their eyesight. When children fall asleep at night, family members need to check on their status and turn off lights and close curtains.
[0004] Existing light adjustment methods usually use light sensors to identify the ambient light intensity and adjust the indoor light according to a pre-set threshold. This method cannot meet the actual needs of users, has a low level of intelligence, and provides a poor experience. Summary of the invention
[0005] The present application provides a digital twin light adjustment method, device and system, which are used to solve the technical problems that the existing light adjustment methods cannot meet the actual needs of users, have a low intelligence level and a poor experience.
[0006] The present application provides a digital twin light adjustment method, comprising:
[0007] Obtain the current scene image, current scene voice and current light intensity in the room;
[0008] Determine a user behavior recognition result in the room based on the current scene image, and determine a scene voice recognition result in the room based on the current scene voice;
[0009] Determining the user behavior recognition result based on the scene voice recognition result to determine the user behavior recognition result in the room;
[0010] An ideal illumination intensity corresponding to the user behavior identification result is determined, and based on the ideal illumination intensity and the current illumination intensity, a light adjustment device in the room is controlled to adjust the illumination intensity in the room.
[0011] According to the digital twin light adjustment method provided in the present application, the user behavior recognition result is judged based on the scene voice recognition result to determine the indoor user behavior judgment result, including:
[0012] Performing word segmentation and keyword extraction on the scene speech recognition result to obtain keywords corresponding to the scene speech recognition result;
[0013] Compare the keyword with each preset keyword in a preset keyword library to determine a user behavior prediction result corresponding to the keyword; the preset keyword library includes multiple preset keywords and a user behavior prediction result corresponding to each preset keyword;
[0014] If the user behavior prediction result corresponding to the keyword is consistent with the user behavior recognition result, the user behavior recognition result is determined as the user behavior discrimination result.
[0015] According to the digital twin light adjustment method provided in this application, determining the ideal light intensity corresponding to the user behavior discrimination result includes:
[0016] Identify the user identity in the current scene image to determine the user identity of the indoor user;
[0017] Inputting the user identity and user behavior discrimination result of the indoor user into a light intensity prediction model to obtain an ideal light intensity corresponding to the user behavior discrimination result output by the light intensity prediction model;
[0018] The illumination intensity prediction model is obtained by training based on sample user identities, sample user behaviors of the sample user identities, and sample illumination intensities corresponding to the sample user behaviors.
[0019] According to the digital twin light adjustment method provided in the present application, based on the ideal light intensity and the current light intensity, controlling the light adjustment device in the room to adjust the light intensity in the room includes:
[0020] Get outdoor light intensity;
[0021] If the outdoor light intensity is greater than or equal to a preset light intensity threshold, then based on the difference between the ideal light intensity and the current light intensity, determine the opening and closing degree adjustment amount of the indoor shading device, and based on the opening and closing degree adjustment amount, control the shading device to adjust the indoor light intensity;
[0022] If the outdoor light intensity is less than the preset light intensity threshold, the power adjustment amount of the indoor lighting device is determined based on the difference between the ideal light intensity and the current light intensity, and based on the power adjustment amount, the lighting device is controlled to adjust the indoor light intensity.
[0023] According to the digital twin light adjustment method provided in the present application, based on the ideal light intensity and the current light intensity, controlling the light adjustment device in the room to adjust the light intensity in the room includes:
[0024] Based on the current scene image, determining an activity area of the user in the room;
[0025] Determine the light adjustment device corresponding to the activity area based on the activity area and the correspondence between each light adjustment device in the room and each activity area;
[0026] Based on the ideal illumination intensity and the current illumination intensity, a light adjustment device corresponding to the active area is controlled to adjust the illumination intensity of the active area.
[0027] According to the digital twin light adjustment method provided in the present application, the user behavior recognition result is judged based on the scene voice recognition result to determine the indoor user behavior judgment result, including:
[0028] If the scene speech recognition result is empty, sending a prompt speech containing a semantically filled slot;
[0029] Receiving voice information input by the user based on the prompt voice;
[0030] Based on the transcribed text of the voice information, a behavior keyword corresponding to the semantic filling slot in the transcribed text is determined, and the behavior keyword is used as a user behavior determination result in the room.
[0031] The present application provides a digital twin light adjustment device, comprising:
[0032] An acquisition unit, used to acquire the current scene image, current scene voice and current light intensity in the room;
[0033] A recognition unit, configured to determine a user behavior recognition result in the room based on the current scene image, and to determine a scene voice recognition result in the room based on the current scene voice;
[0034] A determination unit, configured to determine the user behavior recognition result based on the scene speech recognition result, and determine the user behavior determination result in the room;
[0035] An adjustment unit is used to determine an ideal light intensity corresponding to the user behavior identification result, and control a light adjustment device in the room to adjust the light intensity in the room based on the ideal light intensity and the current light intensity.
[0036] The present application provides a digital twin light adjustment system, comprising an image sensor, a pickup, a light sensor, a light adjustment device, and the digital twin light adjustment device;
[0037] The digital twin light adjustment device is connected to the image sensor, the pickup, the light sensor, and the light adjustment device based on a communication module;
[0038] The communication module includes at least one of a Bluetooth module, a WIFI module, a 4G module and a 5G module.
[0039] The present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the digital twin light adjustment method when executed by a processor.
[0040] The present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the digital twin light adjustment method is implemented when the processor executes the program.
[0041] The digital twin light adjustment method, device and system provided in the present application determine the user behavior recognition result indoors according to the current scene image, and determine the scene voice recognition result indoors according to the current scene voice; judge the user behavior recognition result according to the scene voice recognition result, and determine the user behavior judgment result indoors; determine the ideal light intensity corresponding to the user behavior judgment result according to the user behavior judgment result indoors, and control the light adjustment device indoors to adjust the light intensity indoors according to the ideal light intensity and the current light intensity. Since the user behavior is obtained through scene image recognition and determined after being judged in combination with the scene voice, it has a high degree of credibility and can accurately reflect the actual needs of the user in the current scene. At the same time, the indoor light intensity is automatically adaptively adjusted according to the user behavior without the need for manual user participation, thereby improving the intelligence level of light adjustment and the comfort of users living or working. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0043] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0044] Figure 1 A schematic diagram of the process of the digital twin light adjustment method provided in this application;
[0045] Figure 2 A schematic diagram of the structure of the digital twin light adjustment device provided in this application;
[0046] Figure 3 A schematic diagram of the structure of the digital twin light adjustment system provided in this application;
[0047] Figure 4 A control logic diagram of the digital twin light adjustment system provided in this application;
[0048] Figure 5 A schematic diagram of the hardware environment of the digital twin light adjustment method provided in this application;
[0049] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application.
[0050] Reference numerals:
[0051] 501: terminal device; 502: server. DETAILED DESCRIPTION
[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0053] It should be noted that the terms "first", "second" etc. in the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0054] Figure 1 A flowchart of the digital twin light adjustment method provided in this application is shown in FIG. Figure 1 As shown, the method includes step 110, step 120, step 130 and step 140.
[0055] Step 110: Acquire the current scene image, current scene voice and current light intensity in the room.
[0056] Specifically, the digital twin light adjustment method provided in the embodiment of the present application is applicable to light adjustment in a building interior. The building interior here can be the interior referred to in a home, including a living room, a bedroom, a study, etc.; it can also be the interior referred to in an office or business premises, including a conference room, an office, an activity room, etc. The executor of the digital twin light adjustment method is a digital twin light adjustment device. The digital twin light adjustment device can be a separately set control device, or it can be integrated with other smart devices in the room, such as smart speakers, smart air conditioners, smart refrigerators, smart TVs, and smart lamps.
[0057] The current scene image is the image of the indoor environment at the current moment. The current scene image records the various behaviors of each user in the room at the current moment, including sleeping, reading, watching TV, exercising, etc. The current scene image can be obtained by an image sensor set up indoors. The image sensor can be set up indoors alone or in a smart device. If the output of the image sensor is a video stream, each frame in the video stream can be used as a scene image.
[0058] The current scene voice is the ambient voice in the room at the current moment. The current scene voice records the sounds made by each user and each smart device at the current moment, including the voices of each user talking and reading in the room, the program sounds made by the smart TV or smart speaker, etc. The current scene voice can be obtained through the microphone set up in the room. The microphone can be set up separately in the room or in the smart device.
[0059] The current light intensity is the light intensity in the room at the current moment. The current light intensity reflects the luminous flux of visible light received by objects in the room. The current light intensity is closely related to the user's comfort level. For example, when a user watches TV indoors, a lower light intensity is usually required; when a user reads a book indoors, a higher light intensity is usually required.
[0060] Step 120: Determine the indoor user behavior recognition result based on the current scene image, and determine the indoor scene voice recognition result based on the current scene voice.
[0061] Specifically, the user behavior recognition result is a recognition result obtained by recognizing the user's indoor behavior according to the scene image. For example, the user behavior recognition result may be exercise, reading, watching TV, sleeping, etc.
[0062] The scene voice recognition result is a recognition result obtained by recognizing the sound generated by the user's behavior in the room according to the scene voice. For example, the scene voice recognition result can be semantically meaningless background music, semantic text corresponding to the user's voice, etc.
[0063] For the behavior recognition of the current scene image or the speech recognition of the current scene speech, an artificial intelligence algorithm can be used. For example, for image behavior recognition, an iDT (improved Dense Trajectories) algorithm can be used; for speech recognition, a DTW (Dynamic Time Warping) algorithm can be used.
[0064] Step 130: The user behavior recognition result is judged based on the scene voice recognition result to determine the indoor user behavior judgment result.
[0065] Specifically, the accuracy of the user behavior recognition result obtained by the image recognition method alone is low, and it is necessary to combine the scene voice recognition result to obtain a more accurate user behavior recognition result. The user behavior recognition result is the user behavior result after the scene voice is recognized.
[0066] For example, when a user is lying on the sofa in the living room watching TV, if image recognition is performed on the current scene image, the user behavior recognition result obtained may be watching TV or sleeping. If voice recognition is performed on the current scene voice, the scene voice recognition result obtained may be news reports, advertisements, TV dramas, etc. The user behavior recognition result is judged based on the scene voice recognition result, and it is determined that the indoor user behavior judgment result is watching TV.
[0067] The discrimination method may be text similarity comparison. For example, the user behavior recognition result and the scene voice recognition result are compared for text similarity. If the text similarity is greater than or equal to a preset threshold, it indicates that the user behavior recognition result and the scene voice recognition result are consistent, and the user behavior recognition result can be determined as the user behavior discrimination result; if the text similarity is less than the preset threshold, it indicates that the user behavior recognition result and the scene voice recognition result are inconsistent. At this time, the user behavior recognition result cannot be determined as the user behavior discrimination result, and it is necessary to combine other information to determine the user behavior discrimination result, such as using voice query to make the discrimination.
[0068] Step 140: determine the ideal light intensity corresponding to the user behavior identification result, and control the indoor light adjustment device to adjust the indoor light intensity based on the ideal light intensity and the current light intensity.
[0069] Specifically, a light intensity adjustment table may be pre-set. The light intensity adjustment table records the correspondence between user behavior and ideal light intensity. After determining the indoor user behavior discrimination result, the user behavior discrimination result is queried in the light intensity adjustment table, and the ideal light intensity corresponding to the matching user behavior is used as the ideal light intensity corresponding to the user behavior discrimination result.
[0070] The ideal light intensity is compared with the current light intensity to determine the light intensity adjustment amount. According to the light intensity adjustment amount, the control amount of the indoor light adjustment device is generated, so as to control the light adjustment device and adjust the indoor light intensity.
[0071] Light adjustment equipment may include shading equipment and lighting equipment. Shading equipment includes curtains and blinds. The opening and closing degree of the curtains can be controlled by controlling the moving position of the curtain traction mechanism on the curtain rail, thereby increasing or decreasing the luminous flux of natural light entering the room, thereby increasing or decreasing the indoor light intensity. Lighting equipment includes various types of lamps. The lighting intensity of the lamps can be controlled by increasing or decreasing the power of the lamps, thereby increasing or decreasing the indoor light intensity.
[0072] The digital twin light adjustment method provided in the embodiment of the present application determines the user behavior recognition result indoors according to the current scene image, and determines the scene voice recognition result indoors according to the current scene voice; discriminates the user behavior recognition result according to the scene voice recognition result, and determines the user behavior discrimination result indoors; determines the ideal light intensity corresponding to the user behavior discrimination result according to the user behavior discrimination result indoors, and controls the light adjustment device indoors to adjust the light intensity indoors according to the ideal light intensity and the current light intensity. Since the user behavior is obtained through scene image recognition and determined after being discriminated in combination with the scene voice, it has a high degree of credibility and can accurately reflect the actual needs of the user in the current scene. At the same time, it automatically makes adaptive adjustments to the light intensity indoors according to the user behavior without the need for manual user participation, thereby improving the intelligence level of light adjustment and the comfort of users living or working.
[0073] Based on the above embodiment, step 130 includes:
[0074] Perform word segmentation and keyword extraction on the scene speech recognition results to obtain keywords corresponding to the scene speech recognition results;
[0075] Compare the keyword with each preset keyword in a preset keyword library to determine a user behavior prediction result corresponding to the keyword; the preset keyword library includes multiple preset keywords and a user behavior prediction result corresponding to each preset keyword;
[0076] If the user behavior prediction result corresponding to the keyword is consistent with the user behavior recognition result, the user behavior recognition result is determined as the user behavior discrimination result.
[0077] Specifically, the scene speech recognition result may be a continuous string of a certain length. In this string, there may be many meaningless words, which are meaningless for the determination of the user behavior recognition result. Therefore, the scene recognition result can be processed by text, and the text processing method includes word segmentation, keyword extraction, and noise word removal.
[0078] The purpose of word segmentation for scene speech recognition results is to cut a long text into words with independent meanings and minimize the multiple meanings of a word to facilitate computer data information mining. Word segmentation methods can include forward maximum matching method, reverse maximum matching method, word segmentation algorithm based on N-gram language model, word segmentation algorithm based on sequence annotation, etc.
[0079] The purpose of extracting keywords from the scene speech recognition results is to extract the subject words with rich semantics from the word segmentation results. Keyword extraction methods can include TF-IDF (Term Frequency-Inverse Document Frequency), TextRank algorithm, etc.
[0080] Before keyword extraction, the segmentation results can also be filtered for noise words. For example, some meaningless modal particles can be considered as noise words and can be removed by text matching.
[0081] After obtaining the keyword corresponding to the scene speech recognition result, the keyword can be compared with each preset keyword in the preset keyword library. The preset keyword library includes multiple preset keywords and the user behavior prediction result corresponding to each preset keyword. For example, the preset keyword library may include "live report" or "news from this station", and the corresponding user behavior prediction result is watching TV.
[0082] Here, the preset keyword library can also be established according to the needs of the user. For example, the user can set up custom keywords according to their actual needs, and associate them with the user behavior prediction results and store them in the preset keyword library. The user can adjust the indoor light by speaking these custom keywords when speaking indoors. For example, the user can set a custom keyword "activity" and store it in association with the user behavior prediction result "exercise". When the digital twin light adjustment device detects that the scene voice recognition result contains "activity", it can be determined that the user behavior prediction result obtained according to the scene voice recognition result is exercise. Then, based on the user behavior prediction result, the user behavior recognition result obtained from the scene image is judged.
[0083] When the keyword corresponding to the scene speech recognition result matches the preset keyword, the user behavior prediction result corresponding to the preset keyword can be used as the user behavior prediction result corresponding to the keyword. If the user behavior prediction result corresponding to the keyword is consistent with the user behavior recognition result, the user behavior recognition result is determined as the user behavior discrimination result.
[0084] The digital twin light adjustment method provided in the present application determines the user behavior prediction results by performing word segmentation and keyword extraction on the scene speech recognition results, which is used to judge the user behavior recognition results, thereby improving the accuracy of user behavior judgment and being able to accurately reflect the actual needs of the user in the current scene.
[0085] Based on any of the above embodiments, step 140 includes:
[0086] Identify the user identity in the current scene image to determine the user identity of the indoor user;
[0087] Inputting the user identity and user behavior discrimination result of the indoor user into the light intensity prediction model, and obtaining the ideal light intensity corresponding to the user behavior discrimination result output by the light intensity prediction model;
[0088] The light intensity prediction model is trained based on sample users, sample user behaviors of sample users, and sample light intensities corresponding to the sample user behaviors.
[0089] Specifically, considering that each user has different requirements for light intensity, a light intensity prediction model can be established to provide personalized light adjustment services for each user. For example, when a user with eye disease reads indoors, he may need a higher light intensity than an ordinary person; or, the elderly in the family prefer a higher light intensity, while the same light intensity is uncomfortable for the young people in the family.
[0090] Therefore, the user identity in the current scene image can be identified to determine the user identity of the indoor user. The user identity can be defined by the user to identify the user's role in the home scene or office scene. For example, in the home scene, the user can define the user identity as {child, young person, elderly}, or define the user identity as {son, daughter, father, mother}.
[0091] User identification can use artificial intelligence algorithms, for example, using a convolutional neural network model as an initial model, taking family members' photos and family members' identity labels corresponding to the photos as samples, training the initial model, and obtaining a family members' identity identification model. After the current scene image is input into the family members' identity identification model, the user identity of the indoor user can be obtained.
[0092] The ideal light intensity corresponding to the user behavior judgment result can also be realized by using artificial intelligence algorithm. The following is an explanation using family members as users.
[0093] First, the light intensity preferences of each family member under different behaviors can be collected in advance and organized into a sample set. Specifically, each family member is used as a sample user, and their identity is labeled to obtain the sample user identity. The possible behaviors of each family member indoors are recorded, and these behaviors are used as sample user behaviors. The best light intensity experienced by each family member under different sample user behaviors is recorded as the sample light intensity.
[0094] Secondly, the neural network model is used as the initial model and the sample set obtained above is used for training to improve the initial model's prediction ability for the ideal light intensity and obtain a light intensity prediction model.
[0095] Finally, the user identity and user behavior discrimination results of the indoor users obtained through the current scene image and the current scene voice are input into the light intensity prediction model, and the ideal light intensity corresponding to the user behavior discrimination results output by the light intensity prediction model can be obtained.
[0096] The digital twin light adjustment method provided in this application adopts the establishment of a light intensity prediction model to provide personalized light adjustment services for each user, thereby improving the user's living or working comfort.
[0097] Based on any of the above embodiments, step 140 includes:
[0098] Get outdoor light intensity;
[0099] If the outdoor light intensity is greater than or equal to a preset light intensity threshold, then based on the difference between the ideal light intensity and the current light intensity, determine the opening and closing degree adjustment amount of the indoor shading device, and based on the opening and closing degree adjustment amount, control the shading device to adjust the indoor light intensity;
[0100] If the outdoor light intensity is less than the preset light intensity threshold, the power adjustment amount of the indoor lighting device is determined based on the difference between the ideal light intensity and the current light intensity, and the lighting device is controlled to adjust the indoor light intensity based on the power adjustment amount.
[0101] Specifically, there are two ways to adjust indoor light: one is to control the shading device, and the other is to control the lighting device. The former can use natural light to complete the adjustment, while the latter needs to consume electricity to complete the adjustment, which brings an economic burden to the user, and the adjustment effect is not as good as natural light in improving the user experience.
[0102] You can collect outdoor light intensity. Set a preset light intensity threshold to measure the intensity of outdoor natural light. The preset light intensity threshold can be set as needed.
[0103] If the outdoor light intensity is greater than or equal to the preset light intensity threshold, it indicates that there is sufficient natural light outdoors, such as sunny weather, and can be used to adjust the indoor light. The opening and closing degree adjustment amount of the shading device (such as smart curtains) can be determined according to the difference between the ideal light intensity and the current light intensity, and the shading device can be controlled to adjust the indoor light intensity. For example, the greater the difference between the ideal light intensity and the current light intensity, the greater the opening and closing degree adjustment amount of the shading device.
[0104] If the outdoor light intensity is less than the preset light intensity threshold, it indicates that the outdoor natural light is insufficient, such as cloudy days, and cannot be used to adjust the indoor light. The power adjustment amount of the indoor lighting device can be determined according to the difference between the ideal light intensity and the current light intensity, and the lighting device can be controlled to adjust the indoor light intensity. For example, the greater the difference between the ideal light intensity and the current light intensity, the greater the power adjustment amount of the lighting device.
[0105] The digital twin light adjustment method provided in this application reduces power consumption, reduces the financial burden on users, and the adjustment effect can further improve user experience.
[0106] Based on any of the above embodiments, step 140 further includes:
[0107] Based on the current scene image, determine the user's indoor activity area;
[0108] Determine the light adjustment device corresponding to the activity area based on the activity area and the correspondence between each indoor light adjustment device and each activity area;
[0109] Based on the ideal light intensity and the current light intensity, a light adjustment device corresponding to the active area is controlled to adjust the light intensity of the active area.
[0110] Specifically, for a living room, when its area is large, there may be multiple sets of ceiling lights and wall lights, etc., each lamp is used to illuminate a part of the living room. For a large conference room, multiple sets of lamps may be set, each lamp is used to illuminate a part of the conference room.
[0111] The room can be divided into multiple activity areas. The size of the activity area can be set as needed. Then the light adjustment devices in the room are grouped to determine the corresponding relationship between each light adjustment device in the room and each activity area.
[0112] According to the current scene image, the user's position in the scene image can be determined, and the user's activity area in the room can be obtained by corresponding to the room. According to the correspondence between each light adjustment device in the room and each activity area, the light adjustment device corresponding to the activity area where the user is located can be determined.
[0113] Then, according to the ideal illumination intensity and the current illumination intensity, the light adjustment device corresponding to the active area is controlled to adjust the illumination intensity of the active area.
[0114] The digital twin light adjustment method provided in the present application enables users to illuminate only the activity area where they are located without illuminating the entire room when performing activities indoors, thereby achieving the purpose of energy saving and consumption reduction.
[0115] Based on any of the above embodiments, step 130 includes:
[0116] If the scene speech recognition result is empty, a prompt voice containing a semantically filled slot is sent;
[0117] Receiving voice information input by the user based on the prompt voice;
[0118] Based on the transcribed text of the voice information, the behavior keywords corresponding to the semantic filling slots in the transcribed text are determined, and the behavior keywords are used as the indoor user behavior discrimination results.
[0119] Specifically, when the current scene voice recognition fails, the scene voice recognition result is empty, and the user behavior recognition result cannot be determined as the user behavior discrimination result. Voice query can also be used for discrimination.
[0120] The semantically filled slot is a clearly defined attribute of the entity, which can be used as information to be completed in the conversation. The digital twin light adjustment device can send a prompt voice containing the semantically filled slot to the user. For example, the digital twin light adjustment device can send "What are you currently doing", where the semantically filled slot is "What to do".
[0121] The digital twin light adjustment device receives the voice information input by the user according to the prompt voice, transcribes the voice information, and obtains the transcribed text. According to the content of the semantic filling slot, the transcribed text is detected to obtain the behavior keyword corresponding to the semantic filling slot, and the behavior keyword is used as the user behavior judgment result.
[0122] For example, if the user's voice information is "I am reading a book", it is converted into text, and the words in the text are matched with the semantic filling slots for similarity, and the behavior keyword "reading a book" corresponding to the semantic filling slot "what to do" is obtained. The digital twin light adjustment method provided in this application uses voice query to determine the user behavior judgment result, which makes up for the shortcomings of scene voice recognition and improves the intelligence level of light adjustment.
[0123] Based on any of the above embodiments, Figure 2 A schematic diagram of the structure of the digital twin light adjustment device provided in this application, such as Figure 2 As shown, the device comprises:
[0124] An acquisition unit 210 is used to acquire a current scene image, a current scene voice and a current light intensity in the room;
[0125] The recognition unit 220 is used to determine the indoor user behavior recognition result based on the current scene image, and determine the indoor scene voice recognition result based on the current scene voice;
[0126] A determination unit 230, configured to determine the user behavior recognition result based on the scene speech recognition result, and determine the indoor user behavior determination result;
[0127] The adjustment unit 240 is used to determine the ideal light intensity corresponding to the user behavior determination result, and control the light adjustment device in the room to adjust the light intensity in the room based on the ideal light intensity and the current light intensity.
[0128] The digital twin light adjustment device provided in the embodiment of the present application determines the user behavior recognition result indoors according to the current scene image, and determines the scene voice recognition result indoors according to the current scene voice; judges the user behavior recognition result according to the scene voice recognition result, and determines the user behavior judgment result indoors; determines the ideal light intensity corresponding to the user behavior judgment result according to the user behavior judgment result indoors, and controls the light adjustment device indoors to adjust the light intensity indoors according to the ideal light intensity and the current light intensity. Since the user behavior is obtained through scene image recognition and determined after being judged in combination with the scene voice, it has a high degree of credibility and can accurately reflect the actual needs of the user in the current scene. At the same time, it automatically makes adaptive adjustments to the light intensity indoors according to the user behavior without the need for manual user participation, thereby improving the intelligence level of light adjustment and the comfort of users living or working.
[0129] Based on any of the above embodiments, the determination unit is specifically used for:
[0130] Perform word segmentation and keyword extraction on the scene speech recognition results to obtain keywords corresponding to the scene speech recognition results;
[0131] Compare the keyword with each preset keyword in a preset keyword library to determine a user behavior prediction result corresponding to the keyword; the preset keyword library includes multiple preset keywords and a user behavior prediction result corresponding to each preset keyword;
[0132] If the user behavior prediction result corresponding to the keyword is consistent with the user behavior recognition result, the user behavior recognition result is determined as the user behavior discrimination result.
[0133] Based on any of the above embodiments, the adjustment unit is specifically used for:
[0134] Identify the user identity in the current scene image to determine the user identity of the indoor user;
[0135] Inputting the user identity and user behavior discrimination result of the indoor user into the light intensity prediction model, and obtaining the ideal light intensity corresponding to the user behavior discrimination result output by the light intensity prediction model;
[0136] The light intensity prediction model is obtained by training based on sample user identities, sample user behaviors of the sample user identities, and sample light intensities corresponding to the sample user behaviors.
[0137] Based on any of the above embodiments, the adjustment unit is further specifically configured to:
[0138] Get outdoor light intensity;
[0139] If the outdoor light intensity is greater than or equal to a preset light intensity threshold, then based on the difference between the ideal light intensity and the current light intensity, determine the opening and closing degree adjustment amount of the indoor shading device, and based on the opening and closing degree adjustment amount, control the shading device to adjust the indoor light intensity;
[0140] If the outdoor light intensity is less than the preset light intensity threshold, the power adjustment amount of the indoor lighting device is determined based on the difference between the ideal light intensity and the current light intensity, and the lighting device is controlled to adjust the indoor light intensity based on the power adjustment amount.
[0141] Based on any of the above embodiments, the adjustment unit is further specifically configured to:
[0142] Based on the current scene image, determine the user's indoor activity area;
[0143] Determine the light adjustment device corresponding to the activity area based on the activity area and the correspondence between each indoor light adjustment device and each activity area;
[0144] Based on the ideal light intensity and the current light intensity, a light adjustment device corresponding to the active area is controlled to adjust the light intensity of the active area.
[0145] Based on any of the above embodiments, the device further includes:
[0146] A voice query unit, used to send a prompt voice containing a semantic filling slot if the scene voice recognition result is empty;
[0147] Receiving voice information input by the user based on the prompt voice;
[0148] Based on the transcribed text of the voice information, the behavior keywords corresponding to the semantic filling slots in the transcribed text are determined, and the behavior keywords are used as the indoor user behavior discrimination results.
[0149] Based on any of the above embodiments, Figure 3 A schematic diagram of the structure of the digital twin light adjustment system provided in this application, such as Figure 3 As shown, the system includes an image sensor 310, a pickup 320, a light sensor 330, a light adjustment device 340, and a digital twin light adjustment device 350;
[0150] The digital twin light adjustment device 350 is connected to the image sensor 310, the microphone 320, the light sensor 330, and the light adjustment device 340 based on the communication module 360; the communication module 360 includes at least one of a Bluetooth module, a WIFI module, a 4G module, and a 5G module.
[0151] Specifically, Figure 4The control logic diagram of the digital twin light adjustment system provided in this application is as follows: Figure 4 As shown in the figure, the control logic of the digital twin light adjustment system is as follows:
[0152] Step 410: Acquire the current environment video stream data through the image sensor, acquire the current environment light intensity through the light sensor, and collect the current environment user audio information through the microphone.
[0153] Step 420: Based on the image acquisition information, the home scene of the user is identified and classified using deep learning technology.
[0154] Step 430: Combine the multi-frame video stream information and preliminarily determine the user's current behavior (watching TV in the living room, reading under a desk lamp, sleeping in the bedroom, etc.) through the user behavior recognition model.
[0155] Step 440: Combine the voice recognition information to assist in identifying the user's behavior. For example, when watching TV in the living room, there will be sounds of advertisements, TV series, etc.; when reading under a desk lamp, there will be reading sounds or no sound; when sleeping in the bedroom, there will be basically no sound.
[0156] Step 450: Establish an optimal light intensity model for user behavior based on digital twin technology, and actively determine whether the light needs to be adjusted to meet user behavior needs based on the light intensity information returned by the light sensor.
[0157] Step 460: If the light needs to be adjusted, the light adjustment equipment (desk lamp, curtain, chandelier, light strip, etc.) is linked to actively adjust the ambient light to provide the user with the best experience.
[0158] Step 470: If there is no need to adjust the light, return to step 410 and continue detecting.
[0159] The digital twin light adjustment system provided in the embodiment of the present application can automatically and adaptively adjust the indoor light intensity according to user behavior without the need for manual user participation, thereby improving the intelligence level of light adjustment and improving the user's living or working comfort.
[0160] Based on any of the above embodiments, the present application also provides a digital twin light adjustment method. The digital twin light adjustment method is widely used in smart home (Smart Home), smart home, smart home device ecology, smart residential (Intelligence House) ecology and other whole-house intelligent digital control application scenarios. In this embodiment, Figure 5 Schematic diagram of the hardware environment of the digital twin light adjustment method provided in this application. The digital twin light adjustment method can be applied to Figure 5In the hardware environment shown, which is composed of a terminal device 501 and a server 502, the server 502 is connected to the terminal device 501 through a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 502. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 502.
[0161] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 501 may be but is not limited to a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing equipment, a smart dishwasher, a smart projection equipment, a smart TV, a smart clothes drying rack, a smart curtain, a smart audio and video, a smart socket, a smart speaker, a smart speaker, a smart fresh air equipment, a smart kitchen and bathroom equipment, a smart bathroom equipment, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification equipment, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.
[0162] Based on any of the above embodiments, based on any of the above embodiments, Figure 6 A schematic diagram of the structure of the electronic device provided in this application, such as Figure 6 As shown, the electronic device may include: a processor (Processor) 610, a communication interface (Communications Interface) 620, a memory (Memory) 630 and a communication bus (Communications Bus) 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic command in the memory 630 to execute the following method:
[0163] Acquire the current scene image, current scene voice and current light intensity of the room; determine the user behavior recognition result of the room based on the current scene image, and determine the scene voice recognition result of the room based on the current scene voice; judge the user behavior recognition result based on the scene voice recognition result, and determine the user behavior judgment result of the room; determine the ideal light intensity corresponding to the user behavior judgment result, and control the light adjustment device in the room to adjust the light intensity in the room based on the ideal light intensity and the current light intensity.
[0164] In addition, the logic commands in the above-mentioned memory 630 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several commands to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.
[0165] The processor in the electronic device provided in the embodiment of the present application can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.
[0166] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the methods provided in the above embodiments are implemented.
[0167] Its specific implementation is consistent with the aforementioned method implementation and can achieve the same beneficial effects, so it will not be repeated here.
[0168] An embodiment of the present application provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0169] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0170] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A digital twin light adjustment method, It is characterized in that include: Obtain the current scene image, current scene voice and current light intensity in the room; The current scene voice is used to record the sounds made by each user and each smart device in the room at the current moment; Determine a user behavior recognition result in the room based on the current scene image, and determine a scene voice recognition result in the room based on the current scene voice; Determining the user behavior recognition result based on the scene voice recognition result to determine the user behavior recognition result in the room; Determine an ideal light intensity corresponding to the user behavior identification result, and control a light adjustment device in the room to adjust the light intensity in the room based on the ideal light intensity and the current light intensity; The step of judging the user behavior recognition result based on the scene speech recognition result to determine the indoor user behavior judgment result includes: Performing word segmentation and keyword extraction on the scene speech recognition result to obtain keywords corresponding to the scene speech recognition result; Compare the keyword with each preset keyword in a preset keyword library to determine a user behavior prediction result corresponding to the keyword; the preset keyword library includes multiple preset keywords and a user behavior prediction result corresponding to each preset keyword; If the user behavior prediction result corresponding to the keyword is consistent with the user behavior recognition result, the user behavior recognition result is determined as the user behavior discrimination result.
2. The digital twin light adjustment method according to claim 1, It is characterized in that The determining the ideal illumination intensity corresponding to the user behavior identification result includes: Identify the user identity in the current scene image to determine the user identity of the indoor user; Inputting the user identity and user behavior discrimination result of the indoor user into a light intensity prediction model to obtain an ideal light intensity corresponding to the user behavior discrimination result output by the light intensity prediction model; The illumination intensity prediction model is obtained by training based on sample user identities, sample user behaviors of the sample user identities, and sample illumination intensities corresponding to the sample user behaviors.
3. The digital twin light adjustment method according to claim 1, It is characterized in that The step of controlling the light adjustment device in the room to adjust the light intensity in the room based on the ideal light intensity and the current light intensity includes: Get outdoor light intensity; If the outdoor light intensity is greater than or equal to a preset light intensity threshold, then based on the difference between the ideal light intensity and the current light intensity, determine the opening and closing degree adjustment amount of the indoor shading device, and based on the opening and closing degree adjustment amount, control the shading device to adjust the indoor light intensity; If the outdoor light intensity is less than the preset light intensity threshold, the power adjustment amount of the indoor lighting device is determined based on the difference between the ideal light intensity and the current light intensity, and based on the power adjustment amount, the lighting device is controlled to adjust the indoor light intensity.
4. The digital twin light adjustment method according to claim 1, It is characterized in that The step of controlling the light adjustment device in the room to adjust the light intensity in the room based on the ideal light intensity and the current light intensity includes: Based on the current scene image, determining an activity area of the user in the room; Determine the light adjustment device corresponding to the activity area based on the activity area and the correspondence between each light adjustment device in the room and each activity area; Based on the ideal illumination intensity and the current illumination intensity, a light adjustment device corresponding to the active area is controlled to adjust the illumination intensity of the active area.
5. The digital twin light adjustment method according to claim 1, It is characterized in that The step of judging the user behavior recognition result based on the scene speech recognition result to determine the indoor user behavior judgment result includes: If the scene speech recognition result is empty, sending a prompt speech containing a semantically filled slot; Receiving voice information input by the user based on the prompt voice; Based on the transcribed text of the voice information, a behavior keyword corresponding to the semantic filling slot in the transcribed text is determined, and the behavior keyword is used as a user behavior determination result in the room.
6. A digital twin light adjustment device, It is characterized in that include: An acquisition unit, used to acquire the current scene image, current scene voice and current light intensity in the room; The current scene voice is used to record the sounds made by each user and each smart device in the room at the current moment; A recognition unit, configured to determine a user behavior recognition result in the room based on the current scene image, and to determine a scene voice recognition result in the room based on the current scene voice; A determination unit, configured to determine the user behavior recognition result based on the scene speech recognition result, and determine the user behavior determination result in the room; an adjustment unit, configured to determine an ideal light intensity corresponding to the user behavior identification result, and control a light adjustment device in the room to adjust the light intensity in the room based on the ideal light intensity and the current light intensity; The discrimination unit is specifically used for: Performing word segmentation and keyword extraction on the scene speech recognition result to obtain keywords corresponding to the scene speech recognition result; Compare the keyword with each preset keyword in a preset keyword library to determine a user behavior prediction result corresponding to the keyword; the preset keyword library includes multiple preset keywords and a user behavior prediction result corresponding to each preset keyword; If the user behavior prediction result corresponding to the keyword is consistent with the user behavior recognition result, the user behavior recognition result is determined as the user behavior discrimination result.
7. A digital twin light adjustment system, It is characterized in that It includes an image sensor, a pickup, a light sensor, a light adjustment device, and a digital twin light adjustment device as claimed in claim 6; The digital twin light adjustment device is connected to the image sensor, the pickup, the light sensor, and the light adjustment device based on a communication module; The communication module includes at least one of a Bluetooth module, a WIFI module, a 4G module and a 5G module.
8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the digital twin light adjustment method according to any one of claims 1 to 5 is implemented.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the digital twin light adjustment method as described in any one of claims 1 to 5 is implemented.
