Digital twin multimodal device control methods, storage media and electronic devices

Abstract

The present application discloses a digital twin multimodal device control method, storage medium and electronic device, which relates to the field of smart home technology, wherein the above method includes: based on the target object image and / or object voice data, the target object is identified in the emotional state to obtain the current emotional state of the target object; when the current emotional state is a negative state, the historical emotional state of the target object in the historical time period is obtained; according to the current emotional state and the historical emotional state, the target device operation to be executed by the target smart device is determined; and the target smart device is controlled to execute the target device operation. The above technical solution solves the problem of poor sustainability of device operation caused by the inability to obtain user feedback information in a timely manner in the method of controlling the device to execute device operation based on the identified emotion category in the related art.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and more specifically, to a digital twin multimodal device control method, storage medium, and electronic device. Background Art

[0002] Currently, smart devices can recognize emotions based on user speech or text, and perform corresponding device operations based on the recognized emotions. However, after performing the device operation, if the user no longer speaks or inputs text, the smart device cannot obtain the effect of the device operation, that is, it cannot know the effect of the device operation on the user's mood.

[0003] For example, if the user is identified as feeling down or sad, the system can soothe them through voice interaction or playing songs. However, this soothing can only temporarily alleviate the user's negative emotions. If the user stops making sounds, it will be impossible to detect whether the user's emotions have returned to normal.

[0004] It can be seen that the method of controlling the device to perform device operations based on the recognized emotion category in the related art has the problem of poor sustainability of device operations due to the inability to obtain user feedback information in a timely manner. Summary of the Invention

[0005] The purpose of this application is to provide a digital twin multimodal device control method, storage medium and electronic device, so as to at least solve the problem of poor sustainability of device operation caused by the inability to obtain user feedback information in a timely manner in the method of controlling the device to perform device operations based on the identified emotion category in the related art.

[0006] According to one aspect of an embodiment of the present application, a digital twin multimodal device control method is provided, including: based on the target object image and / or object voice data, performing emotional state recognition on the target object to obtain the current emotional state of the target object; when the current emotional state is a negative state, obtaining the historical emotional state of the target object within a historical time period; determining the target device operation to be executed by the target smart device based on the current emotional state and the historical emotional state; and controlling the target smart device to execute the target device operation.

[0007] According to another aspect of the embodiment of the present application, a digital twin multimodal device control device is also provided, including: a first recognition unit, used to identify the emotional state of a target object based on the target object image and / or object voice data, and obtain the current emotional state of the target object; a first acquisition unit, used to obtain the historical emotional state of the target object within a historical time period when the current emotional state is a negative state; a determination unit, used to determine the target device operation to be executed by the target smart device based on the current emotional state and the historical emotional state; and a control unit, used to control the target smart device to perform the target device operation.

[0008] In an exemplary embodiment, the device also includes: a second acquisition unit, used to acquire the target object image captured by the image acquisition component before the emotional state of the target object is recognized based on the target object image and / or object voice data to obtain the current emotional state of the target object, wherein the target object image is an image of the object face containing the target object; and a third acquisition unit, used to acquire the object voice data captured by the voice acquisition component, wherein the object voice data is the voice data emitted by the target object.

[0009] In an exemplary embodiment, the device further includes: a second recognition unit, configured to perform facial region recognition on the target object image after the target object image is acquired by the image acquisition component, to obtain a group of facial images; and a third recognition unit, configured to perform object recognition on each facial image in the group of facial images, to obtain object information of a group of objects, wherein the group of objects includes the target object.

[0010] In an exemplary embodiment, the device further includes: an adjustment unit for adjusting the image acquisition component from a closed state to an open state when it is detected that the distance between the target object and the target smart device is less than or equal to a preset distance threshold before the emotional state of the target object is recognized based on the target object image and / or object voice data to obtain the current emotional state of the target object; and an acquisition unit for acquiring an image of the target object through the image acquisition component to obtain the target object image when the object part of the target object located in the image to be acquired by the image acquisition component meets a preset condition.

[0011] In an exemplary embodiment, the acquisition unit includes: a first acquisition module, which is used to acquire an image of the target object through the image acquisition component when the object part of the target object located in the image to be acquired includes the object's face, and the ratio between the area of ​​the object's face and the area of ​​the image to be acquired is greater than or equal to a target ratio, so as to obtain the target object image; or, a second acquisition module, which is used to acquire an image of the target object through the image acquisition component when the object part of the target object located in the image to be acquired includes the object's face and the object's hand, so as to obtain the target object image.

[0012] In an exemplary embodiment, the first recognition unit includes: a first recognition module, used to perform emotion category recognition on the facial image of the object to obtain the current emotion category of the target object, wherein the current emotional state includes the current emotion category; a second recognition module, used to perform part action recognition on the part image of the preset part to obtain the target part action when the target object image also includes the part image of the preset part of the target object; a first determination module, used to determine the emotion intensity matching the target part action as the current emotion intensity of the target object, wherein the current emotional state also includes the current emotion category.

[0013] In an exemplary embodiment, the preset parts include the object's hands and the object's torso, and the target part movements include target hand movements and target body movements; the first determination module includes: a first determination submodule, used to determine the weighted sum of the emotion intensity matching the target hand movement and the emotion intensity matching the target body movement as the current emotion intensity of the target object.

[0014] In an exemplary embodiment, the first recognition unit includes: a third recognition module, which is used to perform emotional state recognition on the target object based on the target object image and the object voice data, respectively, to obtain multiple emotional states; and a fusion module, which is used to perform emotional state fusion on the multiple emotional states to obtain the current emotional state of the target object.

[0015] In an exemplary embodiment, the third recognition module includes: a first recognition submodule, used to perform emotional state recognition on the object facial image of the target object to obtain a first emotional state, wherein the first emotional state includes a first emotional category, and the target object image includes the object facial image; a second recognition submodule, used to perform emotional state recognition on the object voice data to obtain a second emotional state, wherein the second emotional state includes a second emotional category.

[0016] In an exemplary embodiment, the fusion module includes: a second determination submodule, which is used to determine any one of the first emotion category and the second emotion category as the current emotion category when the first emotion category and the second emotion category are consistent; a third determination submodule, which is used to determine the emotion category with high confidence in the first emotion category and the second emotion category as the current emotion category when the first emotion category and the second emotion category are inconsistent; or, to determine the emotion category in the first emotion category and the second emotion category that matches the part state of the preset part of the target object as the current emotion category.

[0017] In an exemplary embodiment, the current emotional state also includes the current emotional intensity of the target object; the fusion module also includes: a fourth determination submodule, used to determine the first emotional intensity in the second emotional state as the current emotional intensity when the first emotion category and the second emotion category are consistent; or, a fifth determination submodule, used to determine the weighted sum of the first emotional intensity and the second emotional intensity as the current emotional intensity when the target object image also includes a part image of a preset part of the target object, wherein the first emotional intensity is the emotional intensity contained in the second emotional state, and the second emotional intensity is the emotional intensity identified from the part image of the preset part.

[0018] In an exemplary embodiment, the current emotional state also includes the current emotional intensity of the target object; the fusion module further includes: a sixth determination submodule for determining the emotional intensity identified from the part image of the preset part as the current emotional intensity when the first emotion category and the second emotion category are inconsistent, the current emotion category is the first emotion category, and the target object image also includes a part image of a preset part of the target object; a seventh determination submodule for determining the first emotional intensity in the second emotional state as the current emotional intensity when the first emotion category and the second emotion category are inconsistent, and the current emotion category is the second emotion category; an eighth determination submodule for determining the weighted sum of the first emotional intensity and the second emotional intensity as the current emotional intensity when the first emotion category and the second emotion category are inconsistent, the current emotion category is the second emotion category, and the target object image also includes a part image of a preset part of the target object, wherein the first emotional intensity is the emotional intensity included in the second emotional state, and the second emotional intensity is the emotional intensity identified from the part image of the preset part.

[0019] In an exemplary embodiment, the determination unit also includes: a second determination module for determining that the device operation to be executed by the target smart device is a first device operation when it is determined that the target object is in a negative emotion according to the current emotional state and the historical emotional state, and the emotional intensity of the negative emotion is reduced; a third determination module for determining that the device operation to be executed by the target smart device is a second device operation when it is determined that the target object is in a negative emotion according to the current emotional state and the historical emotional state, and the emotional intensity of the negative emotion is increased; a fourth determination module for determining that the device operation to be executed by the target smart device is a third device operation when it is determined that the target object is in a negative emotion according to the current emotional state and the historical emotional state, and the emotional intensity of the negative emotion is greater than or equal to a preset intensity threshold and the duration reaches a preset duration threshold.

[0020] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned digital twin multimodal device control method when running.

[0021] According to another aspect of an embodiment of the present application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor executes the digital twin multimodal device control method through the computer program.

[0022] In an embodiment of the present application, a method is adopted in which the current emotional state of a specific object is identified, and when the current emotional state is a negative state, a corresponding device operation is performed on the object in combination with the historical emotional state of the object. The current emotional state of the target object is obtained by performing emotional state identification on the target object based on the target object image and / or object voice data; when the current emotional state is a negative state, the historical emotional state of the target object in a historical time period is obtained; based on the current emotional state and the historical emotional state, the target device operation to be performed by the target smart device is determined; and the target smart device is controlled to perform the target device operation. Since, when it is detected that the current emotional state of the specific object is a negative state, the device operation performed by the smart device is determined based on the current emotional state and the historical emotional state, and since the historical emotional state and the current emotional state can reflect the impact of the executed device operation on the user's emotional state, the purpose of timely obtaining user feedback information is achieved, and the technical effect of improving the sustainability of device operation is achieved, thereby solving the problem of poor sustainability of device operation caused by the inability to timely obtain user feedback information in the method of controlling the device to perform device operation based on the identified emotion category in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 This is a schematic diagram of the hardware environment of an optional digital twin multimodal device control method according to an embodiment of the present application;

[0026] Figure 2 This is a flow chart of an optional digital twin multimodal device control method according to an embodiment of the present application;

[0027] Figure 3 is a flow chart of another optional digital twin multimodal device control method according to an embodiment of the present application;

[0028] Figure 4 This is a structural block diagram of an optional digital twin multimodal device control device according to an embodiment of the present application;

[0029] Figure 5 This is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] According to one aspect of the embodiment of the present application, a digital twin multimodal device control method is provided. The digital twin multimodal device control method is widely used in smart home (Smart Home), smart home, smart home device ecology, smart residential (Intelligence House) ecology and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, the above-mentioned digital twin multimodal device control method can be applied to Figure 1 In the hardware environment shown in FIG. 1 , which is composed of a terminal device 102 and a server 104. Figure 1 As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or the client installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0033] The aforementioned network may include, but is not limited to, at least one of the following: a wired network and a wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: a wide area network, a metropolitan area network, and a local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity) and Bluetooth. The terminal device 102 may be, but is not limited to, a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing machine, a smart dishwasher, a smart projection device, a smart TV, a smart clothes drying rack, smart curtains, smart audio and video, a smart socket, a smart speaker, a smart fresh air device, smart kitchen and bathroom equipment, smart bathroom equipment, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purifier, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.

[0034] The digital twin multimodal device control method of the embodiment of the present application can be executed by the server 104, or by the terminal 102, or by both the server 104 and the terminal 102. The digital twin multimodal device control method of the embodiment of the present application can also be executed by the client installed on the terminal 102.

[0035] Taking the server 104 as an example to execute the digital twin multimodal device control method in this embodiment, Figure 2 This is a flow chart of an optional digital twin multimodal device control method according to an embodiment of the present application, such as Figure 2 As shown, the process of the method may include the following steps:

[0036] Step S202 : Based on the target object image and / or the target object voice data, the target object's emotional state is recognized to obtain the target object's current emotional state.

[0037] The digital twin multimodal device control method in this embodiment can be applied to a scenario where the target smart device is controlled to perform corresponding device operations based on the emotional state of the target object to adjust the emotional state of the target object. The above-mentioned target object can be the user of the target smart device (for example, a user), and the above-mentioned target smart device can be a smart home device, for example, a smart refrigerator, a smart speaker, etc. The executed device operations can include TTS (Text To Speech, from text to speech) voice playback operations, and can also include other device operations, which are not limited in this embodiment.

[0038] The above-mentioned target intelligent devices can be digital twin multimodal devices. Digital twin multimodal devices can be simulation intelligent devices that make full use of physical models, sensor updates, operation history and other data, and integrate multi-disciplinary, multi-physical quantities, multi-scales, and multi-probabilities. These simulation intelligent devices can be used to complete simulation mapping in virtual space, thereby reflecting the full life cycle process of the physical intelligent device corresponding to the simulation intelligent device.

[0039] The target smart device can collect object data of the target object. The target object's object data can include at least one of a target object image and object voice data. For example, the target object image of the target object can be facial image data, body image data, etc. of the target user, and the target object voice data can be voice data of the target user, etc. It can also include other object data, which is not limited in this embodiment. The target smart device can collect the target object's object data in one or more ways, which can include but is not limited to one of the following: collecting the target object's object data through a collection component on the target smart device, collecting the target object's object data through other devices associated with the target smart device, and collecting the target object's object data through other means. In addition, the target smart device can also perform data processing on the collected target object's object data to obtain the target object's object data, which is not limited in this embodiment.

[0040] The target smart device can upload the object data of the target object to the server, and the server can receive the object data of the target object and perform emotional state recognition based on the object data of the target object to obtain the current emotional state of the target object. The current emotional state of the target object can be one of a positive state, a neutral state, and a negative state. It can include the current emotion category of the target object, such as happy, neutral, sad, surprised, afraid, angry, disgusted, etc., and can also include the current emotional intensity, such as the intensity. The emotional intensity can be represented by an emotional level, for example, from level 0 to level 10, where level 0 represents the most intense emotion and level 10 represents the most intense emotion. Other representation methods are also possible.

[0041] The server performs emotional state recognition based on the object data of the target object. There may be one or more ways to obtain the current emotional state of the target object. The method may be to input the object data of the target object into the emotional state recognition model to obtain the current emotional state of the target object output by the emotional state recognition model. The method may also be to call the interface related to the emotion recognition algorithm through the corresponding API (Application Programming Interface) to obtain the current emotional state of the target object corresponding to the object data of the target object through the interface. The emotional state of the target object may also be recognized on the object data of the target object in other ways to obtain the current emotional state of the target object. This is not limited in this embodiment.

[0042] Step S204: when the current emotional state is a negative state, obtaining the historical emotional state of the target object within a historical time period.

[0043] If the target object's current emotional state is negative, the target smart device can be controlled to perform corresponding device operations to adjust the target object's negative state. Currently, most smart home devices perform one-time emotion recognition based on user voice or text and make corresponding soothing operations. For example, when it is recognized that the user is depressed or sad, voice or songs can be played to soothe the user. However, this soothing can only temporarily alleviate the user's negative emotions. If the user no longer makes a sound, it is impossible to perceive whether the user's emotions have truly returned to calm. If the user's mood is negative for a long time, it is easy to have a negative impact on the user's work and life.

[0044] In order to at least partially solve the above problems, in this embodiment, if the current emotional state of the target object is negative, the server can obtain the historical emotional state of the target object in the historical time period, detect the changes in the target object's historical emotions, and make corresponding feedback based on the target object's emotional changes, which can ensure that the feedback made can better soothe the target object's negative emotions.

[0045] The server may obtain the historical emotional state of the target object within a historical time period in one or more ways. It may obtain the historical emotional state of the target object within a historical time period from a database based on the object information of the target object (for example, name, etc.), or obtain the historical emotional state of the target object from a target smart device or other related smart devices. The historical emotional state of the target object may also be obtained in other ways, which is not limited in this embodiment.

[0046] Step S206 : determining a target device operation to be executed by the target smart device according to the current emotional state and the historical emotional states.

[0047] After obtaining the historical emotional state of the target subject over a historical time period, the server can determine a target device operation to be executed by the target smart device based on the current emotional state and the historical emotional state. The target device operation can be one or more, including but not limited to the following: playing soothing music, continuing to play soothing NLP speech, playing intensified soothing NLP speech, etc. Other device operations can also be performed, such as sending a prompt message to a preset device to prompt other devices to soothe the target subject's negative emotions, which is not limited in this embodiment.

[0048] Step S208: controlling the target smart device to execute the target device operation.

[0049] After determining the target device operation to be executed, the server can control the target smart device to execute the corresponding target device operation. Optionally, the server can send a device operation instruction to the target smart device to instruct the target smart device to execute the target device operation. After receiving the device operation instruction, the target smart device can execute the target device operation indicated by the device operation instruction.

[0050] Optionally, the target smart device can re-acquire the current emotional state of the target object, and combine the re-acquired current emotional state with the historical emotional state to re-determine the next target device operation to be executed by the target smart device, and after executing the target device operation, continue to execute the above-mentioned next target device operation.

[0051] It should be noted that the above steps S202 to S208 can also be executed by the target smart device or the target smart device in combination with the server. The above is only an exemplary description of execution by the server and does not limit the execution subject of the digital twin multimodal device control method in this embodiment.

[0052] Through the above steps S202 to S208, the emotional state of the target object is identified based on the target object image and / or object voice data to obtain the current emotional state of the target object; when the current emotional state is a negative state, the historical emotional state of the target object in the historical time period is obtained; based on the current emotional state and the historical emotional state, the target device operation to be executed by the target smart device is determined; the target smart device is controlled to execute the target device operation, which solves the problem of poor sustainability of device operation due to the inability to obtain user feedback information in a timely manner in the method of controlling the device to execute the device operation based on the identified emotion category in the related art, thereby improving the sustainability of device operation.

[0053] In an exemplary embodiment, before performing emotional state recognition on the target object based on the target object image and / or the target object voice data to obtain the current emotional state of the target object, the method further includes:

[0054] S11, acquiring a target object image captured by an image capturing component, wherein the target object image is an image including a face of the target object;

[0055] S12, acquiring the target voice data collected by the voice collecting component, wherein the target voice data is the voice data uttered by the target object.

[0056] The target object data may include at least one of an object image and an object voice of the target object. Optionally, in this embodiment, the target object data may include a target object image and object voice data, where the target object image is an image containing the target object's face, and the object voice data is voice data uttered by the target object.

[0057] The target smart device may be equipped with an image acquisition component and a voice acquisition component, which can respectively acquire an image of the target object and voice data of the target object. The image acquisition component may be a camera, etc., and the voice acquisition component may be a pickup, microphone, etc. The acquired image and voice data of the target object may be transmitted to a server, which can then obtain the target object data.

[0058] For example, the target smart device can be a smart device with a human body sensor, a camera, and a radio. It can also have a human body sensor that triggers the camera to start when a person approaches, and can also activate a microphone and other radio components (the radio component can be kept on at all times based on user authorization). The target smart device can be a smart speaker with a screen, a smart refrigerator with a screen, etc. The target smart device can obtain the user's facial image and voice data through the camera and radio components installed on it.

[0059] Through this embodiment, by acquiring the user's facial image and the voice data uttered by the user, the emotional state recognition is performed, which can improve the accuracy of the emotional state recognition and improve the sustainability of the device in performing device operations.

[0060] In an exemplary embodiment, after acquiring the target object image captured by the image acquisition component, the method further includes:

[0061] S21, performing facial region recognition on the target object image to obtain a set of facial images;

[0062] S22 , performing object recognition on each facial image in a group of facial images to obtain object information of a group of objects, wherein the group of objects includes a target object.

[0063] Emotion recognition methods used in related technologies cannot associate the emotions of multiple users present with their identities, and therefore cannot track the future emotional characteristics of a specific user. In this embodiment, identity recognition can be performed based on facial images to determine the object corresponding to the facial image. For the target object image, the server can perform facial region recognition on the target object image to obtain a set of facial images.

[0064] For a group of facial images, the server can perform object recognition on each facial image to obtain object information for a group of objects. The group of objects here includes the target object and may also include other objects in addition to the target object. For the other objects, emotion tracking can be performed in a similar manner as for the target object, thereby controlling the target smart device or other smart devices to perform corresponding device operations. Here, the number of objects included in the above group of objects is less than or equal to the number of facial images included in the group of facial images (the corresponding objects may not be recognized for some images).

[0065] Object recognition can be performed on each facial image separately by inputting each facial image into a face recognition model, and obtaining an object recognition result output by the face recognition model. Here, the face recognition model inputs the user's face image (i.e., each facial image), and outputs the user information of the current user (i.e., object information, requiring that the current user has registered their face on the smart device).

[0066] Through this embodiment, by performing object recognition on facial images and obtaining object information of each object, it is convenient to track the emotions of different users and improve the sustainability of device operations.

[0067] In an exemplary embodiment, after performing emotional state recognition on the target object based on the target object image and / or the target object voice data to obtain the current emotional state of the target object, the method further includes:

[0068] S31, when the distance between the detected target object and the target smart device is less than or equal to a preset distance threshold, adjusting the image acquisition component from a closed state to an open state;

[0069] S32 , when the target object is located in the image to be collected by the image collection component and the object portion satisfies a preset condition, the image collection component is used to collect an image of the target object to obtain an image of the target object.

[0070] In this embodiment, the object data of the target object includes an image of the target object, and the image of the target object may be similar to that in the aforementioned embodiment. In order to improve the accuracy of image acquisition, the target smart device may detect the distance to the target object. For example, a human body sensor may be arranged on the target smart device, and the human body sensor may detect whether the distance between the target smart device and the target object is less than a preset distance threshold (which may be the perception distance of the human body sensor). The human body sensor may be an infrared ranging sensor. When it is detected that the distance between the target object and the target smart device is less than or equal to the preset distance threshold, the image acquisition component may be controlled to be adjusted from an off state to an on state to acquire an image of the target object.

[0071] The target intelligent device can directly capture an image of the target object to obtain a target captured image. The object image captured in the above manner may fail to recognize the emotional state because it does not contain an object part that can recognize the emotional state. In this embodiment, when the object part of the target object located in the image to be captured by the image capture component meets the preset conditions, the target object can be captured by the image capture component to obtain a target captured image. The above-mentioned object part can include the object face of the target object, and can also include the torso or fingers of the target object. The above-mentioned preset conditions can include one or more. It can be that when the image to be captured includes the object face of the target object, the target object is captured to obtain a target captured image. It can also be that when the image to be captured includes both the object face and the finger parts of the target object, the target object is captured to obtain a target captured image. It can also be other conditions, which are not limited in this embodiment.

[0072] For example, the target smart device can capture an image of the user through the camera to obtain a target captured image when it detects that the user's face, torso, fingers and other parts within the image to be captured by the camera meet preset conditions (for example, the user's face, torso and fingers are detected at the same time).

[0073] Through this embodiment, the image acquisition component is turned on when it is detected that the distance between the user and the target smart device meets the target distance threshold, and the image of the user is captured when it is detected that the object part of the user located at the camera meets the preset conditions. This can improve the usability of the captured object image and improve the accuracy of emotional state recognition.

[0074] In an exemplary embodiment, when a target object is located in an image to be captured by an image capturing component and a part of the target object satisfies a preset condition, capturing an image of the target object by the image capturing component to obtain an image of the target object includes:

[0075] S41, when the target object is located in the image to be captured and the object part includes the object's face, and the ratio between the area of ​​the object's face and the area of ​​the image to be captured is greater than or equal to a target ratio, performing image capture on the target object by an image capture component to obtain an image of the target object; or

[0076] S42 , when the target object is located in the object part of the image to be captured, including the face and hands of the object, performing image capture on the target object by an image capture component to obtain an image of the target object.

[0077] In this embodiment, the preset condition may include one or more conditions, including but not limited to at least one of the following:

[0078] When the target object is located in the object part of the image to be captured, including the object's face, and the ratio between the area of ​​the object's face and the area of ​​the image to be captured is greater than or equal to the target ratio, the image capture component is used to capture the target object to obtain the target object image.

[0079] For example, when the target smart device detects the user's face and the ratio of the area of ​​the user's face to the area of ​​the image to be captured is greater than or equal to the target ratio, the user's image can be captured through the camera to obtain the target object image.

[0080] In the case that the target object is located at the object part of the to-be-collected image, including the face and hands of the object, the image collection component is used to collect the image of the target object to obtain the target object image.

[0081] For example, when the target smart device detects that the image to be captured also includes the user's face or hands, it can capture an image of the user through a camera to obtain the target object image.

[0082] In addition, when other conditions are met, the image acquisition component may be used to acquire an image of the target object to obtain the target object image, which is not limited in this embodiment.

[0083] For example, when a user meets certain conditions, the camera is turned on to take a photo of the user, requiring that the entire face be captured. The following methods determine whether to turn the camera on: the ratio of the user's face area to the image area meets a certain threshold; or the user's face and fingers appear in the image simultaneously.

[0084] Through this embodiment, when it is detected that the user's face proportion is greater than the target threshold, or the user's face and hands are detected at the same time, image capture of the user is performed, which can improve the usability of the captured object image and improve the accuracy of emotional state recognition.

[0085] In an exemplary embodiment, the target object image includes a facial image of the target object; and performing emotional state recognition on the target object based on the target object image and / or the object voice data to obtain the current emotional state of the target object includes:

[0086] S51, performing emotion category recognition on the facial image of the target object to obtain the current emotion category of the target object, wherein the current emotional state includes the current emotion category;

[0087] S52, when the target object image also includes a part image of a preset part of the target object, performing part motion recognition on the part image of the preset part to obtain a target part motion;

[0088] S53: Determine the emotional intensity that matches the target body part movement as the current emotional intensity of the target object, wherein the current emotional state also includes the current emotional category.

[0089] In this embodiment, the target object image may include a facial image of the target object. The server may perform emotion category recognition on the facial image of the target object to obtain the target object's current emotion category and a confidence level corresponding to the current emotion category. The manner in which the server performs emotion category recognition on the facial image of the target object to obtain the target object's current emotion category is similar to the manner in which the server performs emotional state recognition on the target object image to obtain the target object's current emotional state in the aforementioned embodiment, and is not further described here.

[0090] The target object image may also include a part image of a preset part of the target object. The preset part may be a part that can reflect the target object's emotional state, such as a torso or finger. The server may perform part motion recognition on the part image of the preset part to obtain the target part motion. The server performs part motion recognition on the part image of the preset part to obtain the target part motion in a manner similar to the method of performing emotional state recognition on the target object image to obtain the target object's current emotional state in the aforementioned embodiment, and will not be further described here.

[0091] Optionally, the preset part may include the torso of the target object, and the target object image includes a full-body image (full-body picture) of the target object. The full-body image is transmitted to a body motion recognition model (the model may be located in the cloud) to obtain a body motion label of the target object. Here, the body motion recognition model is a classification model used to identify the stretching state of the user's body state. The labels of the stretching state include: squatting, standing with shoulders curled up, sitting with shoulders curled up, sitting relaxed, standing relaxed, etc.

[0092] Optionally, the preset part may include the hand of the target object, and the target object image includes the target object's hand image (a picture of both hands). The object hand image is transmitted to a finger motion recognition model (the model may be located in the cloud) to obtain the gesture of the target object. Here, the finger motion recognition model is a classification model used to identify the stretching state of the user's fingers. The labels of the stretching state include: fisting, relaxation, etc.

[0093] For example, a smart device can identify and segment a captured full-body image of a user's face, hands, and torso, generating three images: the face, hands, and body. These images are then passed to the next processing module. While the face must be captured in its entirety, if the hands and torso are not recognized, empty data can be sent to the next step.

[0094] The facial image can be simultaneously transmitted to the face recognition model and the facial emotion recognition model in the cloud. The face recognition model determines the user's identity (for example, the user is Xiao Ming), and the facial emotion recognition model generates a facial expression label (crying) and the label's confidence level. The full-body image and the hand image are respectively transmitted to the body action recognition model and the finger action recognition model in the cloud to obtain the user's body action label (standing with shoulders hunched) and hand gesture (clenching fist). If the body action or finger action image is empty, the output is also empty.

[0095] The server can also determine the emotional intensity that matches the target body's motion as the target object's current emotional intensity. The current emotional state includes the target object's current emotional category and current emotional intensity (which can be the intensity of the current emotion). The current emotional intensity can be represented by an intensity value in the range [0, 10]. A larger intensity value indicates a stronger current emotional intensity, with 0 representing the lowest emotional intensity and 10 representing the highest emotional intensity.

[0096] For example, each body movement label can correspond to an emotion's intensity. The minimum intensity value is 0, and the maximum is 10. The body movement recognition model takes as input an image of the user's body movement, and outputs a label name and the corresponding intensity level. Each hand gesture label can correspond to an emotion's intensity. The minimum intensity value is 0, and the maximum is 10. The finger movement recognition model takes as input an image of the user's finger movement, and outputs a label name and the corresponding intensity level.

[0097] Through this embodiment, by performing emotional state recognition on the facial image of the object and the part movements of the preset parts, the efficiency and accuracy of emotional state recognition can be improved.

[0098] In an exemplary embodiment, the preset parts include the subject's hands and the subject's torso, and the target part movements include target hand movements and target body movements. Correspondingly, determining the emotional intensity that matches the target part movement as the target subject's current emotional intensity includes:

[0099] S61 : Determine the weighted sum of the emotion intensity matching the target hand movement and the emotion intensity matching the target body movement as the current emotion intensity of the target object.

[0100] The preset body parts may include the subject's hands and torso. Accordingly, the target body part motions may include target hand motions and target body motions. The target hand motions and target body motions are similar to those described in the previous embodiment. The target hand motions may include clenching a fist, relaxing fingers, etc., and the target body motions may include squatting, standing with shoulders hunched, etc. In this embodiment, the types of target hand motions and target body motions are not limited.

[0101] In this embodiment, the server can respectively obtain the emotion intensity that matches the target hand movement and the emotion intensity that matches the target body movement. It can determine the emotion intensity that matches the target hand movement based on the preset correspondence between hand movement and emotion intensity, and determine the emotion intensity that matches the target body movement based on the preset correspondence between body movement and emotion intensity. It can also obtain the emotion intensity that matches the target hand movement and the emotion intensity that matches the target body movement through other methods, which are not limited to this in this embodiment.

[0102] The server may determine either the intensity of the emotion matched with the target hand movement or the intensity of the emotion matched with the target body movement as the target subject's current emotion. To improve the accuracy of emotion recognition, the target subject's current emotion intensity may be determined as a weighted sum of the intensity of the emotion matched with the target hand movement and the intensity of the emotion matched with the target body movement. For example, the target subject's current emotion intensity may be determined as the average of the intensity of the emotion matched with the target hand movement and the intensity of the emotion matched with the target body movement.

[0103] Through this embodiment, by determining the weighted sum of the emotion intensity matching the hand movement and the emotion intensity matching the body movement as the current emotion intensity, the accuracy and efficiency of emotional state recognition can be improved.

[0104] In an exemplary embodiment, performing emotional state recognition on a target object based on the target object image and / or the target object voice data to obtain the current emotional state of the target object includes:

[0105] S71, based on the target object image and the target object voice data, respectively, performing emotional state recognition on the target object to obtain multiple emotional states;

[0106] S72, performing emotional state fusion on the multiple emotional states to obtain the current emotional state of the target object.

[0107] In this embodiment, the target object's object data may include object data of multiple modalities, such as an image of the target object and an audio recording of the target object. If the target object's object data includes both an image and audio recording, the server performs emotional state recognition on the target object based on the image and audio recordings, respectively, to obtain multiple emotional states. Optionally, the emotional states may include the target object's emotion category and the intensity corresponding to the emotion category.

[0108] After obtaining multiple emotional states, the server can randomly select one emotional state from the multiple emotional states as the current emotional state of the target object. In order to improve the accuracy of emotional state recognition, the server can perform emotional state fusion on the multiple emotional states to obtain the current emotional state of the target object.

[0109] Through this embodiment, by identifying and fusing object data of multiple modalities to obtain the user's current emotional state, the efficiency and convenience of emotional state recognition can be improved, and the sustainability of device operations can be improved.

[0110] In an exemplary embodiment, based on the target object image and the target object voice data, the target object's emotional state is recognized to obtain multiple emotional states, including:

[0111] S81, performing emotional state recognition on a facial image of a target object to obtain a first emotional state, wherein the first emotional state includes a first emotion category, and the target object image includes the facial image of the target object;

[0112] S82: Perform emotional state recognition on the target object's voice data to obtain a second emotional state, wherein the second emotional state includes a second emotional category.

[0113] In this embodiment, the target object image may include a facial image of the target object. The server performing emotional state recognition on the target object based on the target object image and the target object voice data may include performing emotional state recognition on the facial image and the target object voice data to obtain a first emotional state and a second emotional state. The first emotional state may include a first emotion category, and the second emotional state may include a second emotion category.

[0114] The method for performing emotional state recognition on the target object's facial image is similar to that in the aforementioned embodiment and will not be described in detail here. The server may perform emotional state recognition on the target object's voice data by inputting the target object's voice data into a voice emotion model, and using the voice emotion model to identify the target object's emotion category (or emotion category), and may also identify the target object's emotion intensity.

[0115] For example, a smart device receives a user's voice query Q1 and transmits it to the cloud. Using a voice emotion model, the device identifies the user's emotion category and intensity (with 0 being the mildest and 10 being the strongest). The recognition result is: Xiao Ming's emotion = aggrieved, excited, intensity = 7. Combining the analysis results of the user's image with the emotion analysis results of the user's voice, and using a comprehensive emotion judgment model strategy, the current user's emotion and intensity can be corrected.

[0116] Here, it should be noted that the object voice data can be obtained by collecting voice data through the sound receiving component of the target smart device after the target object wakes up the target smart device, or it can be obtained by collecting voice data through the sound receiving component of the target smart device in a non-awakening state based on the user's authorization information. This is not limited in this embodiment.

[0117] Through this embodiment, by performing emotional state recognition on the user's target facial image and target voice data, the convenience and accuracy of emotional state recognition can be improved.

[0118] In an exemplary embodiment, the current emotional state includes the current emotional category of the target object. Correspondingly, emotional state fusion is performed on multiple emotional states to obtain the current emotional state of the target object, including:

[0119] S91, when the first emotion category and the second emotion category are consistent, determining either the first emotion category or the second emotion category as the current emotion category;

[0120] S92, when the first emotion category and the second emotion category are inconsistent, the emotion category with higher confidence in the first emotion category and the second emotion category is determined as the current emotion category; or, the emotion category in the first emotion category and the second emotion category that matches the position state of the preset position of the target object is determined as the current emotion category.

[0121] In this embodiment, the first emotion category is an emotion category identified based on the subject's facial image, and the second emotion category is an emotion category identified based on the subject's voice data. The current emotional state includes the target subject's current emotion category. The server performs emotional state identification on the target subject based on the target subject's image and voice data, respectively. Obtaining multiple emotional states may include fusing the first emotion category and the second emotion category to obtain the target subject's current emotion category.

[0122] The first emotion category and the second emotion category may be consistent or inconsistent. Here, consistency means that the two are exactly the same or match, for example, both belong to positive emotions, both belong to negative emotions, etc. The fusion of the first emotion category and the second emotion category can be performed based on the consistency of the first emotion category and the second emotion category. When the first emotion category and the second emotion category are consistent, any emotion category in the first emotion category and the second emotion category can be determined as the current emotion category; when the first emotion category and the second emotion category are inconsistent, the emotion category with a high confidence level in the first emotion category and the second emotion category is determined as the current emotion category, or the emotion category in the first emotion category and the second emotion category that matches the preset part state of the target object is determined as the current category. The current emotion category can also be determined by other means, which is not limited in this embodiment.

[0123] As an optional embodiment, the confidence level of the first emotion category can be determined based on the target object's facial image, that is, the first emotion category and the confidence level of the first emotion category are identified based on the target object's facial image. The confidence level of the second emotion category can be determined based on the target object's voice data, that is, the second emotion category and the confidence level of the second emotion category are identified based on the target object's facial image. The higher the confidence level of an emotion category, the higher its credibility. In the event that the first emotion category and the second emotion category are inconsistent, the emotion category with the higher confidence level of the first emotion category and the second emotion category can be determined as the current emotion category.

[0124] As another optional embodiment, when the target object image includes a part image of a preset part of the target object, the server can identify the part image of the preset part, obtain the part status of the preset part, and determine the emotion category between the first emotion category and the second emotion category that matches the preset part status of the target object as the current emotion category.

[0125] For example, the comprehensive emotion judgment model can use strategies to correct inconsistencies between the conclusions drawn by speech emotion recognition and image emotion recognition. The user's current emotion classification is determined by the emotion classification results of the facial emotion recognition model and the speech emotion recognition model. The user's current emotion classification is determined as follows: if the emotion labels obtained by facial recognition and speech emotion recognition are consistent, the current user's emotion is considered to be that label. If the emotion labels obtained by facial recognition and speech emotion recognition are inconsistent, the user's emotion can be determined in two ways:

[0126] Method 1: Determine the user's emotions based on labels with higher confidence.

[0127] For example, if the confidence level of voice emotion recognition that the user is angry is 99%, while the confidence level of image recognition that the user's emotion is neutral is 60%, then it is considered that the user's true emotion is anger.

[0128] Method 2: Combine body movements and finger movements to judge.

[0129] For example, if voice emotion recognition shows that the user is angry, but facial expression recognition shows a smile, body movements are stretched, and fingers are relaxed, then it is considered that the voice emotion recognition is biased, and the user's emotion is corrected to neutral or calm.

[0130] For example, through facial image recognition, the current user is identified as Xiao Ming, the current facial expression is crying and excited, the shoulders are shaking, the fingers are clenched into fists, and the voice emotion is aggrieved and excited. There is no conflict between the voice emotion recognition result and the image emotion recognition result, then it is considered that the user Xiao Ming's emotions are aggrieved and excited.

[0131] Through this embodiment, by fusing the emotion categories identified by object data of different dimensions and determining the current emotion category of the user, the convenience and efficiency of emotion recognition can be improved.

[0132] In an exemplary embodiment, the current emotional state also includes the current emotional intensity of the target object, which may be the intensity of the current emotion. Correspondingly, performing emotional state fusion on multiple emotional states to obtain the current emotional state of the target object further includes:

[0133] S101, when the first emotion category and the second emotion category are consistent, determining the first emotion intensity in the second emotion state as the current emotion intensity; or

[0134] S102, when the target object image also includes a part image of a preset part of the target object, a weighted sum of the first emotion intensity and the second emotion intensity is determined as the current emotion intensity, wherein the first emotion intensity is the emotion intensity included in the second emotional state, and the second emotion intensity is the emotion intensity identified from the part image of the preset part.

[0135] In this embodiment, the server may fuse the multiple emotional states by fusing the emotional intensities of the multiple emotional states. The fusing of the emotional intensities of the multiple emotional states may be performed by determining the first emotional intensity of the second emotional state as the current emotional intensity when the first emotional category and the second emotional category are consistent.

[0136] Optionally, when the first emotion category and the second emotion category are consistent, and when the target object image also includes an image of a predetermined part of the target object, a weighted sum (e.g., an average) of the first emotion intensity and the second emotion intensity may be determined as the current emotion intensity. The first emotion intensity is the emotion intensity included in the second emotional state, and the second emotion intensity is the emotion intensity identified from the image of the predetermined part.

[0137] The server can perform emotion intensity recognition on a preset portion of the target object to obtain a second emotion intensity that matches the state of the preset portion of the target object. The manner in which the server performs emotion intensity recognition on the preset portion of the target object to obtain the second emotion intensity that matches the state of the preset portion of the target object is similar to the manner in which the server performs emotion state recognition on the target object image to obtain the current emotional state of the target object in the aforementioned embodiment, and is not further described here.

[0138] For example, the method for judging the intensity of user emotions is as follows: if the voice emotion recognition result is consistent with the image emotion recognition result, the intensity of the user's body movements, the intensity of the user's finger movements, and the intensity of the voice emotions can be averaged to obtain the intensity of the user's emotions.

[0139] Through this embodiment, when the multiple recognized emotion categories are consistent, the user's emotion intensity is obtained by weightedly summing the emotion intensity (i.e., intensity) recognized based on voice data and the emotion intensity recognized based on the part image of the preset part, which can improve the accuracy of emotional state recognition.

[0140] In an exemplary embodiment, similar to the aforementioned embodiment, the current emotional state also includes the current emotional intensity of the target object. Correspondingly, performing emotional state fusion on multiple emotional states to obtain the current emotional state of the target object also includes:

[0141] S111, when the first emotion category and the second emotion category are inconsistent, the current emotion category is the first emotion category, and the target object image also includes a part image of a preset part of the target object, determining the emotion intensity recognized from the part image of the preset part as the current emotion intensity;

[0142] S112, when the first emotion category and the second emotion category are inconsistent and the current emotion category is the second emotion category, determining the first emotion intensity in the second emotional state as the current emotion intensity;

[0143] S113, when the first emotion category and the second emotion category are inconsistent, the current emotion category is the second emotion category, and the target object image also includes a part image of a preset part of the target object, a weighted sum of the first emotion intensity and the second emotion intensity is determined as the current emotion intensity, wherein the first emotion intensity is the emotion intensity included in the second emotional state, and the second emotion intensity is the emotion intensity identified from the part image of the preset part.

[0144] In this embodiment, if the first emotion category and the second emotion category are inconsistent, the current emotion intensity of the target object can be determined based on whether the current emotion category is the first emotion category or the second emotion category and whether the target object image contains a part image of a preset part of the target object.

[0145] If the current emotion category is the first emotion category and the target object image includes a region image of a preset region of the target object, the target object's emotion intensity can be determined in a manner similar to the aforementioned embodiment, with the emotion intensity identified from the region image of the preset region being determined as the current emotion intensity. If there are multiple preset regions, the current emotion intensity can be determined as the weighted sum of the emotion intensities identified from the region images of each preset region.

[0146] If the current emotion category is the second emotion category, the first emotion intensity in the second emotion state may be determined as the current emotion intensity. In this case, the target object image may include or exclude the image of the preset part of the target object.

[0147] Optionally, if the current emotion category is the second emotion category, and the target object image also includes a part image of a preset part of the target object, the emotion intensity identified from the part image of the preset part, that is, the second emotion category, can be determined in a manner similar to the above, and the weighted sum of the first emotion intensity and the second emotion intensity can be determined as the current emotion intensity.

[0148] For example, the methods for judging the intensity of user emotions include the following:

[0149] If the voice emotion recognition result is consistent with the image emotion recognition result, the intensity of the user's body movements, the intensity of the user's finger movements, and the intensity of the voice emotion are averaged to obtain the intensity of the user's emotion;

[0150] If the voice emotion recognition result is inconsistent with the image emotion recognition result, and after judgment, the user's true emotion is consistent with the voice emotion recognition result, then the user's emotional intensity is equal to the emotional intensity of the voice emotion recognition result;

[0151] If the voice emotion recognition result is inconsistent with the image emotion recognition result, and after judgment, the user's true emotion is consistent with the facial emotion recognition result, then the intensity of the user's emotion is equal to the average of the intensity of the user's body movements and the intensity of the user's finger movements, which is determined as the intensity of the user's emotion.

[0152] According to this embodiment, if the first emotion category and the second emotion category are inconsistent, the current emotion intensity is determined by comprehensively considering the final emotion category and the emotion intensity recognized from the part image of the preset part, which can improve the accuracy of emotional state recognition.

[0153] In an exemplary embodiment, determining a target device operation to be executed by a target smart device based on the current emotional state and the historical emotional state further includes:

[0154] S121, when it is determined based on the current emotional state and the historical emotional state that the target object is in a negative emotion and the intensity of the negative emotion has decreased, determining the device operation to be executed by the target smart device to be a first device operation;

[0155] S122, when it is determined based on the current emotional state and the historical emotional state that the target object is in a negative emotion and the intensity of the negative emotion is increasing, determining the device operation to be executed by the target smart device to be a second device operation;

[0156] S123, when it is determined based on the current emotional state and the historical emotional state that the target object is in a negative emotion, and the emotional intensity of the negative emotion is greater than or equal to the preset intensity threshold and the duration reaches the preset duration threshold, determine that the device operation to be executed by the target smart device is a third device operation.

[0157] In this embodiment, to determine the target device operation, the target device operation to be executed by the target smart device can be determined based on the current emotional state and historical emotional states. The target device operation includes at least a first device operation, a second device operation, and a third device operation. Here, the first device operation can be an operation to maintain or reduce the level of emotional soothing, the second device operation can be an operation to enhance the level of emotional soothing, and the third device operation can be an operation to issue a warning to the target smart device to indicate that the target person has been in a negative mood for a long time.

[0158] When it is determined based on the current emotional state and the historical emotional state that the target object is in a negative emotion and the emotional intensity of the negative emotion is reduced, the server can determine the first device operation as the device operation to be executed by the target smart device; when it is determined based on the current emotional state and the historical emotional state that the target object is in a negative emotion and the emotional intensity of the negative emotion is increased, the server can determine the second device operation as the device operation to be executed by the target smart device; and when it is determined based on the current emotional state and the historical emotional state that the target object is in a negative emotion and the emotional intensity of the negative emotion is greater than or equal to the preset intensity threshold and the duration reaches the preset duration threshold, the server can determine the third device operation as the device operation to be executed by the target smart device.

[0159] For example, based on the user's identity, the cloud can be used to query the user's historical emotional state and degree over a continuous period of time (for example, the query shows that the user has been in a negative state such as negative and depressed for the past three days, with degrees of 4, 5, and 6 respectively). Based on the user's current and historical emotional state labels and degrees, it can be determined whether the user's emotions are changing in a positive direction. For example, the current user's emotions are aggrieved and excited, with a degree of 7. Compared with the past three days, the negative emotional state has become more serious.

[0160] If the user's negative emotions deepen, we can strengthen comfort and interaction with them through guided communication and other means. If the user's negativity exceeds a certain threshold (for example, threshold = 9) and persists for a period of time, we recommend that the user seek professional advice. If the user's negative emotions ease, we can use normal emotional soothing methods. For example, if we notice that Xiao Ming's negative emotions are getting worse, we can use caring and gentle TTS language to guide Xiao Ming to talk about what's making him unhappy, such as "Baby, why are you so wronged? Come and talk to me."

[0161] For example, in a kindergarten scenario, it can help identify and track children's emotions. When children quarrel or cry loudly over trivial matters such as toys and snacks, the smart device can promptly broadcast the following words to the children: "Friends should be polite to each other, play this toy together", give the children corresponding comfort, and track the subsequent emotions of the two children to identify the impact of the incident on the children. If it is found that the expressions of the two children are not very happy every time they meet, the smart device can be used to broadcast: "Are you still quarreling over a toy? Have you forgotten the happy time you had eating ice cream together?" to ease the relationship between them.

[0162] For example, in a family setting, the relationship between parents often affects a child's development. For example, frequent quarrels between parents can have a negative impact on a child's growth. When a smart device detects that a child is affected by their parents' quarrels, it can use facial expression recognition to determine whether the child is depressed and sad, and then broadcast a reassuring message to the child: "Don't worry, they love you very much." The smart device also uses the child's facial recognition results to correlate historical emotional changes. If a child's mood persists, the smart device can promptly notify the parents and provide scientific counseling suggestions.

[0163] Through this embodiment, the target operation corresponding to the target smart device is determined according to the current emotional state and the historical emotional state, which can improve the accuracy and sustainability of the device performing the device operation.

[0164] The following is an explanation of the digital twin multimodal device control method in the embodiment of the present application with reference to an optional example. In this optional example, the target smart device is a smart home device, the image acquisition component is a camera, and the preset parts include the torso and the hand.

[0165] In related technologies, smart home devices recognize user emotions by receiving user voice in real time. There are two commonly used emotion recognition methods: 1) analyzing user emotions from voice signal waves; 2) converting user voice into text and then identifying user emotions through NLP (Natural Language Processing) technology. For example, if the user voice text is "You are so annoying" and the user is detected to be angry, the smart home device may give some teasing words or play music to relieve the user's emotions.

[0166] However, the aforementioned method of controlling smart home device operations, using single-modal emotion recognition based on user voice, struggles to accurately identify emotions for users with limited access to voice data. Furthermore, single-modal emotion recognition technology only identifies and alleviates user emotions at the current moment. If the user stops speaking, it's impossible to determine whether the user's emotions have been effectively alleviated.

[0167] The present invention proposes a multimodal user emotion recognition and tracking technology that combines voice and images. It is a multimodal emotion recognition and tracking method based on face, body, finger and voice. It can judge the user's current emotional state and provide psychological comfort to users in a negative state in a timely manner. It can also continue to identify the user's emotional fluctuations through images and voice in the next few days, judge whether the user's negative emotions have been alleviated, and provide different comfort and encouragement in a timely manner according to the degree of alleviation, so as to ensure that the user's negative emotions are truly resolved and avoid the occurrence of abnormal situations.

[0168] Furthermore, the multimodal user emotion recognition and tracking technology provided in this optional example can accurately identify each person's emotional state in multi-person scenarios and associate their face with their identity. When there are multiple users, each user's identity and emotion can be precisely associated. This tracking mechanism analyzes users' historical emotional states and determines their psychological tendencies, enabling AI (artificial intelligence) to further provide appropriate emotional soothing measures or psychological counseling.

[0169] The core components of the multimodal user emotion recognition and tracking technology combining voice and image provided in this optional example may include the following models: 1. Voice emotion feature extraction model; 2. Facial emotion feature extraction model; 3. Body movement feature extraction model; 4. Finger movement feature extraction model; 5. Emotion judgment comprehensive model; 6. Facial identity recognition model; 7. Facial emotion association and user emotion tracking mechanism.

[0170] Models 1, 2, 3, and 4 are fed into model 5 as feature vectors to identify and detect user emotions. Considering the unique user population, the comprehensive emotion judgment model should be compatible with single-modal data, such as voice or image data. After identifying the user's emotion, the emotion is associated with the user's identity based on the facial recognition results. The user's historical emotional changes are tracked, and corresponding feedback is provided based on these changes.

[0171] Combine Figure 3 As shown, when a user interacts with a smart device, the process of the digital twin multimodal device control method in this optional example may include the following steps:

[0172] Step 1: When the human body sensor of the smart device recognizes that the distance between the user and the device is less than a certain threshold, the smart device turns on the camera.

[0173] In step 2, when the user meets certain conditions (for example, the camera detects that the ratio of the user's face area to the image area meets a certain threshold, or the user's face and fingers appear simultaneously in the camera), the smart device turns on the camera to take a photo of the user, requiring that the user's face be fully captured. Furthermore, if the smart device detects the user speaking, steps 6 and 7 are triggered simultaneously.

[0174] In step 3, the smart device identifies and segments the facial area, hand area, and human torso area of ​​the captured full-body image of the user to obtain three images of the user's facial image, hand image, and full-body image.

[0175] Step 4: The facial image is transmitted to the face recognition model and facial emotion recognition model in the cloud at the same time, and the face recognition model is used to obtain the user's identity, facial expression label, and label confidence.

[0176] Step 5: The full-body image and the two-hand image are transmitted to the body motion recognition model and the finger motion recognition model in the cloud respectively to obtain the user's body motion label and the user's gesture, and then determine the intensity of the user's emotions.

[0177] Step 6: When the smart device detects that the user is speaking, it can obtain a voice message spoken by the user.

[0178] Step 7: Use the cloud-based speech emotion recognition model to identify the user's emotion category and intensity based on the received speech.

[0179] Step 8: Combining the user's identity, facial expression label, user's body emotion recognition, finger movement recognition, and the emotional category and degree of the user's voice, the current user's emotion and degree are corrected according to the emotional judgment comprehensive model strategy.

[0180] Step 9: Based on the user's identity, query the user's historical emotional state and degree over a continuous period of time from the cloud.

[0181] Step 10: Determine whether the user's emotions are changing in a positive direction based on the user's current and historical emotional state labels and levels.

[0182] Step 11: If the user's negative emotions deepen, we will strengthen comfort and interaction with the user through guided communication and other means. If the user's negativity exceeds a certain threshold and persists for a period of time, we will recommend that the user seek professional advice. If the user's negative emotions have eased, we will use normal emotional comfort methods.

[0183] Through this optional example, user emotions are identified through data in two modalities, image (face + body + finger) and voice. Even if the user does not speak, the user's emotional abnormalities can be detected, thereby improving the AI's emotional relief ability; in multi-person scenarios, through the face emotion association and user emotion tracking mechanism, user identity and user emotion are associated, and corresponding emotional soothing feedback is given to users in different emotional states; for a single user, through the face emotion association and user emotion tracking mechanism, the user's historical emotional changes are associated, and corresponding emotional soothing measures are given for different emotional changes to reduce the occurrence of psychological abnormalities.

[0184] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0185] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory, Read-Only Memory) / RAM (Random Access Memory, Random Access Memory), a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0186] According to another aspect of an embodiment of the present application, a digital twin multimodal device control apparatus for implementing the above-mentioned digital twin multimodal device control method is also provided. Figure 4 This is a structural block diagram of an optional digital twin multimodal device control device according to an embodiment of the present application, such as Figure 4 As shown, the device may include:

[0187] The first recognition unit 402 is configured to perform emotional state recognition on the target object based on the target object image and / or the target object voice data to obtain the current emotional state of the target object;

[0188] The first acquisition unit 404 is connected to the first recognition unit 402 and is used to acquire the historical emotional state of the target object within a historical time period when the current emotional state is a negative state;

[0189] a determining unit 406 , connected to the first acquiring unit 404 , configured to determine a target device operation to be executed by the target smart device based on the current emotional state and the historical emotional state;

[0190] The control unit 408 is connected to the determining unit 406 and is configured to control the target smart device to perform the target device operation.

[0191] It should be noted that the first identification unit 402 in this embodiment can be used to execute the above step S202, the first acquisition unit 404 in this embodiment can be used to execute the above step S204, the determination unit 406 in this embodiment can be used to execute the above step S206, and the control unit 408 in this embodiment can be used to execute the above step S208.

[0192] Through the above module, the emotional state of the target object is identified based on the target object image and / or object voice data to obtain the current emotional state of the target object; when the current emotional state is negative, the historical emotional state of the target object in the historical time period is obtained; based on the current emotional state and the historical emotional state, the target device operation to be executed by the target smart device is determined; the target smart device is controlled to execute the target device operation, which solves the problem of poor sustainability of device operation due to the inability to obtain user feedback information in a timely manner in the method of controlling the device to execute the device operation based on the identified emotion category in the related art, thereby improving the sustainability of device operation.

[0193] In an exemplary embodiment, the apparatus further comprises:

[0194] a second acquisition unit, configured to acquire an image of the target object captured by the image acquisition component before performing emotional state recognition on the target object based on the target object image and / or the object voice data to obtain the current emotional state of the target object, wherein the target object image is an image including the face of the target object;

[0195] The third acquiring unit is configured to acquire the target voice data collected by the voice collecting component, wherein the target voice data is voice data uttered by the target object.

[0196] In an exemplary embodiment, the apparatus further comprises:

[0197] a second recognition unit, configured to, after acquiring the target object image captured by the image acquisition component, perform facial region recognition on the target object image to obtain a set of facial images;

[0198] The third recognition unit is configured to perform object recognition on each facial image in a group of facial images to obtain object information of a group of objects, wherein the group of objects includes a target object.

[0199] In an exemplary embodiment, the apparatus further comprises:

[0200] an adjustment unit, configured to, before performing emotional state recognition on the target object based on the target object image and / or the target object voice data to obtain the target object's current emotional state, adjust the image acquisition component from an off state to an on state when it is detected that the distance between the target object and the target smart device is less than or equal to a preset distance threshold;

[0201] The acquisition unit is used to acquire an image of the target object through the image acquisition component when the target object is located in the image to be acquired by the image acquisition component and the object part meets the preset conditions, so as to obtain an image of the target object.

[0202] In an exemplary embodiment, the acquisition unit includes:

[0203] a first acquisition module configured to acquire an image of the target object by using an image acquisition component when the target object is located in the image to be acquired and the object part includes the face of the target object, and the ratio between the area of ​​the face of the target object and the area of ​​the image to be acquired is greater than or equal to a target ratio; or

[0204] The second acquisition module is used to acquire an image of the target object by using an image acquisition component when the target object is located in an object part including the face and hands of the object within the image to be acquired.

[0205] In an exemplary embodiment, the first identification unit includes:

[0206] A first recognition module is configured to perform emotion category recognition on the facial image of the target object to obtain the current emotion category of the target object, wherein the current emotional state includes the current emotion category;

[0207] a second recognition module configured to, when the target object image further includes a part image of a preset part of the target object, perform part motion recognition on the part image of the preset part to obtain a target part motion;

[0208] The first determining module is configured to determine the emotion intensity that matches the target part movement as the current emotion intensity of the target object, wherein the current emotion state also includes a current emotion category.

[0209] In an exemplary embodiment, the preset parts include the subject's hands and the subject's torso, and the target part motion includes a target hand motion and a target body motion; the first determination module includes:

[0210] The first determination submodule is configured to determine the current emotion intensity of the target object as a weighted sum of the emotion intensity matching the target hand movement and the emotion intensity matching the target body movement.

[0211] In an exemplary embodiment, the first identification unit includes:

[0212] A third recognition module is used to identify the emotional state of the target object based on the target object image and the object voice data, and obtain multiple emotional states;

[0213] The fusion module is used to fuse multiple emotional states to obtain the current emotional state of the target object.

[0214] In an exemplary embodiment, the third identification module includes:

[0215] a first recognition submodule, configured to perform emotional state recognition on a facial image of a target object to obtain a first emotional state, wherein the first emotional state includes a first emotion category and the target object image includes a facial image of the target object;

[0216] The second recognition submodule is configured to perform emotional state recognition on the object voice data to obtain a second emotional state, wherein the second emotional state includes a second emotional category.

[0217] In one exemplary embodiment, the fusion module includes:

[0218] a second determining submodule, configured to determine, when the first emotion category and the second emotion category are consistent, either the first emotion category or the second emotion category as the current emotion category;

[0219] The third determination submodule is used to determine the emotion category with higher confidence in the first emotion category and the second emotion category as the current emotion category when the first emotion category and the second emotion category are inconsistent; or, to determine the emotion category in the first emotion category and the second emotion category that matches the position state of the preset position of the target object as the current emotion category.

[0220] In an exemplary embodiment, the current emotional state further includes the current emotional intensity of the target object; and the fusion module further includes:

[0221] The fourth determining submodule is configured to determine the first emotion intensity in the second emotional state as the current emotion intensity when the first emotion category and the second emotion category are consistent; or

[0222] The fifth determination submodule is used to determine the weighted sum of the first emotion intensity and the second emotion intensity as the current emotion intensity when the target object image also includes a part image of a preset part of the target object, wherein the first emotion intensity is the emotion intensity included in the second emotional state, and the second emotion intensity is the emotion intensity identified from the part image of the preset part.

[0223] In an exemplary embodiment, the current emotional state further includes the current emotional intensity of the target object; and the fusion module further includes:

[0224] a sixth determining submodule, configured to, when the first emotion category and the second emotion category are inconsistent, the current emotion category is the first emotion category, and the target object image further includes a part image of a preset part of the target object, determine the emotion intensity recognized from the part image of the preset part as the current emotion intensity;

[0225] A seventh determination submodule is configured to determine the first emotion intensity in the second emotional state as the current emotion intensity when the first emotion category and the second emotion category are inconsistent and the current emotion category is the second emotion category;

[0226] The eighth determination submodule is used to determine the weighted sum of the first emotion intensity and the second emotion intensity as the current emotion intensity when the first emotion category and the second emotion category are inconsistent, the current emotion category is the second emotion category, and the target object image also includes a part image of a preset part of the target object, wherein the first emotion intensity is the emotion intensity included in the second emotional state, and the second emotion intensity is the emotion intensity identified from the part image of the preset part.

[0227] In an exemplary embodiment, the determining unit further includes:

[0228] a second determining module, configured to, when it is determined based on the current emotional state and the historical emotional state that the target object is in a negative emotion and the emotional intensity of the negative emotion has decreased, determine that the device operation to be executed by the target smart device is a first device operation;

[0229] a third determining module, configured to, when it is determined based on the current emotional state and the historical emotional state that the target object is in a negative emotion and the intensity of the negative emotion is increasing, determine that the device operation to be executed by the target smart device is a second device operation;

[0230] The fourth determination module is used to determine that the device operation to be executed by the target smart device is a third device operation when it is determined that the target object is in a negative emotion based on the current emotional state and the historical emotional state, and the emotional intensity of the negative emotion is greater than or equal to the preset intensity threshold and the duration reaches the preset duration threshold.

[0231] According to another aspect of the embodiments of the present application, a storage medium is further provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the above-mentioned digital twin multimodal device control methods in the embodiments of the present application.

[0232] Optionally, in this embodiment, the above-mentioned storage medium may be located on at least one network device among the multiple network devices in the network shown in the above-mentioned embodiment.

[0233] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps:

[0234] S1, based on the target object image and / or the target object voice data, performing emotional state recognition on the target object to obtain the current emotional state of the target object;

[0235] S2, when the current emotional state is negative, obtaining the historical emotional state of the target object in the historical time period;

[0236] S3, determining a target device operation to be executed by the target smart device based on the current emotional state and the historical emotional state;

[0237] S4, controlling the target intelligent device to execute the target device operation.

[0238] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0239] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0240] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned digital twin multimodal device control method is also provided. The electronic device can be a server, a terminal, or a combination thereof.

[0241] Figure 5 is a structural block diagram of an optional electronic device according to an embodiment of the present application, such as Figure 5 As shown, it includes a processor 502, a communication interface 504, a memory 506 and a communication bus 508, wherein the processor 502, the communication interface 504 and the memory 506 communicate with each other through the communication bus 508, wherein,

[0242] Memory 506, for storing computer programs;

[0243] The processor 502 is configured to execute the computer program stored in the memory 506 to implement the following steps:

[0244] S1, based on the target object image and / or the target object voice data, performing emotional state recognition on the target object to obtain the current emotional state of the target object;

[0245] S2, when the current emotional state is negative, obtaining the historical emotional state of the target object in the historical time period;

[0246] S3, determining a target device operation to be executed by the target smart device based on the current emotional state and the historical emotional state;

[0247] S4, controlling the target intelligent device to execute the target device operation.

[0248] Optionally, the communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The communication interface is used for communication between the electronic device and other devices.

[0249] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Alternatively, the memory may also be at least one storage device located away from the aforementioned processor.

[0250] As an example, the memory 506 may include, but is not limited to, the first identification unit 402, the first acquisition unit 404, the determination unit 406, and the control unit 408 in the digital twin multimodal device control method apparatus. In addition, it may also include, but is not limited to, other module units in the digital twin multimodal device control apparatus, which will not be repeated in this example.

[0251] The above-mentioned processor can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0252] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0253] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only. The device for implementing the above-mentioned digital twin multimodal device control method may be a terminal device, which may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 5 It does not limit the structure of the above electronic device. For example, the electronic device may also include Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 5 Different configurations shown.

[0254] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0255] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0256] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application.

[0257] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0258] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0259] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the purpose of the solution provided in this embodiment.

[0260] In addition, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or at least two units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0261] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A digital twin multimodal device control method, characterized in that: include: Based on the target object image and / or the object voice data, the target object is identified in terms of its emotional state to obtain the current emotional state of the target object; When the current emotional state is a negative state, obtaining the historical emotional state of the target object in a historical time period; Determining a target device operation to be performed by a target smart device according to the current emotional state and the historical emotional state; Controlling the target smart device to perform the target device operation; Wherein, the target object image includes the object facial image of the target object; the emotional state recognition of the target object based on the target object image and / or the object voice data to obtain the current emotional state of the target object includes: performing emotional category recognition on the object facial image to obtain the current emotional category of the target object, wherein the current emotional state includes the current emotional category; when the target object image also includes a part image of a preset part of the target object, performing part action recognition on the part image of the preset part to obtain the target part action; determining the emotional intensity that matches the target part action as the current emotional intensity of the target object, wherein the current emotional state also includes the current emotional intensity, and the preset parts include the object's hands and the object's torso; The target part movement includes a target hand movement and a target body movement; the step of determining the emotion intensity matched with the target part movement as the current emotion intensity of the target object includes: determining the current emotion intensity of the target object as the weighted sum of the emotion intensity matched with the target hand movement and the emotion intensity matched with the target body movement; The method further includes: performing facial area recognition on the target object image to obtain a group of facial images; performing object recognition on each facial image in the group of facial images to obtain object information of a group of objects, wherein the group of objects includes the target object.

2. The method according to claim 1, characterized in that Before performing emotional state recognition on the target object based on the target object image and / or the object voice data to obtain the current emotional state of the target object, the method further includes: Acquire the target object image acquired by the image acquisition component, wherein the target object image is an image including the face of the target object; The object voice data collected by the voice collection component is acquired, wherein the object voice data is the voice data uttered by the target object.

3. The method according to claim 1, characterized in that Before performing emotional state recognition on the target object based on the target object image and / or the object voice data to obtain the current emotional state of the target object, the method further includes: When it is detected that the distance between the target object and the target smart device is less than or equal to a preset distance threshold, the image acquisition component is adjusted from a closed state to an open state; When the target object is located in the object part of the image to be collected by the image collection component and meets a preset condition, the image collection component is used to collect an image of the target object to obtain the target object image.

4. The method according to claim 3, characterized in that When the object part of the target object located in the image to be collected by the image collection component meets a preset condition, the image collection component is used to collect an image of the target object to obtain the image of the target object, including: In the case where the target object is located in the image to be collected and the object part includes the face of the object, and the ratio between the area of ​​the face of the object and the area of ​​the image to be collected is greater than or equal to the target ratio, the image of the target object is collected by the image collection component to obtain the image of the target object; or In the case that the target object is located in the object part of the to-be-collected image including the object's face and the object's hands, the image of the target object is collected by the image collection component to obtain the target object image.

5. The method according to claim 1, characterized in that The step of performing emotional state recognition on the target object based on the target object image and / or the object voice data to obtain the current emotional state of the target object includes: Based on the target object image and the object voice data, respectively, the target object is recognized for its emotional state to obtain multiple emotional states; Emotional state fusion is performed on the multiple emotional states to obtain the current emotional state of the target object.

6. The method according to claim 5, characterized in that Based on the target object image and the object voice data, the target object is respectively subjected to emotional state recognition to obtain multiple emotional states, including: Performing emotional state recognition on the target object's facial image to obtain a first emotional state, wherein the first emotional state includes a first emotional category and the target object image includes the object's facial image; Performing emotional state recognition on the object voice data to obtain a second emotional state, wherein the second emotional state includes a second emotional category.

7. The method according to claim 6, characterized in that The current emotional state includes the current emotional category of the target object; the emotional state fusion of the multiple emotional states to obtain the current emotional state of the target object includes: In a case where the first emotion category and the second emotion category are consistent, determining any one of the first emotion category and the second emotion category as the current emotion category; When the first emotion category and the second emotion category are inconsistent, the emotion category with a higher confidence level between the first emotion category and the second emotion category is determined as the current emotion category; or, the emotion category between the first emotion category and the second emotion category that matches the position state of a preset part of the target object is determined as the current emotion category.

8. The method according to claim 6, characterized in that The current emotional state also includes the current emotional intensity of the target object; and the emotional state fusion of the multiple emotional states to obtain the current emotional state of the target object further includes: In the case where the first emotion category and the second emotion category are consistent, determining the first emotion intensity in the second emotion state as the current emotion intensity; or In a case where the target object image also includes a part image of a preset part of the target object, a weighted sum of a first emotion intensity and a second emotion intensity is determined as the current emotion intensity, wherein the first emotion intensity is the emotion intensity included in the second emotional state, and the second emotion intensity is the emotion intensity identified from the part image of the preset part.

9. The method according to claim 6, characterized in that The current emotional state also includes the current emotional intensity of the target object; and the emotional state fusion of the multiple emotional states to obtain the current emotional state of the target object further includes: In a case where the first emotion category and the second emotion category are inconsistent, the current emotion category is the first emotion category, and the target object image further includes a part image of a preset part of the target object, determining the emotion intensity recognized from the part image of the preset part as the current emotion intensity; In a case where the first emotion category and the second emotion category are inconsistent and the current emotion category is the second emotion category, determining the first emotion intensity in the second emotion state as the current emotion intensity; When the first emotion category and the second emotion category are inconsistent, the current emotion category is the second emotion category, and the target object image also includes a part image of a preset part of the target object, a weighted sum of the first emotion intensity and the second emotion intensity is determined as the current emotion intensity, wherein the first emotion intensity is the emotion intensity included in the second emotional state, and the second emotion intensity is the emotion intensity identified from the part image of the preset part.

10. The method according to any one of claims 1 to 9, characterized in that The determining, according to the current emotional state and the historical emotional state, a target device operation to be executed by the target smart device further includes: In a case where it is determined according to the current emotional state and the historical emotional state that the target object is in a negative emotion and the emotional intensity of the negative emotion is reduced, determining that the device operation to be performed by the target smart device is a first device operation; In a case where it is determined according to the current emotional state and the historical emotional state that the target object is in a negative emotion and the emotional intensity of the negative emotion increases, determining that the device operation to be performed by the target smart device is a second device operation; When it is determined that the target object is in a negative emotion based on the current emotional state and the historical emotional state, and the duration of the emotional intensity of the negative emotion is greater than or equal to a preset intensity threshold and reaches a preset duration threshold, the device operation to be executed by the target smart device is determined to be a third device operation.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 10 when executed.

12. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 10 through the computer program.

Images