A health detection method and device and a storage medium

Abstract

The application provides a health detection method and device and a storage medium, relates to the technical field of computers, and aims to solve the technical problem that a general wearable device cannot comprehensively detect the health state of a to-be-detected object. The health detection method comprises the following steps: acquiring motion data, audio data and video data of the to-be-detected object; determining a motion index of the to-be-detected object according to the motion data; determining an emotion index of the to-be-detected object according to the audio data and the video data; and determining a health index of the to-be-detected object according to the motion index and the emotion index.

Description

A health detection method, device and storage medium Technical Field

[0001] This application relates to the field of computer technology, and in particular to a health detection method, device, and storage medium. Background Technology

[0002] Currently, most of the hardware devices and products on the market related to health monitoring are wearable devices that detect exercise data, such as smartwatches, wristbands, and heart rate belts. They mainly analyze the exercise status of the subject by detecting indicators such as electrocardiogram, heart rate, blood oxygen, and body temperature.

[0003] However, general-purpose wearable devices cannot comprehensively detect the health status of the subject being tested. Summary of the Invention

[0004] This application provides a health detection method, apparatus, and storage medium to solve the technical problem that general technologies cannot comprehensively detect the health status of the object to be detected.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, a health detection method is provided, comprising: acquiring the subject's movement data, audio data, and video data; determining the subject's movement index based on the movement data; determining the subject's emotional index based on the audio and video data; and determining the subject's health index based on the movement index and emotional index.

[0007] Optionally, after determining the health index of the subject to be tested based on the exercise index and the mood index, the method further includes: determining the test result of the subject to be tested based on the target index and the preset pattern curve; the target index includes at least one of the exercise index, the mood index and the health index; the preset pattern curve includes the biological trirhythm curve.

[0008] Optionally, the exercise data includes: number of steps and exercise time; the exercise index of the subject to be tested is determined based on the exercise data, including: determining the ratio between the number of steps and exercise time as the exercise intensity of the subject to be tested; and determining the exercise index based on the number of steps, exercise intensity, preset number of steps, and preset exercise intensity.

[0009] Optionally, the emotion index of the object to be detected is determined based on audio data and video data, including: inputting audio data into a speech emotion recognition model to obtain a first emotion recognition result; inputting video data into a face attribute analysis model to obtain a second emotion recognition result; and determining the emotion index based on the first emotion recognition result and the second emotion recognition result.

[0010] Optionally, the first emotion recognition result includes: at least one first emotion category and a first confidence level corresponding to each first emotion category; the second emotion recognition result includes: at least one second emotion category and a second confidence level corresponding to each second emotion category; determining the emotion index based on the first emotion recognition result and the second emotion recognition result includes: merging the first emotion recognition result and the second emotion recognition result to obtain a target emotion recognition result; the target emotion recognition result includes multiple target emotion categories and a target confidence level corresponding to each target emotion category; selecting at least one target confidence level from the multiple target confidence levels that is greater than a preset confidence level, and determining the emotion index based on the number of times the at least one target emotion category corresponding to the at least one target confidence level occurs within a preset time period.

[0011] Optionally, the health detection method may also include: sending a prompt message to the electronic device corresponding to the object to be detected when the health index is less than a preset threshold and / or the detection result is abnormal.

[0012] In a second aspect, a health detection device is provided, comprising: an acquisition unit and a processing unit; the acquisition unit is used to acquire motion data, audio data and video data of a subject to be detected; the processing unit is used to determine the motion index of the subject to be detected based on the motion data; the processing unit is also used to determine the emotion index of the subject to be detected based on the audio data and video data; the processing unit is also used to determine the health index of the subject to be detected based on the motion index and the emotion index.

[0013] Optionally, the processing unit is also used to determine the detection result of the subject to be detected based on the target index and the preset regularity curve; the target index includes at least one of the following: exercise index, mood index and health index; the preset regularity curve includes the biological trirhythm curve.

[0014] Optionally, the motion data includes: number of steps and motion time; the processing unit is specifically used to: determine the ratio between the number of steps and motion time as the motion intensity of the object to be detected;

[0015] The exercise index is determined based on the number of steps, exercise intensity, preset number of steps, and preset exercise intensity.

[0016] Optionally, the processing unit is specifically used for: inputting audio data into a speech emotion recognition model to obtain a first emotion recognition result; inputting video data into a face attribute analysis model to obtain a second emotion recognition result; and determining an emotion index based on the first emotion recognition result and the second emotion recognition result.

[0017] Optionally, the first emotion recognition result includes: at least one first emotion category and a first confidence level corresponding to each first emotion category; the second emotion recognition result includes: at least one second emotion category and a second confidence level corresponding to each second emotion category; the processing unit is specifically used to: merge the first emotion recognition result and the second emotion recognition result to obtain a target emotion recognition result; the target emotion recognition result includes multiple target emotion categories and a target confidence level corresponding to each target emotion category; select at least one target confidence level from the multiple target confidence levels that is greater than a preset confidence level, and determine the emotion index based on the number of times the at least one target emotion category corresponding to the at least one target confidence level occurs within a preset time period.

[0018] Optionally, the health detection device further includes: a sending unit; the sending unit is used to send a prompt message to the electronic device corresponding to the object to be detected when the health index is less than a preset threshold and / or the detection result is abnormal.

[0019] Thirdly, a health detection device is provided, including a memory and a processor; the memory is used to store computer execution instructions, and the processor is connected to the memory via a bus; when the health detection device is running, the processor executes the computer execution instructions stored in the memory to cause the health detection device to perform the health detection method described in the first aspect.

[0020] The health detection device can be a network device or a component of a network device, such as a chip system within the network device. This chip system supports the network device in implementing the functions involved in the first aspect and any of its possible implementations, such as acquiring, determining, and transmitting the data and / or information involved in the aforementioned health detection method. The chip system includes a chip, but may also include other discrete devices or circuit structures.

[0021] Fourthly, a computer-readable storage medium is provided, comprising computer-executable instructions that, when executed on a computer, cause the computer to perform the health detection method described in the first aspect.

[0022] Fifthly, a computer program product is also provided, which includes computer instructions that, when executed on a health detection device, cause the health detection device to perform the health detection method as described in the first aspect above.

[0023] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on a computer-readable storage medium. This computer-readable storage medium may be packaged together with the processor of the health monitoring device, or it may be packaged separately from the processor of the health monitoring device; this application does not limit this.

[0024] The descriptions of the second, third, fourth, and fifth aspects of this application can be referenced to the detailed description of the first aspect.

[0025] In the embodiments of this application, the names of the aforementioned health monitoring devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. For example, the receiving unit may also be called a receiving module, receiver, etc. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalents.

[0026] The technical solution provided in this application brings at least the following beneficial effects:

[0027] Based on any of the above aspects, this application provides a health detection method that can acquire the subject's motion data, audio data, and video data, determine the subject's motion index based on the motion data, and determine the subject's emotional index based on the audio and video data. Subsequently, the subject's health index can be determined based on the motion index and emotional index. Thus, this application can comprehensively detect the subject's health status based on the subject's motion index and emotional index, solving the technical problem that general wearable devices cannot comprehensively detect the health status of a subject.

[0028] The beneficial effects of the first, second, third, fourth, and fifth aspects of this application can all be referred to in the analysis of the above-mentioned beneficial effects, and will not be repeated here. Attached Figure Description

[0029] Figure 1 is a schematic diagram of an application scenario of a health detection method provided in an embodiment of this application;

[0030] Figure 2 is a schematic diagram of the hardware structure of a health detection device provided in an embodiment of this application;

[0031] Figure 3 is a schematic diagram of the hardware structure of a health detection device provided in an embodiment of this application;

[0032] Figure 4 is a schematic flowchart of a health detection method provided in an embodiment of this application;

[0033] Figure 5 is a schematic flowchart of a health detection method provided in an embodiment of this application;

[0034] Figure 6 is a schematic flowchart of a health detection method provided in an embodiment of this application;

[0035] Figure 7 is a schematic flowchart of a health detection method provided in an embodiment of this application;

[0036] Figure 8 is a schematic flowchart of a health detection method provided in an embodiment of this application;

[0037] Figure 9 is a structural schematic diagram of a health detection device provided in an embodiment of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0040] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0041] As described in the background section, most of the hardware devices and products on the market related to health monitoring are wearable devices that detect exercise data, such as smartwatches, wristbands, and heart rate belts. They mainly analyze the exercise status of the subject by detecting indicators such as electrocardiogram, heart rate, blood oxygen, and body temperature.

[0042] However, general-purpose wearable devices cannot comprehensively detect the health status of the subject being tested.

[0043] Studies have shown that emotional states can be causally linked to physical health through various biological, psychological, and social pathways. Furthermore, physical strength, emotions, and intelligence exhibit cyclical patterns (biological rhythms). From birth to death, each person experiences a cyclical fluctuation of physical strength (23 days), emotions (28 days), and intelligence (33 days). Therefore, the detection and prediction of emotions are of great significance for guiding human work, study, and physical exercise.

[0044] Currently, there are various methods and devices for emotion detection, including specialized medical equipment that extracts and analyzes features by collecting physiological signals such as electroencephalograms (EEGs), electrocardiograms (ECGs), and electrodermal conductance. In recent years, with the development of artificial intelligence (AI) technology, there are increasingly more methods for obtaining emotion data by analyzing and processing human voice and facial images using AI technology.

[0045] With the rapid development of smart homes, smart terminal devices in homes are becoming increasingly common, such as smart speakers, cameras, and smartwatches. The collection and analysis of emotion and motion data based on smart devices is becoming faster and can cover a wider range of people.

[0046] Therefore, how to comprehensively detect the health status of the subject based on their motion data and emotional state is a technical problem that urgently needs to be solved.

[0047] To address the aforementioned issues, this application provides a health detection method that can acquire the subject's motion data, audio data, and video data. Based on the motion data, it determines the subject's motion index, and based on the audio and video data, it determines the subject's emotional index. Subsequently, based on the motion index and emotional index, the subject's overall health index can be determined. Thus, this application can comprehensively detect the subject's health status based on their motion and emotional indices, solving the technical problem that general wearable devices cannot comprehensively detect the health status of a subject.

[0048] This health monitoring method is applicable to a health monitoring system. Figure 1 shows one structure of the health monitoring system. As shown in Figure 1, the health monitoring system includes: an electronic device 101, a motion data acquisition device 102, and an audio and video data acquisition device 103.

[0049] The electronic device 101 is connected to the motion data acquisition device 102 and the audio and video data acquisition device 103 for communication.

[0050] In practical applications, electronic device 101 can connect to any number of motion data acquisition devices 102 and audio and video data acquisition devices 103. For ease of understanding, Figure 1 illustrates an example of an electronic device 101 connected to a motion data acquisition device 102 and an audio and video data acquisition device 103.

[0051] In this embodiment of the application, the motion data acquisition device 102 is used to provide motion data to the electronic device 101 so that the electronic device 101 can determine the motion index of the object to be detected based on the motion data sent by the motion data acquisition device 102.

[0052] Optionally, the motion data acquisition device 102 can be a wearable device for detecting motion data, such as a smartwatch, wristband, or heart rate monitor.

[0053] The audio and video data acquisition device 103 is used to provide audio and video data to the electronic device 101 so that the electronic device 101 can determine the emotion index of the subject to be detected based on the audio and video data sent by the audio and video data acquisition device 102.

[0054] Optionally, the audio and video data acquisition device 103 can be a smart terminal device for a smart home, such as a smart speaker, camera, or smartwatch.

[0055] The electronic device 101 can determine the motion index and the emotional index of the subject to be tested, and then determine the health index of the subject to be tested based on the motion index and the emotional index, thereby achieving a comprehensive detection of the health status of the subject to be tested.

[0056] Optionally, the physical device of electronic device 101 can be a server, a terminal, or other types of electronic devices, and this application embodiment does not limit this.

[0057] Optionally, the aforementioned terminal may be a device that provides voice and / or data connectivity to a user, a handheld device with wireless connectivity, or other processing device connected to a wireless modem. The wireless terminal may communicate with one or more core networks via a radio access network (RAN). The wireless terminal may be a mobile terminal, such as a mobile phone (or "cellular" phone) and a computer with a mobile terminal, or a portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile device that exchanges voice and / or data with the radio access network, such as a mobile phone, tablet computer, laptop computer, netbook, or personal digital assistant (PDA).

[0058] Optionally, the server mentioned above can be one of the servers in a server cluster (composed of multiple servers), a chip in the server, a system-on-a-chip in the server, or a virtual machine (VM) deployed on a physical machine. This application embodiment does not limit this.

[0059] Optionally, when the physical device of electronic device 101 and the physical device of motion data acquisition device 102 are the same (e.g., both are wearable devices), electronic device 101 and physical device and motion data acquisition device 102 can be two independently set devices, or they can be integrated into the same device.

[0060] It is easy to understand that when electronic device 101, physical device, and motion data acquisition device 102 are integrated into the same device, the communication method between electronic device 101, physical device, and motion data acquisition device 102 is the same as the communication between internal modules of the device. In this case, the communication process between the two is the same as the communication process between electronic device 101, physical device, and motion data acquisition device 102 when they are independent of each other.

[0061] For ease of understanding, this application uses the example of electronic device 101 and physical device and motion data acquisition device 102 being independent of each other.

[0062] Optionally, when the physical device of electronic device 101 and the physical device of audio and video data acquisition device 103 are the same (e.g., both are smart cameras), electronic device 101 and physical device and audio and video data acquisition device 103 can be two independently set devices, or they can be integrated into the same device.

[0063] It is easy to understand that when electronic device 101, physical device, and audio and video data acquisition device 103 are integrated into the same device, the communication method between electronic device 101, physical device, and audio and video data acquisition device 103 is the same as the communication between internal modules of the device. In this case, the communication process between the two is the same as the communication process between electronic device 101, physical device, and audio and video data acquisition device 103 when they are independent of each other.

[0064] For ease of understanding, this application uses the example of electronic device 101 and physical device and audio and video data acquisition device 103 being independent of each other.

[0065] Optionally, the motion data acquisition device 102 and the audio and video data acquisition device 103 can also be integrated into a single device. For ease of understanding, this application will illustrate the example of the motion data acquisition device 102 and the audio and video data acquisition device 103 operating independently.

[0066] The basic hardware structure of electronic device 101 includes the components included in the health detection device shown in Figure 2 or Figure 3. The hardware structure of electronic device 101 will be described below using the health detection device shown in Figures 2 and 3 as an example.

[0067] Figure 2 shows a schematic diagram of the hardware structure of a health monitoring device provided in an embodiment of this application. The health monitoring device includes a processor 21, a memory 22, a communication interface 23, and a bus 24. The processor 21, the memory 22, and the communication interface 23 are connected via the bus 24.

[0068] Processor 21 is the control center of the health monitoring device. It can be a single processor or a collective term for multiple processing elements. For example, processor 21 can be a general-purpose central processing unit (CPU) or other general-purpose processors. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0069] As one embodiment, processor 21 may include one or more CPUs, such as CPU 0 and CPU 1 shown in FIG2.

[0070] The memory 22 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0071] In one possible implementation, the memory 22 can exist independently of the processor 21. The memory 22 can be connected to the processor 21 via a bus 24 and is used to store instructions or program code. When the processor 21 calls and executes the instructions or program code stored in the memory 22, it can implement the health detection method provided in the following embodiments of this application.

[0072] In this embodiment, the software programs stored in the memory 22 of the electronic device 101 are different, so the functions implemented by the electronic device 101 are different. The functions performed by each device will be described with reference to the following flowchart.

[0073] In another possible implementation, the memory 22 can also be integrated with the processor 21.

[0074] Communication interface 23 is used for the health monitoring device to connect with other devices via a communication network, such as Ethernet, wireless access network, or wireless local area network (WLAN). Communication interface 23 may include a receiving unit for receiving data and a sending unit for sending data.

[0075] Bus 24 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in Figure 2, but this does not indicate that there is only one bus or one type of bus.

[0076] Figure 3 illustrates another hardware structure of the health monitoring device in an embodiment of this application. As shown in Figure 3, the health monitoring device may include a processor 31 and a communication interface 32. The processor 31 is coupled to the communication interface 32.

[0077] The functions of processor 31 can be referred to in the description of processor 21 above. In addition, processor 31 also has a storage function, and can perform the functions of memory 22 mentioned above.

[0078] The communication interface 32 is used to provide data to the processor 31. This communication interface 32 can be an internal interface of the health monitoring device, or it can be an external interface of the health monitoring device (equivalent to communication interface 23).

[0079] It should be noted that the structure shown in Figure 2 (or Figure 3) does not constitute a limitation on the health detection device. In addition to the components shown in Figure 2 (or Figure 3), the health detection device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0080] The health detection method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0081] The health detection method provided in this application embodiment is applied to the electronic device 101 in the health detection system shown in FIG1. ​​As shown in FIG4, the health detection method provided in this application embodiment includes:

[0082] S401. The electronic device acquires motion data, audio data, and video data of the object to be detected.

[0083] Referring to Figure 1, the motion data acquisition device 102 can acquire the motion data of the object to be detected and send the motion data of the object to be detected to the electronic device.

[0084] For example, a smart terminal device with a motion sensor (i.e., motion data acquisition device 102) can realize a step counting function, thereby determining the step counting data as motion data.

[0085] Smart terminal devices equipped with motion sensors can include mobile assistants, smartwatches, and other smart terminals. These devices can periodically report accumulated steps and location data to a cloud server. The cloud server can then report the acquired steps and location data back to the electronic device.

[0086] Referring to Figure 1, the audio and video data acquisition device 103 can acquire audio and video data of the object to be detected and send the audio and video data of the object to be detected to the electronic device.

[0087] For example, a network HD camera with recording capabilities (i.e., audio and video data acquisition device 103) can record video at 720P or 1080P resolution and push the audio and video streams to a streaming media server cluster in real time using a streaming media protocol. The streaming media server cluster can then report the acquired audio and video streams to electronic devices.

[0088] After acquiring the raw data (including motion data, audio data, and video data) collected by the acquisition device, since the raw data collected by the acquisition device may include noise data, the electronic device can preprocess the acquired raw data to obtain motion data, audio data, and video data.

[0089] Optionally, when the subject is wearing a wearable motion data acquisition device, their user information can be entered into the device. In this case, the subject's user information can be linked to the device code of the motion data acquisition device. Subsequently, the electronic device can determine the subject's motion data based on the device code of the acquisition device.

[0090] Optionally, the motion data acquired by the motion data acquisition device may include the coordinate information of the object to be detected. In this case, after obtaining the coordinate information of the object to be detected, the electronic device can unify and standardize various data from different coordinate systems into data in the GCJ02 coordinate system.

[0091] Among them, GCJ02 is the coordinate system of the geographic information system formulated by the Bureau of Surveying and Mapping.

[0092] Optionally, the electronic device may also be equipped with an audio / video stream frame extraction and filtering application module. This module is used for audio / video separation, audio / video slicing, and keyframe extraction.

[0093] Specifically, the audio and video stream frame extraction and filtering application module can perform preliminary filtering based on audio and video filtering algorithms to remove meaningless data, such as still images and audio clips.

[0094] Next, the audio and video stream frame extraction and filtering application module can save the audio and video segments that need further analysis and processing to the cloud disk, and at the same time abstract them into audio message events and video message events and put them into the message queue, waiting for the consumption process to further analyze and process them.

[0095] S402. The electronic device determines the motion index of the object to be detected based on the motion data.

[0096] Specifically, after acquiring motion data, electronic devices can determine the motion index of the object to be detected based on the motion data.

[0097] In one feasible approach, the electronic device can filter out the valid continuous movement time and number of steps based on the movement step count and location update messages reported by the motion data acquisition device, and calculate the motion index based on the valid motion data.

[0098] In another possible approach, electronic devices can directly determine the number of steps reported by the motion data acquisition device as the motion index of the object to be detected.

[0099] Optionally, the electronic device can also determine the motion index of the object to be detected based on motion data using other algorithms, which is not limited in this embodiment.

[0100] S403. The electronic device determines the emotional index of the subject to be detected based on audio and video data.

[0101] In one feasible approach, electronic devices can input audio and video data collected by audio and video data acquisition devices into a pre-trained fusion model to obtain the emotion index of the object to be detected.

[0102] In another possible implementation, the electronic device can input audio data collected by audio and video data acquisition devices into a pre-trained speech emotion recognition model to obtain a first emotion index of the subject to be detected. Next, the electronic device can input video data collected by the audio and video data acquisition devices into a pre-trained facial attribute analysis model to obtain a second emotion index of the subject to be detected. Finally, the electronic device can combine the first and second emotion indices to obtain the final emotion index of the subject to be detected.

[0103] Optionally, the electronic device can also determine the emotion index of the object to be detected based on audio and video data using other models or algorithms, which is not limited in this embodiment.

[0104] S404. Electronic devices determine the health index of the subject being tested based on the activity index and mood index.

[0105] Specifically, after determining the exercise index and mood index, electronic devices can comprehensively determine the health index of the subject being tested based on these indices.

[0106] In one feasible approach, the health index is calculated using a combination of a mood index and a physical activity index, each worth 50 points, for a total of 100 points. The health index, mood index, and physical activity index satisfy the following formula:

[0107] Health Index = 50 * Exercise Index / 5 + 50 * Mood Index / 5.

[0108] The health index can range from [20, 100].

[0109] As described above, this application provides a health detection method that can acquire the subject's motion data, audio data, and video data, determine the subject's motion index based on the motion data, and determine the subject's emotional index based on the audio and video data. Subsequently, the subject's health index can be determined based on the motion index and emotional index. Thus, this application can comprehensively detect the subject's health status based on the subject's motion index and emotional index, solving the technical problem that general wearable devices cannot comprehensively detect the health status of the subject.

[0110] In some embodiments, referring to Figure 4 and as shown in Figure 5, after the electronic device determines the health index of the subject to be detected based on the motion index and the mood index, it further includes:

[0111] S501. The electronic device determines the detection result of the object to be detected based on the target index and the preset law curve.

[0112] The target indices include at least one of the following: exercise index, mood index, and health index. The pre-defined rhythm curves include the biological three-rhythm curve.

[0113] Specifically, electronic devices can calculate the emotional and physical values ​​and the stage of the human body's three-rhythm curve based on the subject's birth date. Combined with the emotional index, exercise index, and health index obtained from the data analysis and processing of the electronic devices, the test results for the subject can be determined.

[0114] Optionally, the electronic device can also summarize indicators such as exercise index, mood index, and health index according to date and the dimension of the object to be detected, and store them in a data warehouse. Then, the electronic device can display and analyze the data in charts and graphs according to the dimensions of day, week, month, and year, thereby determining the detection results of the object to be detected.

[0115] As can be seen from the above, this application provides a health detection method that can determine the health index of the subject to be tested based on the exercise index and the emotion index, and then determine the test result of the subject to be tested based on the target index and the preset pattern curve, thereby comprehensively determining the test result of the subject to be tested.

[0116] In some embodiments, referring to FIG5 and FIG6, the above-mentioned motion data includes: the number of steps and the motion time. In this case, the method for the electronic device to determine the motion index of the object to be detected based on the motion data in S402 specifically includes:

[0117] S601. The electronic device determines the motion intensity of the object to be detected by the ratio between the number of steps and the motion time.

[0118] S602. The electronic device determines the exercise index based on the number of steps, exercise intensity, preset number of steps, and preset exercise intensity.

[0119] Specifically, electronic devices can set preset steps and preset exercise intensity for continuous exercise for different age groups.

[0120] For example, the electronic device can set the preset number of steps for the subject to be tested in the 30-40 age group to be 6000, and the preset exercise intensity to be 235.

[0121] Next, the electronic device can determine the motion intensity of the object being detected by the ratio between the number of steps and the motion time.

[0122] Next, the electronic device can determine the exercise index based on the number of steps, exercise intensity, preset number of steps, and preset exercise intensity.

[0123] For example, when the absolute value of the difference between the number of steps of the object to be detected and the preset number of steps is less than or equal to 100 steps, and the absolute value of the difference between the motion intensity of the object to be detected and the preset motion intensity is less than or equal to 10, the electronic device can determine the motion index of the object to be detected as 5 points.

[0124] When the absolute value of the difference between the number of steps of the object to be detected and the preset number of steps is greater than 100 steps and less than or equal to 1000 steps, and the absolute value of the difference between the motion intensity of the object to be detected and the preset motion intensity is less than or equal to 15, the electronic device can determine the motion index of the object to be detected as 4 points.

[0125] When the absolute value of the difference between the number of steps of the object to be detected and the preset number of steps is greater than 1000 steps and less than or equal to 2000 steps, and the absolute value of the difference between the motion intensity of the object to be detected and the preset motion intensity is less than or equal to 100, the electronic device can determine the motion index of the object to be detected as 3 points.

[0126] When the absolute value of the difference between the number of steps the object to be detected takes and the preset number of steps is greater than 2000, the electronic device can determine the motion index of the object to be detected as 2 points.

[0127] When the motion intensity of the object to be detected is less than or equal to 200, the electronic device can determine the motion index of the object to be detected as 0 points.

[0128] Optionally, in other cases besides those mentioned above, the electronic device may determine the motion index of the object to be detected as 1 point.

[0129] As can be seen from the above embodiments, a specific implementation method for determining the motion index of the subject to be tested based on motion data is provided, so as to facilitate the comprehensive detection of the health status of the subject to be tested based on the motion index and emotional index of the subject to be tested, thus solving the technical problem that general wearable devices cannot comprehensively detect the health status of the subject to be tested.

[0130] In some embodiments, referring to FIG6 and as shown in FIG7, the method for the electronic device to determine the emotion index of the object to be detected based on audio data and video data in S403 specifically includes:

[0131] S701, The electronic device inputs audio data into the voice emotion recognition model to obtain the first emotion recognition result.

[0132] S702. The electronic device inputs video data into the facial attribute analysis model to obtain the second emotion recognition result.

[0133] Optionally, an AI analysis module can be deployed in the electronic device. After receiving audio and video data, the AI ​​analysis module can dynamically schedule threads to perform audio AI analysis and image AI analysis based on message attributes in a message-driven manner.

[0134] If the message attributes contain an audio file address, the AI ​​analysis module in the electronic device calls the voice emotion recognition model API to perform voiceprint emotion recognition and semantic emotion recognition to obtain the first emotion recognition result.

[0135] If the message attributes contain a video file address, the AI ​​analysis module in the electronic device first uses FFMPEG to obtain keyframe images from the video data, and then calls the facial attribute analysis model API to recognize facial emotions in order to obtain a second emotion recognition result.

[0136] S703: The electronic device determines the emotion index based on the first emotion recognition result and the second emotion recognition result.

[0137] In some embodiments, the first emotion recognition result includes: at least one first emotion category and a first confidence level corresponding to each first emotion category. The second emotion recognition result includes: at least one second emotion category and a second confidence level corresponding to each second emotion category.

[0138] At least one primary emotion category may include anger, fear, happiness, sadness, surprise, and the other six emotions. At least one secondary emotion category may also include anger, fear, happiness, sadness, surprise, and the other six emotions.

[0139] Optionally, the electronic device can determine the emotion index by summing the first confidence level corresponding to each first emotion category in the first emotion recognition result and the second confidence level corresponding to each second emotion category in the second emotion recognition result.

[0140] Optionally, the electronic device can also merge the first emotion recognition result and the second emotion recognition result to obtain the target emotion recognition result, and select at least one target confidence degree from the multiple target confidence degrees included in the target emotion recognition result, wherein the target confidence degree is greater than the preset confidence degree, and determine the emotion index based on the number of times the at least one target emotion recognition result corresponding to the at least one target confidence degree occurs within a preset time period.

[0141] In some embodiments, the method by which an electronic device determines an emotion index based on a first emotion recognition result and a second emotion recognition result specifically includes:

[0142] S703-1 The electronic device merges the first emotion recognition result and the second emotion recognition result to obtain the target emotion recognition result.

[0143] The target emotion recognition results include multiple target emotion categories and the target confidence score corresponding to each target emotion category.

[0144] S703-2. The electronic device selects at least one target confidence level from multiple target confidence levels that is greater than a preset confidence level, and determines the emotion index based on the number of times at least one target emotion category corresponding to at least one target confidence level occurs within a preset time period.

[0145] Specifically, the electronic device can merge the first confidence level corresponding to each first emotion category and the second confidence level corresponding to each second emotion category to obtain a list of emotions with high confidence levels corresponding to the audio and video data.

[0146] For example, an AI analysis module in an electronic device can divide a day into multiple time periods. Then, the AI ​​analysis module in the electronic device can count the number of times each emotion category occurs in multiple time periods, and determine the emotion index of the subject being tested based on the number of occurrences.

[0147] For example, when the maximum score of the emotion index of the subject to be detected is 5, the electronic device can determine the target value as 2 * number of happy emotions - 1 * number of sad emotions - 1 * number of angry emotions.

[0148] When the target value is greater than or equal to 5, the electronic device determines the emotional index of the subject to be tested to be 5 points.

[0149] When the target value is greater than or equal to 0 and less than 5, the electronic device determines the emotional index of the subject to be detected to be 4 points.

[0150] When the target value is greater than or equal to -3 and less than 0, the electronic device determines the emotional index of the subject to be tested to be 3 points.

[0151] When the target value is less than -3, the electronic device determines the emotional index of the subject to be tested to be 2 points.

[0152] Optionally, the electronic device may also update the calculation formula corresponding to the above target value according to other emotion categories (such as fear, surprise, etc.), which is not limited in this embodiment of the application.

[0153] Optionally, if the audio and video data are processed to obtain an emotion list, the electronic device can determine the identity information of the object to be detected based on the key video frame images carried in the video data using a face comparison model.

[0154] If the face comparison fails, the person to be detected is considered a stranger (i.e., an unauthorized person). The person to be detected can use a mobile APP to annotate their face and confirm their identity.

[0155] Subsequently, electronic devices can store the organized list of emotions and identity information in a database.

[0156] As can be seen from the above embodiments, a specific implementation method for determining the emotional index of the subject under test based on audio data and video data is provided, so as to facilitate the comprehensive detection of the health status of the subject under test based on the subject's movement index and emotional index, thus solving the technical problem that general wearable devices cannot comprehensively detect the health status of the subject under test.

[0157] In some embodiments, referring to FIG7 and as shown in FIG8, the health detection method further includes:

[0158] S801. When the health index is less than the preset threshold and / or the detection result is abnormal, the electronic device sends a prompt message to the electronic device corresponding to the object to be detected.

[0159] Optionally, the electronic device corresponding to the object to be detected can be a wearable device worn by the object to be detected, a terminal held by the object to be detected, or other similar devices.

[0160] Specifically, when the health index is below a preset threshold, and / or the test result is abnormal, it indicates that the health status of the person being tested may be abnormal. In this case, the electronic device can send a prompt message to the electronic device corresponding to the person being tested, reminding the person to arrange activities such as studying, working, or exercising.

[0161] Optionally, the electronic device can also generate a health report based on the health index level of the object to be tested over a recent period of time, and send the health report to the electronic device corresponding to the object to be tested.

[0162] For example, an electronic device determines that the target index of the subject has been consistently low for the past three months. If the low score is due to emotional reasons, a prompt message is sent to the subject's corresponding electronic device, reminding the subject to pay attention to their emotions. If the low score is due to lack of exercise, a prompt message is sent to the subject's corresponding electronic device, reminding the subject to increase their exercise.

[0163] Optionally, the electronic device can also send alert messages to the electronic devices of related objects of the object to be detected, according to the alert strategy and channels configured for the object to be detected.

[0164] As can be seen from the above, this application provides a health detection method that can send a prompt message to the electronic device corresponding to the object being detected when the health index is less than a preset threshold and / or the detection result is abnormal, so that the object being detected can promptly view the corresponding health index and / or detection result, thus enriching the methods of health detection.

[0165] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0166] This application embodiment can divide the health monitoring device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0167] Figure 9 is a schematic diagram of a health detection device provided in an embodiment of this application. This health detection device can be used to perform the health detection methods shown in Figures 4-8. The health detection device shown in Figure 9 includes: an acquisition unit 901 and a processing unit 902;

[0168] The acquisition unit 901 is used to acquire motion data, audio data, and video data of the object to be detected.

[0169] Processing unit 902 is used to determine the motion index of the object to be detected based on motion data;

[0170] Processing unit 902 is also used to determine the emotion index of the object to be detected based on audio data and video data;

[0171] The processing unit 902 is also used to determine the health index of the subject to be tested based on the exercise index and the mood index.

[0172] Optionally, the processing unit 902 is also used to determine the detection result of the object to be detected based on the target index and the preset regularity curve; the target index includes at least one of the following: exercise index, mood index and health index; the preset regularity curve includes the biological trirhythm curve.

[0173] Optional, exercise data includes: number of steps and exercise time;

[0174] Processing unit 902 is specifically used for:

[0175] The ratio between the number of steps taken and the duration of exercise is determined as the exercise intensity of the subject being tested.

[0176] The exercise index is determined based on the number of steps, exercise intensity, preset number of steps, and preset exercise intensity.

[0177] Optionally, the processing unit 902 is specifically used for:

[0178] The audio data is input into the speech emotion recognition model to obtain the first emotion recognition result;

[0179] The video data is input into the facial attribute analysis model to obtain the second emotion recognition result;

[0180] An emotion index is determined based on the results of the first emotion recognition and the second emotion recognition.

[0181] Optionally, the first emotion recognition result includes: at least one first emotion category and a first confidence level corresponding to each first emotion category;

[0182] The second emotion recognition result includes: at least one second emotion category and a second confidence level corresponding to each second emotion category;

[0183] Processing unit 902 is specifically used for:

[0184] The first emotion recognition result and the second emotion recognition result are combined to obtain the target emotion recognition result; the target emotion recognition result includes multiple target emotion categories and the target confidence score corresponding to each target emotion category;

[0185] Select at least one target confidence level from multiple target confidence levels that is greater than a preset confidence level, and determine the sentiment index based on the number of times at least one target sentiment category corresponding to at least one target confidence level occurs within a preset time period.

[0186] Optionally, the health monitoring device may also include: a transmitting unit 903;

[0187] The sending unit 903 is used to send a prompt message to the electronic device corresponding to the object to be detected when the health index is less than a preset threshold and / or the detection result is abnormal.

[0188] This application also provides a computer-readable storage medium, which includes computer-executable instructions that, when executed on a computer, cause the computer to perform the health detection method provided in the above embodiments.

[0189] This application also provides a computer program that can be directly loaded into a memory and contains software code. After being loaded and executed by a computer, the computer program can implement the health detection method provided in the above embodiments.

[0190] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this application can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer.

[0191] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and other division methods may exist in actual implementation. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate; components shown as units may be one physical unit or multiple physical units, i.e., they may be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0194] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A health detection method, characterized in that, include: Acquire motion data, audio data, and video data of the object to be detected; The motion index of the object to be detected is determined based on the motion data; Determining the emotion index of the object to be detected based on the audio data and the video data includes: using a message-driven approach, if the message attributes contain an audio file address, then calling the speech emotion recognition model API to perform voiceprint emotion recognition and semantic emotion recognition to obtain a first emotion recognition result; if the message attributes contain a video file address, then using FFMPEG to obtain keyframe images from the video data and calling the face attribute analysis model API. Facial emotion recognition is performed to obtain a second emotion recognition result; the first emotion recognition result includes: at least one first emotion category and a first confidence level corresponding to each first emotion category; the second emotion recognition result includes: at least one second emotion category and a second confidence level corresponding to each second emotion category; the first emotion category and the second emotion category include: anger, fear, happiness, sadness, surprise, and others; the first emotion recognition result and the second emotion recognition result are merged to obtain a target emotion recognition result; the target emotion recognition result includes multiple target emotion categories and a target confidence level corresponding to each target emotion category; at least one target confidence level greater than a preset confidence level is selected from multiple target confidence levels, and the emotion index is determined based on the number of times the at least one target emotion category corresponding to the at least one target confidence level occurs within a preset time period; the health index of the subject to be tested is determined based on the exercise index and the emotion index; the health index, the emotion index, and the exercise index satisfy the following formula: health index = 50 * exercise index / 5 + 50 * emotion index / 5.

2. The health detection method according to claim 1, characterized in that, After determining the health index of the subject to be tested based on the exercise index and the emotion index, the method further includes: determining the test result of the subject to be tested based on a target index and a preset pattern curve; the target index includes at least one of the exercise index, the emotion index, and the health index; the preset pattern curve includes a biological trirhythm curve.

3. The health detection method according to claim 1, characterized in that, The motion data includes: number of steps and motion time; determining the motion index of the object to be detected based on the motion data includes: determining the ratio between the number of steps and the motion time as the motion intensity of the object to be detected; and determining the motion index based on the number of steps, the motion intensity, a preset number of steps, and a preset motion intensity.

4. The health detection method according to claim 2, characterized in that, Also includes: When the health index is less than a preset threshold, and / or the detection result is abnormal, a prompt message is sent to the electronic device corresponding to the object to be detected.

5. A health monitoring device, characterized in that, include: Acquisition unit and processing unit; The acquisition unit is used to acquire motion data, audio data, and video data of the object to be detected; The processing unit is used to determine the motion index of the object to be detected based on the motion data; The processing unit is further configured to determine the emotion index of the object to be detected based on the audio data and the video data, including: in a message-driven manner, if the message attribute contains an audio file address, calling the speech emotion recognition model API to perform voiceprint emotion recognition and semantic emotion recognition to obtain a first emotion recognition result; if the message attribute contains a video file address, using FFMPEG to obtain keyframe images from the video data and calling the face attribute analysis model API. Facial emotion recognition is performed to obtain a second emotion recognition result; the first emotion recognition result includes: at least one first emotion category and a first confidence level corresponding to each first emotion category; the second emotion recognition result includes: at least one second emotion category and a second confidence level corresponding to each second emotion category; the first emotion category and the second emotion category include: anger, fear, happiness, sadness, surprise, and others; the first emotion recognition result and the second emotion recognition result are merged to obtain a target emotion recognition result; the target emotion recognition result includes multiple target emotion categories and a target confidence level corresponding to each target emotion category; at least one target confidence level greater than a preset confidence level is selected from multiple target confidence levels, and the emotion index is determined based on the number of times the at least one target emotion category corresponding to the at least one target confidence level occurs within a preset time period; the processing unit is further configured to determine the health index of the subject to be detected based on the movement index and the emotion index; the health index, the emotion index, and the movement index satisfy the following formula: health index = 50 * movement index / 5 + 50 * emotion index / 5.

6. The health monitoring device according to claim 5, characterized in that, The processing unit is further configured to determine the detection result of the object to be detected based on the target index and the preset regularity curve; the target index includes at least one of the exercise index, the emotion index and the health index; the preset regularity curve includes the biological trirhythm curve.

7. The health monitoring device according to claim 6, characterized in that, The motion data includes: number of steps and motion time; the processing unit is specifically used to: determine the ratio between the number of steps and the motion time as the motion intensity of the object to be detected; and determine the motion index based on the number of steps, the motion intensity, a preset number of steps, and a preset motion intensity.

8. The health monitoring device according to claim 7, characterized in that, Also includes: Transmitting unit; The sending unit is used to send a prompt message to the electronic device corresponding to the object to be detected when the health index is less than a preset threshold and / or the detection result is abnormal.

9. A health monitoring device, characterized in that, It includes a memory and a processor; the memory is used to store computer execution instructions, and the processor is connected to the memory via a bus; when the health detection device is running, the processor executes the computer execution instructions stored in the memory to cause the health detection device to perform the health detection method as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer-executable instructions that, when executed on a computer, cause the computer to perform the health detection method as described in any one of claims 1-4.

Images