Multi-element control method and device for blue light treatment equipment, electronic equipment and storage medium

Abstract

The invention relates to a multi-element control method and device for blue light treatment equipment, electronic equipment and a storage medium. The method comprises the following steps: acquiring video data and audio data of a first object at a first moment, preprocessing the video data and the audio data, extracting corresponding key frames in the video data, and drawing a spectrogram corresponding to the audio data. And calling a YOLO algorithm to detect the five sense organs and the eyeshade of the first object according to the key frame, obtaining the positions and sizes of the five sense organs and the eyeshade of the first object, and judging that the eyeshade falls off when the eyes of the first object are identified. And according to the intersection degree of the relative position relationship between the eye patch and the five sense organs of the first object, obtaining a first probability that the eye patch falls off, and calling a CNN-SVM algorithm model to carry out abnormal sound detection on the spectrogram to obtain a second probability that the abnormal sound occurs. And when the first probability exceeds a first threshold value or the second probability exceeds a second threshold value, sending a first instruction to the blue light treatment equipment so as to control the blue light treatment equipment to stop running.

Description

Technical Field

[0001] The present invention relates to the field of medical auxiliary technology, and in particular to a multi-element control method, device, electronic device and storage medium for blue light therapy equipment. Background Art

[0002] Neonatal jaundice is one of the common symptoms of the neonatal period and can be divided into physiological jaundice and pathological jaundice. On the one hand, jaundice can cause abnormal coagulation mechanism, which may cause gastrointestinal bleeding in severe cases, and even cause hemorrhagic shock and endanger life; on the other hand, it can also cause biliary cirrhosis, further causing liver dysfunction and even liver failure. When the bilirubin in the blood reaches a certain level, it is easy to pass through the blood-brain barrier and enter the skull. Bilirubin is toxic and can easily cause cerebral palsy, intellectual disability, hearing loss, eye movement disorders, etc., causing irreversible damage to the brain and nervous system.

[0003] Doctors will recommend blue light therapy to children according to the degree of jaundice, because blue light can destroy bilirubin and promote its excretion, thereby treating jaundice. In addition, blue light has high energy, and long-term direct exposure to the eyes may cause damage to the retina, so it is necessary to wear an eye mask when using blue light, which can block blue light and other strong light.

[0004] However, during blue light therapy, the eye mask may fall off due to improper wearing, the baby turning over, struggling, etc. Therefore, it is necessary to accompany and care day and night to prevent the eye mask from falling off. However, the treatment of neonatal jaundice often takes 3-5 days, and the treatment time is longer for severe cases. Manual care is labor-intensive and prone to errors, especially for newborns. Parents are mostly tired from taking care of newborns and mothers, and they cannot take care of their children after accidentally falling asleep. In centralized treatment scenarios, medical staff are distracted by taking care of multiple babies at the same time, which may lead to omissions. Therefore, an automatic and intelligent way is needed to automatically monitor the problem of eye masks falling off in children during blue light therapy. In addition, blue light boxes are mostly closed, and the internal situation is not easy to see. When the baby has abnormal breathing, crying, and other abnormal conditions, it may be neglected. Therefore, it is necessary to intelligently detect the abnormal state of the baby during blue light therapy.

[0005] At present, the existing intelligent method for detecting abnormal conditions of infants during blue light therapy is mainly based on target detection in image recognition to identify whether the eye mask has fallen off. However, the judgment criteria based on this method are relatively simple, and it is difficult to identify various other abnormal conditions when there is no obvious abnormality in the image. Because accidental errors are more likely to occur during the detection process, the detection accuracy needs to be further improved. Summary of the invention

[0006] Based on this, it is necessary to provide a multi-element control method, device, electronic device and storage medium for blue light therapy equipment that can improve the accuracy of abnormality detection in patients during blue light therapy in order to address the above technical problems.

[0007] The present invention provides a multivariate control method for a blue light therapy device, the method comprising: Acquire video data and audio data of a first object at a first moment, and preprocess the video data and audio data to extract corresponding key frames in the video data, and draw a spectrogram corresponding to the audio data; calling the YOLO algorithm to detect the facial features and the eye mask of the first object according to the key frame, so as to respectively obtain the positions and sizes of the facial features and the eye mask of the first object, and determining that the eye mask of the first object is removed when the eyes of the first object are identified; Obtaining a first probability of the eye mask falling off according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first object, and calling the CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring; When the first probability exceeds a first threshold or the second probability exceeds a second threshold, a first instruction is sent to the blue light therapy device to control the blue light therapy device to stop operating.

[0008] In one embodiment, the step of obtaining video data and audio data of a first object at a first moment, preprocessing the video data and audio data to extract corresponding key frames in the video data, and drawing a spectrogram corresponding to the audio data includes: Extracting a plurality of key frames from the video data, and performing parameter adjustment and image correction on the key frames to standardize the key frames into image data in a fixed format; The audio data is framed, windowed and short-time Fourier transformed to convert time domain information in the audio data into frequency domain information, and the frequency domain information is spliced ​​in time sequence to draw the spectrogram.

[0009] In one embodiment, calling the YOLO algorithm to detect the facial features and the eye mask of the first object according to the key frame to respectively obtain the positions and sizes of the facial features and the eye mask of the first object, and determining that the eye mask of the first object is removed when the eyes of the first object are identified, includes: Calling the YOLO algorithm to perform facial features detection on the first object in the image data to obtain the positions and sizes of the facial features of the first object, and determining whether the eyes of the first object are detected based on the positions and sizes of the facial features; When the YOLO algorithm detects the eyes of the first object, it is determined that the eye mask of the first object has fallen off; otherwise, it is determined that the eye mask of the first object has not fallen off.

[0010] In one embodiment, the first probability of the eye mask falling off is obtained according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first object, and the CNN-SVM algorithm model is called to perform abnormal sound detection on the spectrogram to obtain the second probability of the abnormal sound occurring, including: Calling the YOLO algorithm to detect the position and size of the eye mask, so as to obtain the relative position relationship between the eye mask and the facial features of the first object in combination with the position and size of the facial features of the first object; A first probability of the eye mask falling off is calculated according to the relative position relationship and the degree of intersection between the eye mask and the facial features of the first object.

[0011] In one embodiment, the first probability of the eye mask falling off is obtained according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first object, and the CNN-SVM algorithm model is called to perform abnormal sound detection on the spectrogram to obtain the second probability of the abnormal sound occurring, and further includes: Frame shifting the spectrogram in the time dimension to generate a plurality of spectrograms in a fixed format, and calling the convolutional neural network structure in the CNN-SVM algorithm model to extract features from the spectrogram to obtain audio features; Using the audio features as input of a multivariate support vector machine, so as to call the multivariate support vector machine to classify the audio features and output a second probability of the abnormal sound occurring; The abnormal sound at least includes the crying sound and abnormal breathing sound of the first object.

[0012] In one embodiment, the method further comprises: When the second probability exceeds a third threshold and the first probability exceeds the third threshold, determining that the eye mask of the first subject is in a detached state, and sending a second instruction to the blue light therapy device to control the blue light therapy device to stop operating; and When the second probability exceeds a second threshold and the duration of the abnormal sound exceeds a first duration, a third instruction is sent to the blue light therapy device to control the blue light therapy device to stop running.

[0013] In one embodiment, the method further comprises: When the blue light therapy device receives the first instruction or the second instruction, sending abnormal reminder information to the user terminal; and When the blue light therapy device receives the third instruction, it calls the CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram corresponding to the audio data again, and sends the abnormal reminder information to the user end when the probability of the abnormal sound occurring exceeds the fourth threshold.

[0014] The present invention also provides a multi-element control device for a blue light therapy device, the device comprising: A data preprocessing module, used to obtain video data and audio data of a first object at a first moment, and preprocess the video data and audio data to extract corresponding key frames in the video data, and draw a spectrogram corresponding to the audio data; an image detection module, configured to call a YOLO algorithm to detect the facial features and eye mask of the first object according to the key frame, so as to respectively obtain the positions and sizes of the facial features and eye mask of the first object, and determine that the eye mask of the first object is removed when the eyes of the first object are identified; a probability calculation module, configured to obtain a first probability of the eye mask falling off according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first object, and to call a CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring; The device control module is used to send a first instruction to the blue light therapy device to control the blue light therapy device to stop running when the first probability exceeds a first threshold or the second probability exceeds a second threshold.

[0015] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the multi-element control method of the blue light therapy device as described in any one of the above is implemented.

[0016] The present invention also provides a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, the multivariate control method of the blue light therapy device as described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the multivariate control method of a blue light therapy device as described in any one of the above is implemented.

[0018] The above-mentioned multivariate control method, device, electronic device and storage medium of the blue light therapy device obtain the video data and audio data of the patient at the same time, and pre-process the video data and audio data to extract the corresponding key frames in the video data, and draw the spectrogram corresponding to the audio data. Then call the YOLO algorithm to detect the patient's facial features and eye mask according to the key frames to respectively obtain the position and size of the patient's facial features and eye mask, and determine that the patient's eye mask falls off when the patient's eyes are identified. Subsequently, the probability of the eye mask falling off is obtained according to the intersection degree of the relative position relationship between the eye mask and the patient's facial features, and the CNN-SVM algorithm model is called to detect abnormal sounds on the spectrogram to obtain the probability of abnormal sounds. When any probability exceeds its corresponding threshold, an instruction is sent to the blue light therapy device to control the blue light therapy device to stop running. The method uses audio recognition and image recognition technology to more accurately detect the eye mask falling off, and at the same time detects whether there is an abnormality in the infant under the blue light therapy state in multiple dimensions, so as to realize the multivariate control of the blue light therapy device and provide convenience for the care of children in blue light therapy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 One of the flow charts of the multivariate control method of the blue light therapy device provided by the present invention; Figure 2 A schematic diagram of a multi-element control architecture of a blue light therapy device in a multi-element control method of a blue light therapy device in a specific embodiment of the present invention; Figure 3 A schematic diagram of the audio data preprocessing process of the multivariate control method of the blue light therapy device in the specific embodiment provided by the present invention; Figure 4 A schematic diagram of a decision control architecture of a multivariate control method for a blue light therapy device in a specific embodiment provided by the present invention; Figure 5 The second flowchart of the multi-element control method of the blue light therapy device provided by the present invention; Figure 6 The third flow chart of the multi-element control method of the blue light therapy device provided by the present invention; Figure 7 The fourth flowchart of the multi-element control method of the blue light therapy device provided by the present invention; Figure 8The fifth flowchart of the multi-element control method of the blue light therapy device provided by the present invention; Fig. 9 The sixth flow chart of the multi-element control method of the blue light therapy device provided by the present invention; Fig.10 The seventh flowchart of the multi-element control method of the blue light therapy device provided by the present invention; Fig.11 A schematic diagram of the structure of a multi-element control device for a blue light therapy device provided by the present invention; Fig.12 This is a diagram of the internal structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Combine the following Figures 1 to 12 The invention describes a multi-element control method, device, electronic device and storage medium for a blue light therapy device.

[0023] like Figure 1 As shown, in one embodiment, a multivariate control method for a blue light therapy device comprises the following steps: Step S110, obtaining video data and audio data of the first object at the first moment, and preprocessing the video data and the audio data to extract corresponding key frames in the video data, and drawing a spectrogram corresponding to the audio data.

[0024] Specifically, the server obtains the video data and audio data of the first object (patient) at the first moment (the same moment), and preprocesses the video data and audio data to extract key frames in the video data and draw corresponding spectrograms in the audio data.

[0025] Combination Figure 2 and Figure 3As shown, in a specific embodiment, the multivariate control method of the blue light therapy device provided by the present invention is mainly implemented by three modules, namely, an acquisition module, a recognition module and a decision control module. Among them, the acquisition module includes an image acquisition unit and a sound acquisition unit, which are responsible for collecting the picture information and sound information of the environment where the child is at the same time, and reporting to the recognition module after sampling. The image acquisition unit and the sound recognition unit can come from the same device, such as a camera, or from different devices. For example, the audio acquisition unit can be a dedicated audio collector. In addition, multiple image acquisition devices or audio devices can also be equipped to improve the recognition accuracy.

[0026] In this embodiment, the audio recognition unit recognizes the collected audio data and pre-processes the audio, that is, frames and windows the sound, converts the time domain information into frequency domain information through short-time Fourier transform, performs log scale and 0 to 255 transformation, and then splices the processed frequency domain information in time sequence to draw a spectrogram.

[0027] Step S120, calling the YOLO algorithm to detect the facial features and the eye mask of the first object according to the key frame, so as to respectively obtain the positions and sizes of the facial features and the eye mask of the first object, and when the eyes of the first object are identified, determining that the eye mask of the first object has fallen off.

[0028] Specifically, the server calls the YOLO algorithm to detect the facial features and eye mask of the patient (first subject) according to the key frame extracted in step S110, so as to respectively obtain the positions and sizes of the facial features and eye mask of the patient (first subject), and when the eyes of the patient (first subject) are identified, it is determined that the eye mask of the patient (first subject) is in a detached state at this time.

[0029] Combination Figure 2 As shown, in a specific embodiment, the multivariate control method of the blue light therapy device provided by the present invention, the image recognition unit recognizes the collected image data, performs preprocessing, extracts key frame samples from the video, and performs parameter adjustment and correction processing, and standardizes it into a fixed width and height image. Then, the YOLO algorithm is used to detect the facial features of the baby to obtain the position and size of the facial features. If the eyes are recognized, it means that the eye mask is off, and then the probability of the eye mask falling off is determined to be 100%, and the probability result is transmitted to the judgment decision module.

[0030] Step S130, obtaining a first probability of the eye mask falling off according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first object, and calling the CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring.

[0031] Specifically, the server obtains the probability of the eye mask falling off (first probability) based on the intersection degree of the relative position relationship between the eye mask and the facial features of the patient (first object), and calls the CNN-SVM algorithm model to detect abnormal sounds on the spectrogram to obtain the probability of abnormal sounds occurring (second probability).

[0032] Combination Figure 2 As shown, in a specific embodiment, the multivariate control method of the blue light therapy device provided by the present invention, if the eye is not recognized and it is difficult to determine that the probability of the eye mask falling off is 100%, the YOLO algorithm is called to detect the eye mask to obtain the position and size of the eye mask, and then the probability of the eye mask falling off (first probability) is determined according to the relative position relationship between the position and size of the eye mask and the position and size of the facial features, and the intersection degree, and the probability result is transmitted to the judgment decision module.

[0033] In this embodiment, the generated audio spectrogram is frame-shifted in the time dimension to generate multiple spectrograms of fixed size, and the CNN-SVM algorithm model is used to detect abnormal sounds such as crying and breathing. The algorithm input is the spectrogram of audio data, which is automatically extracted through a convolutional neural network structure. The extracted features are then input into a multivariate support vector machine for classification, and the probability of occurrence of abnormal sounds such as the probability of crying and abnormal breathing is output.

[0034] Step S140, when the first probability exceeds the first threshold or the second probability exceeds the second threshold, sending a first instruction to the blue light therapy device to control the blue light therapy device to stop running.

[0035] Specifically, when the probability (first probability) of the eye mask falling off exceeds the corresponding set threshold (first threshold) or the probability (second probability) of abnormal sound occurring exceeds the corresponding set threshold (second threshold), the server sends an instruction (first instruction) to the blue light therapy device to control the blue light therapy device to stop running.

[0036] Combination Figure 4 As shown, in a specific embodiment, the multi-element control method of the blue light therapy device provided by the present invention, the decision control module makes a decision, issues a control instruction and gives a prompt. The decision control module includes a control unit and a prompt unit, and the control unit is electrically connected to the blue light therapy device, and can shut down the blue light therapy device to stop it from emitting light. The reminder unit has an audio reminder and a reminder light reminder, and can remotely push the reminder information to the user end to effectively remind the caregiver.

[0037] In this embodiment, if the probability of the eye mask falling off is greater than 95%, it is determined that the child's eye mask has fallen off, and the blue light therapy device is shut down by the control unit, and the prompt light flashes and a reminder message is pushed to the caregiver user end to remind attention. If the probability of crying is greater than 95%, or the probability of abnormal breathing is greater than 95%, it is determined that the child's state is abnormal, the prompt light flashes and a prompt audio is played, and a reminder message is pushed to the caregiver user end to remind attention. If the probability of crying is greater than 85%, or the probability of abnormal breathing is greater than 85%, and the probability of the eye mask falling off is greater than 85%, it is determined that the child's eye mask has fallen off, and the blue light therapy device is shut down by the control unit, and the prompt light flashes and a reminder message is pushed to the caregiver user end to remind attention. If no one confirms the abnormality and resets the reminder unit within 1 minute after crying or abnormal breathing is detected, the blue light therapy device is shut down, and crying and abnormal breathing detection are performed again. If the probability of crying is greater than 85%, or the probability of abnormal breathing is greater than 85%, the prompt light flashes again and the prompt audio is played directly, and the reminder message is pushed to the caregiver user end to remind attention.

[0038] The above-mentioned multivariate control method of the blue light therapy device obtains the video data and audio data of the patient at the same time, and pre-processes the video data and audio data to extract the corresponding key frames in the video data, and draws the spectrogram corresponding to the audio data. Then, the YOLO algorithm is called to detect the facial features and eye mask of the patient according to the key frames to obtain the position and size of the facial features and eye mask of the patient respectively, and when the patient's eyes are identified, it is determined that the patient's eye mask falls off. Subsequently, the probability of the eye mask falling off is obtained according to the intersection degree of the relative position relationship between the eye mask and the facial features of the patient, and the CNN-SVM algorithm model is called to detect abnormal sounds on the spectrogram to obtain the probability of abnormal sounds. When any probability exceeds the corresponding threshold, an instruction is sent to the blue light therapy device to control the blue light therapy device to stop running. The method uses audio recognition and image recognition technology to more accurately detect the eye mask falling off, and at the same time detects whether there is an abnormality in the infant under the blue light therapy state in multiple dimensions, so as to realize the multivariate control of the blue light therapy device and provide convenience for the care of children in blue light therapy.

[0039] like Figure 5 As shown, in one embodiment, the multivariate control method of the blue light therapy device provided by the present invention obtains video data and audio data of a first object at a first moment, and pre-processes the video data and audio data to extract corresponding key frames in the video data, and draws a spectrogram corresponding to the audio data, specifically comprising the following steps: Step S112: extract multiple key frames from the video data, and perform parameter adjustment and image correction on the key frames to standardize the key frames into image data in a fixed format.

[0040] Step S114, performing frame windowing and short-time Fourier transform on the audio data to convert the time domain information in the audio data into frequency domain information, and splicing the frequency domain information in time sequence to draw a spectrogram.

[0041] like Figure 6 As shown, in one embodiment, the multivariate control method of the blue light therapy device provided by the present invention calls the YOLO algorithm to detect the facial features and eye mask of the first object according to the key frame, so as to respectively obtain the position and size of the facial features and eye mask of the first object, and when the eyes of the first object are identified, it is determined that the eye mask of the first object falls off, specifically comprising the following steps: Step S122, calling the YOLO algorithm to perform facial feature detection on the first object in the image data to obtain the facial feature positions and sizes of the first object, and based on the facial feature positions and irrelevant sizes, determining whether the eyes of the first object are detected.

[0042] Step S124: when the YOLO algorithm detects the eyes of the first subject, it is determined that the eye mask of the first subject is removed; otherwise, it is determined that the eye mask of the first subject is not removed.

[0043] like Figure 7 As shown, in one embodiment, the multivariate control method of the blue light therapy device provided by the present invention obtains a first probability of the eye mask falling off according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first object, and calls the CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring, specifically comprising the following steps: Step S132, calling the YOLO algorithm to detect the position and size of the eye mask, so as to obtain the relative position relationship between the eye mask and the facial features of the first object in combination with the position and size of the facial features of the first object.

[0044] Step S134, calculating a first probability that the eye mask falls off according to the relative position relationship and the degree of intersection between the eye mask and the facial features of the first object.

[0045] like Figure 8 As shown, in one embodiment, the multivariate control method of the blue light therapy device provided by the present invention obtains a first probability of the eye mask falling off according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first object, and calls the CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring, and specifically includes the following steps: Step S136, frame-shifting the spectrogram in the time dimension to generate multiple spectrograms in a fixed format, and calling the convolutional neural network structure in the CNN-SVM algorithm model to extract features from the spectrogram to obtain audio features.

[0046] Step S138, using the audio features as input of a multivariate support vector machine to call the multivariate support vector machine to classify the audio features and output a second probability of the occurrence of abnormal sound.

[0047] The abnormal sound at least includes the crying sound and abnormal breathing sound of the first subject.

[0048] like Fig. 9 As shown, in one embodiment, the multivariate control method of the blue light therapy device provided by the present invention further includes the following steps: Step S910: When the second probability exceeds the third threshold and the first probability exceeds the third threshold, it is determined that the eye mask of the first object is in a detached state, and a second instruction is sent to the blue light therapy device to control the blue light therapy device to stop running.

[0049] Step S920: When the second probability exceeds the second threshold and the duration of the abnormal sound exceeds the first duration, a third instruction is sent to the blue light therapy device to control the blue light therapy device to stop running.

[0050] like Fig.10 As shown, in one embodiment, the multivariate control method of the blue light therapy device provided by the present invention further includes the following steps: Step S1010: When the blue light therapy device receives the first instruction or the second instruction, it sends abnormal reminder information to the user terminal.

[0051] Step S1020, when the blue light therapy device receives the third instruction, it calls the CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram corresponding to the audio data again, and sends abnormal reminder information to the user terminal when the probability of abnormal sound occurrence exceeds the fourth threshold.

[0052] The multi-element control device for a blue light therapy device provided by the present invention is described below. The multi-element control device for a blue light therapy device described below and the multi-element control method for a blue light therapy device described above can be referred to each other.

[0053] like Fig.11 As shown, in one embodiment, a multivariate control device for blue light therapy equipment includes a data preprocessing module 1110 , an image detection module 1120 , a probability calculation module 1130 and a device control module 1140 .

[0054] The data preprocessing module 1110 is used to obtain the video data and audio data of the first object at the first moment, and preprocess the video data and audio data to extract the corresponding key frames in the video data and draw the spectrogram corresponding to the audio data.

[0055] The image detection module 1120 is used to call the YOLO algorithm to detect the facial features and eye mask of the first object according to the key frames, so as to obtain the positions and sizes of the facial features and eye mask of the first object respectively, and when the eyes of the first object are identified, determine that the eye mask of the first object has fallen off.

[0056] The probability calculation module 1130 is used to obtain a first probability of the eye mask falling off according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first object, and call the CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring.

[0057] The device control module 1140 is used to send a first instruction to the blue light therapy device to control the blue light therapy device to stop running when the first probability exceeds the first threshold or the second probability exceeds the second threshold.

[0058] In this embodiment, the data preprocessing module 1110 of the multi-element control device for blue light therapy equipment provided by the present invention is specifically used for: A plurality of key frames are extracted from the video data, and parameters of the key frames are adjusted and images are rectified to standardize the key frames into image data in a fixed format.

[0059] The audio data is framed, windowed and short-time Fourier transformed to convert the time domain information in the audio data into frequency domain information, and the frequency domain information is spliced ​​in time sequence to draw a spectrogram.

[0060] In this embodiment, the multi-element control device for blue light therapy equipment provided by the present invention, the image detection module 1120 is specifically used for: The YOLO algorithm is called to perform facial features detection on the first object in the image data to obtain the positions and sizes of the facial features of the first object, and based on the positions and sizes of the facial features, it is determined whether the eyes of the first object are detected.

[0061] When the YOLO algorithm detects the eyes of the first subject, it is determined that the eye mask of the first subject is removed; otherwise, it is determined that the eye mask of the first subject is not removed.

[0062] In this embodiment, the multivariate control device for blue light therapy equipment provided by the present invention, the probability calculation module 1130 is specifically used for: The YOLO algorithm is called to detect the position and size of the eye mask, so as to obtain the relative position relationship between the eye mask and the facial features of the first subject in combination with the position and size of the facial features of the first subject.

[0063] A first probability of the eye mask falling off is calculated according to the relative position relationship and the degree of intersection between the eye mask and the facial features of the first object.

[0064] In this embodiment, in the multivariate control device for blue light therapy equipment provided by the present invention, the probability calculation module 1130 is further used for: The spectrogram is frame-shifted in the time dimension to generate multiple spectrograms in a fixed format, and the convolutional neural network structure in the CNN-SVM algorithm model is called to extract features from the spectrogram to obtain audio features.

[0065] The audio features are used as input of a multivariate support vector machine, so as to call the multivariate support vector machine to classify the audio features and output a second probability of the occurrence of the abnormal sound.

[0066] The abnormal sound at least includes the crying sound and abnormal breathing sound of the first subject.

[0067] In this embodiment, the blue light therapy device multi-element control device provided by the present invention further includes a device control submodule for: When the second probability exceeds the third threshold and the first probability exceeds the third threshold, it is determined that the eye mask of the first object is in a detached state, and a second instruction is sent to the blue light therapy device to control the blue light therapy device to stop running.

[0068] When the second probability exceeds the second threshold and the duration of the abnormal sound exceeds the first duration, a third instruction is sent to the blue light therapy device to control the blue light therapy device to stop running.

[0069] In this embodiment, the blue light therapy device multi-element control device provided by the present invention further includes a message reminder module for: When the blue light therapy device receives the first instruction or the second instruction, it sends abnormal reminder information to the user terminal.

[0070] When the blue light therapy device receives the third instruction, it calls the CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram corresponding to the audio data again, and sends an abnormal reminder message to the user end when the probability of the abnormal sound occurring exceeds the fourth threshold.

[0071] Fig.12 An example of a physical structure diagram of an electronic device is shown. The electronic device may be a smart terminal, and its internal structure diagram may be as follows: Fig.12 As shown. The electronic device includes a processor, an internal memory and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a multivariate control method of a blue light therapy device is implemented, and the method includes: Obtaining video data and audio data of the first object at a first moment, and preprocessing the video data and the audio data to extract corresponding key frames in the video data, and drawing a spectrogram corresponding to the audio data; Calling the YOLO algorithm to detect the facial features and the eye mask of the first subject according to the key frame, so as to respectively obtain the positions and sizes of the facial features and the eye mask of the first subject, and determining that the eye mask of the first subject is removed when the eyes of the first subject are identified; A first probability of the eye mask falling off is obtained according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first subject, and a CNN-SVM algorithm model is called to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring; When the first probability exceeds the first threshold or the second probability exceeds the second threshold, a first instruction is sent to the blue light therapy device to control the blue light therapy device to stop running.

[0072] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present invention, and does not constitute a limitation on the electronic device to which the scheme of the present invention is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0073] On the other hand, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements a multi-element control method for a blue light therapy device, the method comprising: Obtaining video data and audio data of the first object at a first moment, and preprocessing the video data and the audio data to extract corresponding key frames in the video data, and drawing a spectrogram corresponding to the audio data; Calling the YOLO algorithm to detect the facial features and the eye mask of the first subject according to the key frame, so as to respectively obtain the positions and sizes of the facial features and the eye mask of the first subject, and determining that the eye mask of the first subject is removed when the eyes of the first subject are identified; A first probability of the eye mask falling off is obtained according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first subject, and a CNN-SVM algorithm model is called to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring; When the first probability exceeds the first threshold or the second probability exceeds the second threshold, a first instruction is sent to the blue light therapy device to control the blue light therapy device to stop running.

[0074] In another aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, a multivariate control method for a blue light therapy device is implemented, the method comprising: Obtaining video data and audio data of the first object at a first moment, and preprocessing the video data and the audio data to extract corresponding key frames in the video data, and drawing a spectrogram corresponding to the audio data; Calling the YOLO algorithm to detect the facial features and the eye mask of the first subject according to the key frame, so as to respectively obtain the positions and sizes of the facial features and the eye mask of the first subject, and determining that the eye mask of the first subject is removed when the eyes of the first subject are identified; A first probability of the eye mask falling off is obtained according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first subject, and a CNN-SVM algorithm model is called to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring; When the first probability exceeds the first threshold or the second probability exceeds the second threshold, a first instruction is sent to the blue light therapy device to control the blue light therapy device to stop running.

[0075] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0076] By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0077] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multivariate control method for a blue light therapy device, characterized in that: The method comprises: Acquire video data and audio data of a first object at a first moment, and preprocess the video data and audio data to extract corresponding key frames in the video data, and draw a spectrogram corresponding to the audio data; calling the YOLO algorithm to detect the facial features and the eye mask of the first object according to the key frame, so as to respectively obtain the positions and sizes of the facial features and the eye mask of the first object, and determining that the eye mask of the first object is removed when the eyes of the first object are identified; Obtaining a first probability of the eye mask falling off according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first object, and calling the CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring; When the first probability exceeds a first threshold or the second probability exceeds a second threshold, a first instruction is sent to the blue light therapy device to control the blue light therapy device to stop operating.

2. The multi-element control method of blue light therapy equipment according to claim 1, characterized in that: The step of obtaining video data and audio data of a first object at a first moment, preprocessing the video data and audio data to extract corresponding key frames in the video data, and drawing a spectrogram corresponding to the audio data includes: Extracting a plurality of key frames from the video data, and performing parameter adjustment and image correction on the key frames to standardize the key frames into image data in a fixed format; The audio data is framed, windowed and short-time Fourier transformed to convert time domain information in the audio data into frequency domain information, and the frequency domain information is spliced ​​in time sequence to draw the spectrogram.

3. The multi-element control method of blue light therapy equipment according to claim 2, characterized in that: The calling of the YOLO algorithm to detect the facial features and the eye mask of the first object according to the key frame to respectively obtain the positions and sizes of the facial features and the eye mask of the first object, and determining that the eye mask of the first object is removed when the eyes of the first object are identified, includes: Calling the YOLO algorithm to perform facial features detection on the first object in the image data to obtain the positions and sizes of the facial features of the first object, and determining whether the eyes of the first object are detected based on the positions and sizes of the facial features; When the YOLO algorithm detects the eyes of the first object, it is determined that the eye mask of the first object has fallen off; otherwise, it is determined that the eye mask of the first object has not fallen off.

4. The multi-element control method of blue light therapy equipment according to claim 3, characterized in that: The method of obtaining a first probability of the eye mask falling off according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first object, and calling the CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring includes: Calling the YOLO algorithm to detect the position and size of the eye mask, so as to obtain the relative position relationship between the eye mask and the facial features of the first object in combination with the position and size of the facial features of the first object; A first probability of the eye mask falling off is calculated according to the relative position relationship and the degree of intersection between the eye mask and the facial features of the first object.

5. The multi-element control method of blue light therapy equipment according to claim 4, characterized in that: The method further comprises: obtaining a first probability of the eye mask falling off according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first object, and calling a CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring. Frame shifting the spectrogram in the time dimension to generate a plurality of spectrograms in a fixed format, and calling the convolutional neural network structure in the CNN-SVM algorithm model to extract features from the spectrogram to obtain audio features; Using the audio features as input of a multivariate support vector machine to call the multivariate support vector machine to classify the audio features and output a second probability of the abnormal sound occurring; The abnormal sound at least includes the crying sound and abnormal breathing sound of the first object.

6. The multi-element control method of blue light therapy equipment according to claim 1, characterized in that: The method further comprises: When the second probability exceeds a third threshold and the first probability exceeds the third threshold, determining that the eye mask of the first subject is in a detached state, and sending a second instruction to the blue light therapy device to control the blue light therapy device to stop operating; and When the second probability exceeds a second threshold and the duration of the abnormal sound exceeds a first duration, a third instruction is sent to the blue light therapy device to control the blue light therapy device to stop running.

7. The multi-element control method of blue light therapy equipment according to claim 6, characterized in that: The method further comprises: When the blue light therapy device receives the first instruction or the second instruction, sending abnormal reminder information to the user terminal; and When the blue light therapy device receives the third instruction, it calls the CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram corresponding to the audio data again, and sends the abnormal reminder information to the user end when the probability of the abnormal sound occurring exceeds the fourth threshold.

8. A multi-element control device for blue light therapy equipment, characterized in that: The device comprises: A data preprocessing module, used to obtain video data and audio data of a first object at a first moment, and preprocess the video data and audio data to extract corresponding key frames in the video data, and draw a spectrogram corresponding to the audio data; an image detection module, configured to call a YOLO algorithm to detect the facial features and eye mask of the first object according to the key frame, so as to respectively obtain the positions and sizes of the facial features and eye mask of the first object, and determine that the eye mask of the first object is removed when the eyes of the first object are identified; a probability calculation module, configured to obtain a first probability of the eye mask falling off according to the intersection degree of the relative position relationship between the eye mask and the facial features of the first object, and to call a CNN-SVM algorithm model to perform abnormal sound detection on the spectrogram to obtain a second probability of the abnormal sound occurring; The device control module is used to send a first instruction to the blue light therapy device to control the blue light therapy device to stop running when the first probability exceeds a first threshold or the second probability exceeds a second threshold.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images