Control method and apparatus for monitoring and playback system, computer device, and storage medium
By using pre-trained target neural network model on the smart terminal for gaze recognition and controlling the display status, the problem of high power consumption in monitoring applications is solved, and a lower power consumption and a more accurate wake-up mechanism is achieved.
Patent Information
- Application Number
- PCT/CN2024/088351
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-04-17
- Publication Date
- 2024-12-05
AI Technical Summary
Smart terminal devices consume a lot of power when displaying monitoring screens for a long time, which affects the system life, and it is difficult for the existing technology to effectively reduce power consumption.
The pre-trained target neural network model is used for gaze recognition, and the display status of the terminal is controlled by the recognition result. If no one is watching the screen, it will enter sleep mode, saving power consumption.
Through precise gaze recognition technology, the power consumption of smart terminals is effectively reduced, the service life of the device is extended, and the screen is quickly awakened when needed, improving the energy efficiency performance of the system.
Smart Images

Figure CN2024088351_05122024_PF_FP_ABST
Abstract
Description
Control method, device, computer equipment and storage medium of monitoring system Technical Field
[0001] The present disclosure belongs to the technical field of image recognition and terminal monitoring, and particularly relates to a control method, apparatus, computer equipment, and storage medium of a monitoring system. Background Art
[0002] With the rapid development of information technology, modern electronic devices are also rapidly developing towards intelligence, lightweightness, and portability. For smart terminals, displaying surveillance images for extended periods of time results in high system power consumption, impacting system lifespan. Therefore, achieving low-power operation is a pressing technical challenge in the surveillance and broadcasting field.
[0003] Summary of the Invention
[0004] The present disclosure aims to solve at least one of the technical problems existing in the prior art and provides a control method, device, computer equipment and storage medium for a monitoring system.
[0005] In a first aspect, the technical solution adopted to solve the technical problem of the present disclosure is a control method of a monitoring system, comprising:
[0006] Acquire detection images;
[0007] Using a pre-trained target neural network model, performing gaze recognition on the detection image to obtain a recognition result of the detection image;
[0008] The recognition result is sent to a terminal, so that the terminal determines a display state based on at least the recognition result.
[0009] In some embodiments, the target neural network model is trained by the following steps:
[0010] Training the teacher network learning model to be trained according to the acquired training data set to obtain a trained teacher network learning model;
[0011] According to the training data set and the trained teacher network learning model, the student network learning model to be trained is trained using a knowledge distillation training method to obtain a trained student network learning model as the target neural network model.
[0012] In some embodiments, the training dataset includes a plurality of training images annotated with sample labels;
[0013] The method of training the student network learning model to be trained using a knowledge distillation training method based on the training data set and the trained teacher network learning model to obtain a trained student network learning model as the target neural network model includes:
[0014] Inputting the training image into the trained teacher network learning model to obtain a first output result of the trained teacher network learning model;
[0015] Inputting the training image into the student network learning model to be trained to obtain a second output result of the student network learning model to be trained;
[0016] Determining a first loss function according to the first output result and the second output result;
[0017] Determining a second loss function according to the second output result and the sample label of the training image;
[0018] Obtaining a weighted loss function according to the first loss function and the second loss function;
[0019] The parameters of the student network learning model to be trained are adjusted according to the weighted loss function until the weighted loss function converges, thereby obtaining a trained student network learning model as the target neural network model.
[0020] In some embodiments, the training data set is determined by the following steps:
[0021] Acquire an original data set; the original data set includes a plurality of initial sample images;
[0022] Performing face recognition on the initial sample image, and if a face is present, determining a first reference frame containing the face, and determining key point information of the face;
[0023] Determining facial posture information based on the key point information;
[0024] updating the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain a second reference frame;
[0025] According to the first reference frame and the second reference frame, marking the sample label for a portion of the sample image of the initial sample image located in the second reference frame, and using the portion of the sample image marked with the sample label as the training image;
[0026] The sample labels include a first category label representing gaze, and a second category label and a third category label representing non-gaze; wherein,
[0027] The first category label indicates that when a ratio of a second reference frame in the training image to the training image is greater than or equal to a first preset threshold, a pitch angle of the face is less than or equal to a first set value, a yaw angle of the face is less than or equal to a second set value, or a rotation angle of the face is less than or equal to a third set value;
[0028] The second category label indicates that when a ratio of the second reference frame in the training image to the training image is greater than or equal to a first preset threshold, a pitch angle of the face is greater than a first set value, a yaw angle of the face is greater than a second set value, or a rotation angle of the face is greater than a third set value;
[0029] The third category label indicates that a ratio of the second reference frame in the training image to the training image is less than a first preset threshold and greater than a second preset threshold.
[0030] In some embodiments, the posture information includes at least the pitch angle, yaw angle, and rotation angle of the face;
[0031] The updating of the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain a second reference frame includes:
[0032] When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the yaw angle of the face is less than or equal to a second set value, and the rotation angle of the face is less than or equal to a third set value, the specific coordinate point in the first reference frame is moved according to the position information of the first reference frame and within a first set range to obtain a second reference frame.
[0033] In some embodiments, the step of marking the portion of the initial sample image located in the second reference frame with the sample label according to the first reference frame and the second reference frame, and using the portion of the sample image marked with the sample label as the training image, includes:
[0034] determining an overlapping area between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame;
[0035] When a first ratio of the overlapping area to the area of the second reference frame is greater than or equal to a third preset threshold, a first preset range is generated according to a first preset probability, a second preset range is generated according to a second preset probability, a third preset range is generated according to a third preset probability, and a fourth preset range is generated according to a fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; and the sum of the first preset probability, the second preset probability, the third preset probability, and the fourth preset probability is 1;
[0036] When the first ratio is within the first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the first preset range and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the first preset probability;
[0037] When the first ratio is within the second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the second preset range and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is a second preset probability;
[0038] When the first ratio is within the third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the third preset range and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the third preset probability;
[0039] When the first ratio is within the fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the fourth preset range and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the fourth preset probability;
[0040] The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.
[0041] In some embodiments, the posture information includes at least the pitch angle, yaw angle, and rotation angle of the face;
[0042] The updating of the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain a second reference frame includes:
[0043] When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is greater than a first set value, the yaw angle of the face is greater than a second set value, and the rotation angle of the face is greater than a third set value, the specific coordinate point in the first reference frame is moved according to the position information of the first reference frame and within a second set range to obtain a second reference frame.
[0044] In some embodiments, the step of marking the portion of the initial sample image located in the second reference frame with the sample label according to the first reference frame and the second reference frame, and using the portion of the sample image marked with the sample label as the training image, includes:
[0045] determining an overlapping area between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame;
[0046] When a third ratio of the overlapping area to the area of the second reference frame is greater than or equal to a third preset threshold, a first preset range is generated according to a first preset probability, a second preset range is generated according to a second preset probability, a third preset range is generated according to a third preset probability, and a fourth preset range is generated according to a fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; and the sum of the first preset probability, the second preset probability, the third preset probability, and the fourth preset probability is 1;
[0047] When the third ratio is within the first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the second category label, so that the third ratio is within the first preset range and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the first preset probability;
[0048] When the third ratio is within the second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a second category label, so that the third ratio is within the second preset range and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the second preset probability;
[0049] When the third ratio is within the third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a second category label, so that the third ratio is within the third preset range and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the third preset probability;
[0050] When the third ratio is within the fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the second category label, so that the third ratio is within the fourth preset range and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the fourth preset probability;
[0051] The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.
[0052] In some embodiments, the posture information includes at least the pitch angle, yaw angle, and rotation angle of the face;
[0053] The updating of the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain a second reference frame includes:
[0054] When the ratio of the first reference frame to the initial sample image is less than a first preset threshold and greater than a second preset threshold, the specific coordinate point in the first reference frame is moved according to the position information of the first reference frame and within a third set range to obtain a second reference frame; the first preset threshold is greater than the second preset threshold.
[0055] In some embodiments, the step of marking the portion of the initial sample image located in the second reference frame with the sample label according to the first reference frame and the second reference frame, and using the portion of the sample image marked with the sample label as the training image, includes:
[0056] determining an overlapping area between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame;
[0057] If a fifth ratio of the overlapping area to the area of the second reference frame is less than a third preset threshold, generating a fifth preset range according to a fifth preset probability, generating a sixth preset range according to a sixth preset probability, and generating a seventh preset range according to a seventh preset probability; the fifth preset probability is greater than the sixth preset probability, and the sixth preset probability is greater than the seventh preset probability; and the sum of the fifth preset probability, the sixth preset probability, and the seventh preset probability is 1;
[0058] When the fifth ratio is within the fifth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a third category label, so that the fifth ratio is within the fifth preset range and a sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the fifth preset probability;
[0059] When the fifth ratio is within the sixth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as having a third category label, so that the sixth ratio is within the sixth preset range and a sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the sixth preset probability;
[0060] When the fifth ratio is within the seventh preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within the seventh preset range and the sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the seventh preset probability;
[0061] The data within the seventh preset range is smaller than the data within the sixth preset range, and the data within the sixth preset range is smaller than the data within the fifth preset range.
[0062] In some embodiments, the sample labels further include a fourth category label representing non-gaze;
[0063] In the step of obtaining the training data set, if face recognition is performed on the initial sample image to determine that no face exists, the step further includes:
[0064] The initial sample image is labeled with a fourth category label, and the initial sample image labeled with the fourth category label is used as the training image.
[0065] In some embodiments, the sample labels further include a fourth category label representing non-gaze;
[0066] After performing face recognition on the initial sample image to determine the presence of a face and determining a first reference frame containing the face, the method further includes:
[0067] Determining a target center point in the initial sample image according to the size of the initial sample image and within a fourth set range, and determining a third reference frame with the target center point as the center;
[0068] If the third reference frame does not overlap with any first reference frame in the initial sample image, the portion of the sample image of the initial sample image located in the third reference frame is marked as a fourth category label, and the portion of the sample image marked with the fourth category label is used as the training image.
[0069] In a second aspect, an embodiment of the present disclosure further provides a control method for a monitoring system, wherein the monitoring system includes a detection device and a terminal; the control method for the monitoring system includes:
[0070] The detection device acquires a detection image, and uses a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and sends the recognition result to the terminal;
[0071] The terminal determines a display state based on the recognition result.
[0072] In some embodiments, the terminal includes a display module and a main control module;
[0073] The terminal determines a display state based on the recognition result, including:
[0074] The main control module sends a wake-up request to the display module when it is determined that the recognition result indicates that the object is gazing; and sends a sleep request to the display module when it is determined that the recognition result indicates that the object is not gazing and the time between the current system time and the last time the display module was awakened is greater than a preset time.
[0075] The display module responds to the wake-up request and determines that the display state is normal display; and responds to the sleep request and determines that the display state is sleep.
[0076] On the third aspect, an embodiment of the present disclosure also provides a detection device, which is configured to acquire a detection image, and use a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and send the recognition result to a terminal so that the terminal determines a display state at least based on the recognition result.
[0077] In a fourth aspect, an embodiment of the present disclosure further provides a control device for a monitoring system, which includes a detection device and a terminal;
[0078] The detection device is configured to acquire a detection image, and perform gaze recognition on the detection image using a pre-trained target neural network model to obtain a recognition result of the detection image; and send the recognition result to the terminal;
[0079] The terminal is configured to determine a display state based on the recognition result.
[0080] In a fifth aspect, an embodiment of the present disclosure further provides a computer device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the control method of the monitoring system as described in any one of the first aspect above, and / or the steps of the control method of the monitoring system as described in any one of the second aspect above are performed.
[0081] In a sixth aspect, an embodiment of the present disclosure further provides a computer non-volatile readable storage medium, wherein a computer program is stored on the computer non-volatile readable storage medium, and when the computer program is executed by a processor, the steps of the control method of the monitoring system as described in any one of the first aspects above, and / or the steps of the control method of the monitoring system as described in any one of the second aspects above are executed. BRIEF DESCRIPTION OF THE DRAWINGS
[0082] FIG1 is a flow chart of a control method of a monitoring system provided by an embodiment of the present disclosure;
[0083] FIG2 is a schematic diagram of a network architecture of an exemplary target neural network model provided by an embodiment of the present disclosure;
[0084] FIG3 is a flow chart of target neural network model training according to an embodiment of the present disclosure;
[0085] FIG4 is a schematic diagram of an exemplary network architecture of knowledge distillation provided by an embodiment of the present disclosure;
[0086] FIG5 is a flowchart of generating a training data set according to an embodiment of the present disclosure;
[0087] FIG6 is a schematic diagram of an exemplary face summary target key points provided by an embodiment of the present disclosure;
[0088] FIG7 is a schematic diagram of exemplary different facial postures provided by an embodiment of the present disclosure;
[0089] FIG8a is a diagram of an exemplary terminal workflow provided by an embodiment of the present disclosure;
[0090] FIG8b is a flowchart of an exemplary detection device provided in an embodiment of the present disclosure;
[0091] FIG9 is a schematic diagram of a detection device provided in an embodiment of the present disclosure;
[0092] FIG10 is a schematic diagram of a control device of a monitoring system according to an embodiment of the present disclosure;
[0093] FIG11 is a schematic structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0094] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.
[0095] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "an" or "the" do not indicate a quantitative limitation, but rather indicate the presence of at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.
[0096] In this disclosure, "multiple or several" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.
[0097] With the in-depth research and widespread adoption of artificial intelligence algorithms, particularly deep learning neural networks, smart electronic devices and related application scenarios have become ubiquitous, such as facial recognition, voice recognition, smart homes, security surveillance, and autonomous driving. However, due to the limitations of the classic von Neumann computing architecture, data storage and processing are separated, with computing and storage functions performed by the central processing unit (CPU) and memory, respectively. This performance gap creates a "memory wall," resulting in significant energy consumption in frequent transfers between the computing core and memory. However, current storage-computing gradient designs can address this issue. Integrated storage and computing combines memory and computing more tightly than traditional computer architectures, reducing the overhead associated with memory accesses and addressing the "memory wall" problem. The three key elements of artificial intelligence are computing power, data, and algorithms. Integrated storage and computing offers high computing power, low power consumption, and low latency, making them poised to play a significant role in the future of artificial intelligence. Currently, common storage and computing methods include flash memory and static random-access memory (SRAM). Generally, many national or industry standards associated with smart terminal products have certain standby power consumption requirements. Designing an intelligent wake-up function on such products to support 24-hour standby wake-up through voice, personnel detection, and other methods requires ensuring that the entire detection system has low standby power consumption to meet the corresponding energy efficiency standards.
[0098] In view of this, the embodiment of the present disclosure provides a control method for a monitoring system, which essentially eliminates one or more of the problems caused by the limitations and defects of the relevant technologies. Specifically, the embodiment of the present disclosure obtains a detection image; uses a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and sends the recognition result to the terminal so that the terminal determines the display state based on at least the recognition result. That is, the display state of the terminal is controlled by using the high-precision gaze recognition result. For example, if no one is looking at the screen for a long time, the terminal automatically enters a sleep state, thereby saving system power consumption; if after entering the sleep state, the recognition result indicates that a person is looking at the screen, the system can be automatically started to restore the normal display of the screen. The present disclosure is applied to intelligent wake-up scenarios in a gaze environment (such as an environment where a person is staring at a display screen). While saving system power consumption, it combines neural network gaze recognition technology to improve the recognition accuracy of people looking at the screen, thereby more accurately controlling the timing of the terminal entering sleep or actively waking up.
[0099] To facilitate understanding, we first describe in detail a control method for a monitoring system provided by an embodiment of the present disclosure. This method can be executed by a detection device with a certain level of computing power. For example, this detection device can be a memory-computing integrated chip with integrated Flash or SRAM, which provides faster response times and reduces power consumption. The detection device can be part of the monitoring system.
[0100] FIG1 is a flow chart of a control method of a monitoring system provided by an embodiment of the present disclosure. As shown in FIG1 , the method includes steps S11 to S13, wherein:
[0101] S11. Acquire a detection image.
[0102] In this step, the detection image is an image captured by a camera and is used to detect whether someone is staring at the screen. The detection image can be an image captured by the camera in real time.
[0103] The shooting device can be, for example, a camera or other device with image acquisition function, such as a mobile phone, a vehicle terminal, a smart home device, a security monitoring device, etc.
[0104] S12. Using the pre-trained target neural network model, perform gaze recognition on the detection image to obtain a recognition result of the detection image.
[0105] Specifically, the detection image can be input into a pre-trained target neural network model, which extracts and analyzes features from the detection image to determine whether someone is staring at the screen. The target neural network model can use a lightweight classification model, and its recognition result may be that someone is staring at the screen, or it may be that no one is staring at the screen. Here, the screen can be, for example, a terminal display or an image acquisition device such as a camera.
[0106] In this step, a lightweight target neural network model is designed and migrated to the storage and computing integrated chip. A low-power storage and computing integrated method is used to assist the target neural network in reasoning, complete the wake-up service of the smart terminal, reduce power consumption, and speed up the response.
[0107] S13. Send the recognition result to the terminal, so that the terminal determines the display state based on at least the recognition result.
[0108] Specifically, the recognition result is sent to the terminal, and the main control module in the terminal can determine the display status of the terminal according to the indication of the recognition result; the display screen in the terminal can respond to the currently determined display status and determine whether to display the picture normally or enter sleep.
[0109] The control method of the monitoring system provided by the present invention realizes monitoring by adopting a low-power storage and computing integrated method. The designed lightweight target neural network model is migrated to the storage and computing integrated chip, and the lightweight feature is used to reduce the power consumption of the storage and computing integrated chip. At the same time, combined with the neural network gaze recognition technology, the recognition accuracy of people looking at the screen is improved, so as to more accurately control the timing of the terminal entering sleep or actively waking up. When the display screen enters the sleep state, the system power consumption is saved. If the terminal is not actively awakened, it can remain in standby for a long time, thereby solving the problem of high power consumption and reducing a lot of maintenance costs.
[0110] Since the intelligent wake-up application scenario of the present disclosure is limited by the hardware of the integrated storage and computing chip, it is necessary to design a lightweight target neural network model to complete the target detection task. The following is a detailed introduction to the target neural network model training process provided by the embodiment of the present disclosure.
[0111] In some embodiments, FIG2 is a schematic diagram of the network architecture of an exemplary target neural network model provided by an embodiment of the present disclosure. As shown in FIG2 , it includes seven convolutional layers conv. For example, the first convolutional layer conv1 uses a convolution kernel with a kernel size of 3×3 for convolution, a convolution step length of stride 2, and a padding value of 1; the second convolutional layer conv2 uses a convolution kernel with a kernel size of 3×3 for convolution, a convolution step length of stride 2, and a padding value of 1; the third convolutional layer conv3 uses a convolution kernel with a kernel size of 3×3 for convolution, a convolution step length of stride 1, and a padding value of 1; the fourth convolutional layer conv4 uses a convolution kernel with a kernel size of 3×3 for convolution, a convolution step length of stride 2, and a padding value of 1; the fifth convolutional layer conv5 uses a convolution kernel with a kernel size of 7×7 for convolution, a convolution step length of stride 2, and a padding value of 1; the sixth convolutional layer conv6 uses a convolution kernel with a kernel size of 7×7 for convolution, a convolution step length of stride 2, and a padding value of 1. The convolution kernel of size 3×3 is used for convolution, the stride of convolution is 1, and the padding value is 1; the seventh convolution layer conv7 uses the convolution kernel of size 3×3 for convolution, the stride of convolution is 2, and the padding value is 1.
[0112] The above-mentioned target neural network model meets the lightweight requirements and is migrated to the storage and computing integrated chip. During its application process, it has faster response speed and lower power consumption.
[0113] Because the model is relatively small, directly training the designed model may not achieve high-precision recognition results. Therefore, a large model distillation method is used to improve performance. A MobilNetV3 with a width of 0.5 is used as the large model, also known as the teacher network learning model. Finally, the large teacher model is converted into a smaller student network learning model (student model) as the target neural network model after training.
[0114] In some embodiments, FIG3 is a flowchart of a target neural network model training according to an embodiment of the present disclosure. As shown in FIG3 , the target neural network model is trained by the following steps S21 to S22:
[0115] S21. Train the teacher network learning model to be trained according to the acquired training data set to obtain a trained teacher network learning model.
[0116] Here, the training data set can be a labeled public data set and / or collected video data. However, unlike face recognition, the gaze recognition scenario is based on face recognition and further uses the information of key points in the face to determine whether the human eye is in a state of staring at the screen. Therefore, compared with the face recognition scenario, the number of positive samples (i.e., gaze samples) in the existing public data set is often smaller. In order to improve the accuracy of model training, a large amount of sample data support is required. Therefore, the training data set disclosed in the present invention can be based on the labeled public data set and / or collected video data, and further enrich the sample types of the data set to make the obtained training images more diverse. The method of enriching the sample types can be referred to the following steps S211 to S217, and the detailed process is not described here. Among them, the training data set includes multiple training images, and each training image includes a sample label corresponding to it.
[0117] Exemplarily, a training image from a training dataset is input into the teacher network learning model to be trained, and an output corresponding to the training image is obtained, where the output indicates whether a person is gazing at the screen. If the sample label of the training image indicates that a person is gazing at the screen, a third loss function is determined based on the output and the sample label of the training image. The above process is repeated, sequentially traversing the training images in the training dataset, and determining the third loss function to continuously train the teacher network learning model to be trained, ultimately obtaining a trained teacher network learning model.
[0118] S22. Based on the training data set and the trained teacher network learning model, the knowledge distillation training method is used to train the student network learning model to obtain the trained student network learning model as the target neural network model.
[0119] Figure 4 is a schematic diagram of an exemplary network architecture for knowledge distillation provided by an embodiment of the present disclosure. As shown in Figure 4, it includes a teacher model and a student model. The teacher model includes m layers of network layers; the student model includes n layers of network layers. Here, the teacher model and the student model can be homogeneous networks or heterogeneous networks.
[0120] The process of training the student model also uses the hard labels in the training dataset (i.e., the sample labels carried by the training images) to train the teacher model, and then uses the soft labels obtained in the teacher model (i.e., the output of the softmax layer in the teacher model) combined with the hard labels in the training dataset to jointly determine the loss function of the student model.
[0121] Continuing with Figure 4, the output of layer m of the teacher model is processed by the softmax layer to obtain the first output. The output of layer n of the student model is processed by the softmax layer to obtain the second output. The loss function is constructed based on the first and second outputs and the sample labels.
[0122] This embodiment uses the knowledge distillation method to convert a larger teacher model into a smaller student model while retaining performance close to that of the teacher model, thereby solving the problem of insufficient hardware for deploying the target neural network model at the edge segment.
[0123] The specific training process is shown in steps S221 to S226, where:
[0124] S221. Input the training image into the trained teacher network learning model to obtain a first output result of the trained teacher network learning model.
[0125] Exemplarily, the training image A in the training data set is input into the trained teacher network learning model to obtain a first output result corresponding to the training image A. The first output result is the result output by the trained teacher network learning model indicating whether someone is staring at the screen in the training image A, such as the predicted label of the training image A predicted by the trained teacher network learning model.
[0126] S222: Input the training image into the student network learning model to be trained to obtain a second output result of the student network learning model to be trained.
[0127] Continuing with the above example, the training image A in the training data set is input into the student network learning model to be trained to obtain the second output result corresponding to the training image A. The second output result is the result output by the student network learning model to be trained, indicating whether someone is staring at the screen in the training image A, for example, the predicted label of the training image A predicted by the student network learning model to be trained.
[0128] It should be noted that the same training image corresponds to a set of first output results and second output results.
[0129] S223. Determine a first loss function according to the first output result and the second output result.
[0130] According to a set of first output results and second output results corresponding to the same training image, the mean square error of the first output result and the second output result is calculated as the first loss function; or according to a set of first output results and the second output results corresponding to the same training image, the first loss function is determined using the KLD loss function algorithm (also known as KLDloss).
[0131] S224. Determine a second loss function based on the second output result and the sample label of the training image.
[0132] The second loss function can be determined based on the overlap IOU between the second output result (i.e., the predicted label of the training image A predicted by the student network learning model to be trained) and the sample label of the training image (i.e., the real hard label).
[0133] S225. Obtain a weighted loss function according to the first loss function and the second loss function.
[0134] The first loss function and the second loss function are weighted according to a preset weight to obtain a weighted loss function. Here, the preset weight can be determined based on experience and is not specifically limited in the embodiment of the present disclosure.
[0135] Obtain the weighted coefficients of the first loss function and the second loss function. In this embodiment, since the weighted loss function is composed of the first loss function and the second loss function, by determining one of the weighted coefficients (α), the other weighted coefficient (i.e., 1-α) can be determined. Afterwards, the first loss function L1 and the second loss function L2 are weighted and summed according to the weighted coefficients to determine the weighted loss function. The weighted loss function loss can be specifically expressed as follows in combination with the above description: L = α × L1 + (1-α) × L2.
[0136] S226. Adjust the parameters of the student network learning model to be trained according to the weighted loss function until the weighted loss function converges, and obtain the trained student network learning model as the target neural network model.
[0137] Furthermore, while lightweight models reduce system power consumption, they often suffer from insufficient detection accuracy. To improve the recognition accuracy of a lightweight target neural network model, this paper improves the model training process. Specifically, it diversifies the training images at the training sample level and trains the model using training images corresponding to multiple category labels, thereby improving the training accuracy of the target neural network model.
[0138] In some embodiments, the training dataset can be a dataset that is further enriched with sample types based on the original dataset. FIG5 is a flowchart of generating a training dataset according to an embodiment of the present disclosure. As shown in FIG5 , the steps of generating a diversified training image include S211 to S217, wherein:
[0139] S211. Obtain the original data set.
[0140] The original data set includes multiple initial sample images.
[0141] S212: Perform face recognition on the initial sample image. If a face is found, execute step S213; if no face is found, execute step S217.
[0142] For example, the face detection network retinaface or the neural network yolov5 model can be used to perform face recognition on the initial sample image.
[0143] S213: Determine a first reference frame containing the face, and determine key point information of the face.
[0144] When it is determined that a face exists in the initial sample image, a first reference frame is marked for the area where the face is located, so that the first reference frame frames the face. Position information of the first reference frame indicates the position of the face in the initial sample image.
[0145] When it is determined that a face exists in the initial sample image, target key points of the face are further determined. The key point information represents the coordinates of the target key points in the initial sample image. FIG6 is a schematic diagram of an exemplary face summary target key points provided by an embodiment of the present disclosure. As shown in FIG6 , the target key points here can be customized according to the actual scenario. For example, the target key points may include key points such as left eye 1, right eye 2, nose 3, left corner of mouth 4, right corner of mouth 5, and the center of the left and right corners of mouth 6.
[0146] S214: Determine facial posture information based on key point information.
[0147] Figure 7 is a schematic diagram of an exemplary different facial postures provided by an embodiment of the present disclosure. As shown in Figure 7, specifically, a posture estimation algorithm (such as the posture estimation algorithm in POSIT in OpenCV) can be used to project the target key points (3D key points) of the six world coordinate systems to five 2D key points through changes such as rotation and translation, and then estimate the transformation parameters. Finally, the posture information of the face in the 2D plane, such as the pitch angle, yaw angle and roll angle of the face, can be obtained.
[0148] The subsequent combination of multi-angle posture information can improve the accuracy of sample labels of training images, thereby improving the accuracy of model training.
[0149] S215 : Update the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain a second reference frame.
[0150] Different posture information and / or different ratios of the first reference frame to the initial sample image correspond to different ways of updating the first reference frame, thereby obtaining different second reference frames. The specific updating method may be, for example, randomly moving specific coordinate points.
[0151] In one embodiment, the posture information includes the pitch angle, yaw angle, and rotation angle of the face.
[0152] When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the yaw angle of the face is less than or equal to a second set value, and the rotation angle of the face is less than or equal to a third set value, based on the position information of the first reference frame and within the first set range, a specific coordinate point in the first reference frame is moved to obtain a second reference frame.
[0153] Among them, the first preset threshold, the first set value, the second set value, and the third set value can all be set according to the actual application scenario. For example, the first preset threshold is set to 50%, the first set value is set to 25°, the second set value is set to 25°, and the third set value is set to 45°. That is to say, under the conditions that the ratio of the first reference frame to the initial sample image is greater than or equal to 50%, the deflection angle of the face is less than or equal to 25°, the pitch angle of the face is less than or equal to 25°, and the rotation angle of the face is less than or equal to 45°, it is determined that the person is in the state of staring at the screen.
[0154] Here, when the ratio of the first reference frame to the initial sample image is greater than or equal to the first preset threshold, the pitch angle of the face is less than or equal to the first set value, the deflection angle of the face is less than or equal to the second set value, and the rotation angle of the face is less than or equal to the third set value, it can be determined that the sample label to be marked is the first category label, and the first category label is a label used to characterize gaze. It should be noted that this is only the determination of the sample label to be marked, and it is not a marking process. According to the current determination that the sample label to be marked is the first category label, the corresponding first set range is determined. The mapping relationship between the first category label and the first set range is pre-set.
[0155] Then, based on the position information of the first reference frame, the specific coordinate points in the first reference frame are moved within a first set range to obtain a second reference frame. Here, the first set range corresponding to the first category label may be a pre-set movement range for the specific coordinate points. The specific coordinate points may be vertices of the outer contour of the first reference frame, such as Vertex 1 in the upper left corner, Vertex 2 in the upper right corner, Vertex 3 in the lower left corner, and Vertex 4 in the lower right corner. In this embodiment, Vertex 1 in the upper left corner and Vertex 4 in the lower right corner may be selected as the specific coordinate points for the diversified update of the reference frame.
[0156] For example, the coordinates of the vertex 1 in the upper left corner are (lt x ,lt y ), the coordinates of vertex 4 in the lower right corner are (rb x ,rb y ); Let the coordinates of vertex 1 (lt x ,lt y ) has a random moving value of δ x , δ y ; The coordinates of vertex 4 (rb x ,rb y )’s random moving value σ x , σ y , the value can be positive or negative. After random movement, the coordinates of the new vertex 1 are (lt1 x ,lt1 y ), that is: lt1 x =lt x +δ x lt1 y =lt y +δ y
[0157] The coordinates of the new vertex 4 are (rb1 x ,rb1 y ), that is: rb1 x =rb x +σ x rb1 y =rb y +σ y
[0158] Among them, the first setting range includes δ x , δ y , σ x and σ y The range of, for example, δ x ∈[-w1,0],δ y ∈[-h1,0],σ x ∈[0,rw1],σ y ∈[0,h1]. Where w1=(rb x-lt x )×0.8;h1=(rb y -lt y When the specific coordinate point is moved according to the first set range, the corresponding first reference frame expands outward, and the expansion range is relatively small to ensure a better gaze effect of the face in the second reference frame.
[0159] In one embodiment, the posture information includes the pitch angle, yaw angle, and rotation angle of the face.
[0160] When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is greater than a first set value, the yaw angle of the face is greater than a second set value, and the rotation angle of the face is greater than a third set value, the specific coordinate point in the first reference frame is moved according to the position information of the first reference frame and within a second set range to obtain a second reference frame.
[0161] The first preset threshold, the first set value, the second set value, and the third set value can all be set according to actual application scenarios. For example, the first preset threshold is set to 50%, the first set value is set to 25°, the second set value is set to 25°, and the third set value is set to 45°. In other words, if the ratio of the first reference frame to the initial sample image is greater than or equal to 50%, the deflection angle of the face is greater than 25°, the pitch angle of the face is greater than 25°, and the rotation angle of the face is greater than 45°, it is determined that the person is not in the state of staring at the screen.
[0162] Here, when the ratio of the first reference frame to the initial sample image is greater than or equal to the first preset threshold, the pitch angle of the face is greater than the first set value, the yaw angle of the face is greater than the second set value, and the rotation angle of the face is greater than the third set value, it can be determined that the sample label to be marked is the second category label, and the second category label is a label used to characterize non-gaze. It should be noted that this is only the determination of the sample label to be marked, not the marking process. According to the current determination that the sample label to be marked is the second category label, the corresponding second set range is determined. The mapping relationship between the second category label and the second set range is pre-set.
[0163] Then, based on the position information of the first reference frame, the specific coordinate points in the first reference frame are moved within a second set range to obtain a second reference frame. Here, the second set range corresponding to the second category label can be a pre-set movement range for the specific coordinate points. The specific coordinate points can be vertices of the outer contour of the first reference frame, such as Vertex 1 in the upper left corner, Vertex 2 in the upper right corner, Vertex 3 in the lower left corner, and Vertex 4 in the lower right corner. In this embodiment, Vertex 1 in the upper left corner and Vertex 4 in the lower right corner can be selected as the specific coordinate points for the diversified update of the reference frame.
[0164] For example, the coordinates of the vertex 1 in the upper left corner are (lt x ,lt y ), the coordinates of vertex 4 in the lower right corner are (rb x ,rb y ); Let the coordinates of vertex 1 (lt x ,lt y ) has a random moving value of δ x , δ y ; The coordinates of vertex 4 (rb x ,rb y )’s random moving value σ x , σ y , the value can be positive or negative. After random movement, the coordinates of the new vertex 1 are (lt1 x ,lt1 y ), that is: lt1 x =lt x +δ x lt1 y =lt y +δ y
[0165] The coordinates of the new vertex 4 are (rb1 x ,rb1 y ), that is: rb1 x =rb x +σ x rb1 y =rb y +σ y
[0166] The second setting range can be the same as or different from the first setting range. This disclosure takes the second setting range and the first setting range as an example for explanation. The second setting range includes δ x , δ y , σ x and σ y The range of, for example, δ x ∈[-w1,0],δ y ∈[-h1,0],σ x ∈[0,w1],σ y ∈[0,h1]. Where w1=(rb x -lt x )×0.8;h1=(rb y -lt y )×0.5. When the specific coordinate point is moved according to the second set range, the corresponding first reference frame expands outward, and the expansion range is relatively small, so as to ensure the diversity of samples of non-gaze effects of faces within the second reference frame, thereby improving the accuracy of subsequent determination of positive and negative samples.
[0167] In one embodiment, the posture information includes the pitch angle, yaw angle, and rotation angle of the face.
[0168] When the ratio of the first reference frame to the initial sample image is less than the first preset threshold and greater than the second preset threshold, the specific coordinate point in the first reference frame is moved according to the position information of the first reference frame and within the third set range to obtain the second reference frame; the first preset threshold is greater than the second preset threshold.
[0169] The first preset threshold and the second preset threshold can be set according to actual application scenarios. For example, the first preset threshold is set to 50%, and the second preset threshold can be 0. In other words, if the ratio of the first reference frame to the initial sample image is less than 50% and greater than 0, it is determined that the person is not staring at the screen.
[0170] Here, if the ratio of the first reference frame to the initial sample image is less than a first preset threshold and greater than a second preset threshold, the sample label to be marked can be determined to be a third category label, which is a label used to characterize non-gaze. It should be noted that this is only the determination of the sample label to be marked, not the marking process. Based on the current determination that the sample label to be marked is a third category label, the corresponding third set range is determined. The mapping relationship between the third category label and the third set range is pre-set.
[0171] Then, based on the position information of the first reference frame, the specific coordinate points in the first reference frame are moved within a third set range to obtain a second reference frame. Here, the third set range corresponding to the third category label may be a pre-set movement range for the specific coordinate points. The specific coordinate points may be vertices of the outer contour of the first reference frame, such as Vertex 1 in the upper left corner, Vertex 2 in the upper right corner, Vertex 3 in the lower left corner, and Vertex 4 in the lower right corner. In this embodiment, Vertex 1 in the upper left corner and Vertex 4 in the lower right corner may be selected as the specific coordinate points for the diversified update of the reference frame.
[0172] For example, the coordinates of the vertex 1 in the upper left corner are (lt x ,lt y ), the coordinates of vertex 4 in the lower right corner are (rb x ,rb y ); Let the coordinates of vertex 1 (lt x ,lt y ) has a random moving value of δ x , δ y ; The coordinates of vertex 4 (rb x ,rb y )’s random moving value σ x , σy , the value can be positive or negative. After random movement, the coordinates of the new vertex 1 are (lt1 x ,lt1 y ), that is: lt1 x =lt x +δ x lt1 y =lt y +δ y
[0173] The coordinates of the new vertex 4 are (rb1 x ,rb1 y ), that is: rb1 x =rb x +σ x rb1 y =rb y +σ y
[0174] Among them, the third setting range includes δ x , δ y , σ x and σ y The range of, for example, δ x ∈[-w2,-w3],δ y ∈[-h2,-h3],σ x ∈[w3,w2],σ y ∈[h3,h2]. Where w2=(rb x -lt x )×2.5;w3=(rb x -lt x )×0.8;h2=(rb y -lt y )×2.5, h3=(rb y -lt y ) × 0.8. When a specific coordinate point is moved within the third set range, the corresponding first reference frame expands outward, and the expansion range is larger than that of the first set range, ensuring a better non-gaze effect of the face within the second reference frame. Based on the non-gaze effect corresponding to the second set range, the diversity of the face non-gaze effect samples is further improved, thereby improving the accuracy of the subsequent determination of positive and negative samples.
[0175] The present disclosure is combined with the above embodiments, and refines the gaze and non-gaze states (for example, the state corresponding to the first category label belonging to the gaze state, the state corresponding to the second category label belonging to the non-gaze state, and the state corresponding to the third category label) according to the ratio of the first reference frame to the initial sample image and the posture information of the face, thereby improving the accuracy of subsequent determination of positive and negative samples.
[0176] S216 , marking sample labels for a portion of the sample images of the initial sample image located in the second reference frame according to the first reference frame and the second reference frame, and using the portion of the sample images marked with sample labels as training images.
[0177] With respect to the specific implementation process of the above-mentioned step S215, it is determined that the sample labels include a first category label representing gaze, and a second category label and a third category label representing non-gaze; wherein, the first category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is less than or equal to the first set value, the yaw angle of the face is less than or equal to the second set value, or the rotation angle of the face is less than or equal to the third set value; the second category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is greater than the first set value, the yaw angle of the face is greater than the second set value, or the rotation angle of the face is greater than the third set value; the third category label indicates that the ratio of the second reference frame in the training image to the training image is less than the first preset threshold and greater than the second preset threshold.
[0178] In one embodiment, when a ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, a pitch angle of the face is less than or equal to a first set value, a yaw angle of the face is less than or equal to a second set value, and a rotation angle of the face is less than or equal to a third set value, a specific process of marking the first category label includes:
[0179] The overlapping area of the second reference frame and the first reference frame may be determined based on the position information of the second reference frame and the position information of the first reference frame.
[0180] It should be noted that since the second reference frame is obtained by expanding the first reference frame, the larger the overlapping area between the second reference frame and the first reference frame, the greater the proportion of the face in the second reference frame.
[0181] In order to balance the number of positive samples, when the first ratio of the overlapping area to the area of the second reference frame is greater than or equal to the third preset threshold, the first preset range is generated according to the first preset probability, the second preset range is generated according to the second preset probability, the third preset range is generated according to the third preset probability, and the fourth preset range is generated according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1.
[0182] Afterwards, according to the preset range controlled by the above-mentioned preset probability, when the first ratio is within the first preset range, the part of the sample images in the initial sample image located in the second reference frame is determined and marked as the first category label, so that the first ratio is within the first preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the first preset probability; when the first ratio is within the second preset range, the part of the sample images in the initial sample image located in the second reference frame is determined and marked as the first category label, so that the first ratio is within the second preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is The ratio is a second preset probability; when the first ratio is within a third preset range, the part of the sample image in the initial sample image located in the second reference frame is determined and marked as the first category label, so that the first ratio is within the third preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the third preset probability; when the first ratio is within a fourth preset range, the part of the sample image in the initial sample image located in the second reference frame is determined and marked as the first category label, so that the first ratio is within the fourth preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the fourth preset probability.
[0183] The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.
[0184] Exemplarily, the third preset threshold is set to 0.5; if the first ratio is greater than or equal to 0.5, then the first preset range of 0.5 to 0.7 (excluding 0.7) is selected with a probability of 50% (first preset probability), 0.7 to 0.8 (excluding 0.8) is selected with a probability of 30% (second preset probability), 0.8 to 0.9 (excluding 0.9) is selected with a probability of 10% (third preset probability), and 0.9 to 1.0 is selected with a probability of 10% (fourth preset probability). When the first ratio is within the first preset range, some sample images located in the second reference frame are marked as the first category label, so that the second ratio between the number of training images with the first ratio within the first preset range and marked with the first category label and the total number of training images marked with the first category label in the training data set is 50%. When the first ratio is within the second preset range, some sample images located in the second reference frame are marked as having the first category label, so that the final second ratio between the number of training images with the first ratio within the second preset range and the first category label and the total number of training images with the first category label in the training dataset is 30%. When the first ratio is within the third preset range, some sample images located in the second reference frame are marked as having the first category label, so that the final second ratio between the number of training images with the first ratio within the third preset range and the first category label and the total number of training images with the first category label in the training dataset is 10%. When the first ratio is within the fourth preset range, some sample images located in the second reference frame are marked as having the first category label, so that the final second ratio between the number of training images with the first ratio within the fourth preset range and the first category label and the total number of training images with the first category label in the training dataset is 10%.
[0185] This embodiment controls the proportions of positive samples corresponding to different first ratios to avoid having only facial features on every positive sample training image, thereby improving the diversity of positive samples.
[0186] In one embodiment, when the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is greater than a first set value, the yaw angle of the face is greater than a second set value, and the rotation angle of the face is greater than a third set value, the specific process of marking the second category label includes:
[0187] The overlapping area of the second reference frame and the first reference frame may be determined based on the position information of the second reference frame and the position information of the first reference frame.
[0188] It should be noted that since the second reference frame is obtained by expanding the first reference frame, the larger the overlapping area between the second reference frame and the first reference frame, the greater the proportion of the face in the second reference frame.
[0189] When the third ratio of the overlapping area to the area of the second reference frame is greater than or equal to the third preset threshold, the first preset range is generated according to the first preset probability, the second preset range is generated according to the second preset probability, the third preset range is generated according to the third preset probability, and the fourth preset range is generated according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1.
[0190] Afterwards, according to the preset range controlled by the above-mentioned preset probability, when the third ratio is within the first preset range, the part of the sample images in the initial sample image located in the second reference frame is determined and marked as the second category label, so that the third ratio is within the first preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the first preset probability; when the third ratio is within the second preset range, the part of the sample images in the initial sample image located in the second reference frame is determined and marked as the second category label, so that the third ratio is within the second preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is The ratio is a second preset probability; when the third ratio is within the third preset range, the part of the sample images in the initial sample image located in the second reference frame is determined and marked as the second category label, so that the third ratio is within the third preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the third preset probability; when the third ratio is within the fourth preset range, the part of the sample images in the initial sample image located in the second reference frame is determined and marked as the second category label, so that the third ratio is within the fourth preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the fourth preset probability.
[0191] The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.
[0192] Exemplarily, the third preset threshold is set to 0.5; if the third ratio is greater than or equal to 0.5, then the first preset range of 0.5 to 0.7 (excluding 0.7) is selected with a probability of 50% (first preset probability), 0.7 to 0.8 (excluding 0.8) is selected with a probability of 30% (second preset probability), 0.8 to 0.9 (excluding 0.9) is selected with a probability of 10% (third preset probability), and 0.9 to 1.0 is selected with a probability of 10% (fourth preset probability). When the third ratio is within the first preset range, some sample images located in the second reference frame are marked as the second category label, so that the fourth ratio between the number of training images with the third ratio within the first preset range and marked with the second category label and the total number of training images marked with the second category label in the training data set is 50%. When the third ratio is within the second preset range, some sample images located in the second reference frame are marked as having the second category label, so that a fourth ratio between the number of training images with the third ratio within the second preset range and the second category label and the total number of training images with the second category label in the training dataset is 30%. When the third ratio is within the third preset range, some sample images located in the second reference frame are marked as having the second category label, so that a fourth ratio between the number of training images with the third ratio within the third preset range and the second category label and the total number of training images with the second category label in the training dataset is 10%. When the third ratio is within the fourth preset range, some sample images located in the second reference frame are marked as having the second category label, so that a fourth ratio between the number of training images with the third ratio within the fourth preset range and the second category label and the total number of training images with the second category label in the training dataset is 10%.
[0193] The above is the process of generating negative samples corresponding to the second category label. This embodiment controls the proportion of negative samples corresponding to different third ratios so that the generated training images have a large proportion of human faces and are in a non-gaze state, thereby improving the diversity of negative samples.
[0194] In one embodiment, when the ratio of the first reference frame to the initial sample image is less than a first preset threshold and greater than a second preset threshold, the specific process of marking the third category label includes:
[0195] The overlapping area of the second reference frame and the first reference frame may be determined based on the position information of the second reference frame and the position information of the first reference frame.
[0196] When the fifth ratio of the overlapping area to the area of the second reference frame is less than the third preset threshold, the fifth preset range is generated according to the fifth preset probability, the sixth preset range is generated according to the sixth preset probability, and the seventh preset range is generated according to the seventh preset probability; the fifth preset probability is greater than the sixth preset probability, and the sixth preset probability is greater than the seventh preset probability; the sum of the fifth preset probability, the sixth preset probability and the seventh preset probability is 1.
[0197] Afterwards, according to the preset range controlled by the above-mentioned preset probability, the part of the sample image of the initial sample image located in the second reference frame is marked as the third category label. When the fifth ratio is within the fifth preset range, the part of the sample image in the initial sample image located in the second reference frame is determined and marked as the third category label, so that the fifth ratio is within the fifth preset range, and the sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the fifth preset probability; when the fifth ratio is within the sixth preset range, the part of the sample image in the initial sample image located in the second reference frame is determined and marked as the third category label, so that the sixth ratio is within the sixth preset range. Assume that the sixth ratio between the number of training images within the range and marked with the third category label and the total number of training images marked with the third category label in the training data set is the sixth preset probability; when the fifth ratio is within the seventh preset range, determine and mark some sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within the seventh preset range, and the sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the seventh preset probability; wherein, the data within the seventh preset range is smaller than the data within the sixth preset range, and the data within the sixth preset range is smaller than the data within the fifth preset range.
[0198] Exemplarily, the third preset threshold is set to 0.5; if the fifth ratio is less than 0.5, then the fifth preset range of 0.25 to 0.5 (excluding 0.5) is selected with a probability of 60% (the fifth preset probability), 0.15 to 0.25 (excluding 0.25) is selected with a probability of 30% (the sixth preset probability), and 0.01 to 0.15 is selected with a probability of 10% (the seventh preset probability). When the fifth ratio is within the fifth preset range, some sample images located in the second reference frame are marked as having the third category label, so that the sixth ratio between the number of training images with the fifth ratio within the fifth preset range and marked with the third category label and the total number of training images with the third category label in the training dataset is 60%. When the fifth ratio is within the sixth preset range, some sample images located in the second reference frame are marked as having the third category label, so that the sixth ratio between the number of training images with the fifth ratio within the sixth preset range and marked with the third category label and the total number of training images with the third category label in the training dataset is 30%. When the fifth ratio is within the seventh preset range, some sample images located in the second reference frame are marked as third category labels, so that the sixth ratio between the number of training images with the fifth ratio within the seventh preset range and marked with the third category label and the total number of training images marked with the third category label in the training data set is 10%.
[0199] The above is the process for generating negative samples corresponding to the third category label. This embodiment controls the proportion of negative samples corresponding to different ratios of overlapping areas to the area of the second reference frame, ensuring that the generated training images contain a relatively small proportion of faces and are in non-gaze states, thereby improving the diversity of negative samples.
[0200] S217 : Mark the initial sample image with a fourth category label, and use the initial sample image marked with the fourth category label as a training image.
[0201] The sample labels also include a fourth category label representing non-gaze.
[0202] In some embodiments, after executing S212, it also includes: according to the size of the initial sample image and in accordance with the fourth set range, a target center point can be determined in the initial sample image, and a third reference frame can be determined with the target center point as the center; if the third reference frame has no overlap with any first reference frame in the initial sample image, the portion of the sample image of the initial sample image located in the third reference frame is marked as a fourth category label, and the portion of the sample image marked with the fourth category label is used as a training image.
[0203] Here, the target center point is randomly set on the initial sample image to define the third reference frame, and the sample images corresponding to the third reference frame without a face are used as negative sample training images.
[0204] For example, the width of the initial sample image is w' and the height is h'; the coordinates of the randomly generated target center point are (cnew x ,cnew y ), where cnew x ∈[0.05×w', 0.95×w'], cnew y ∈[0.05×h', 0.95×h']. The randomly generated third reference frame has a width of wnew and a height of hnew, where wnew∈[w'×0.1,w'×0.9] and hnew∈[h'×0.1,h'×0.9]. This determines the third reference frame. If the third reference frame does not overlap with any first reference frame in the initial sample image, it means that there is no face in the third reference frame. In this case, the portion of the initial sample image that is within the third reference frame can be labeled with the fourth category label, and the portion of the sample image labeled with the fourth category label can be used as the training image.
[0205] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.
[0206] It should be noted that, in addition to the gaze detection and device wake-up functions mentioned in the embodiments of this disclosure, portable smart terminal application services in smart homes and wearable devices can also adopt this energy-efficient storage and computing approach to significantly reduce costs. Furthermore, with the development of storage and computing technology, it can support larger deep learning models and more extensive and universal services. Furthermore, because different storage and computing platforms support different functions and operators, this disclosure can design and build a network based on the actual hardware platform and is not limited to the target neural network model proposed in this disclosure.
[0207] In addition, the embodiments of the present disclosure also provide another control method for a monitoring system. The control method for the monitoring system can be performed by a monitoring system including a detection device and a terminal. The detection device can be, for example, a memory-computing integrated chip integrated with Flash or SRAM. The control method for the monitoring system includes:
[0208] The detection device obtains a detection image, and uses a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and sends the recognition result to the terminal.
[0209] The terminal determines the display state based on the recognition result.
[0210] The embodiment of the present disclosure provides a control method for a monitoring system, which uses high-precision gaze recognition results to control the display state of the terminal. For example, if no one is looking at the screen for a long time, the terminal automatically enters a sleep state, thereby saving system power consumption; if after entering the sleep state, the recognition result indicates that someone is looking at the screen, the system can be automatically restarted to restore the normal display of the screen. The present disclosure is applied to intelligent wake-up scenarios in a gaze environment (such as an environment where a person is staring at a display screen). While saving system power consumption, it combines neural network gaze recognition technology to improve the recognition accuracy of people looking at the screen, thereby more accurately controlling the timing of the terminal entering sleep or actively waking up.
[0211] In some embodiments, the terminal includes a display module and a main control module; the terminal determines the display state based on the recognition result, including:
[0212] When the main control module determines that the recognition result indicates that the object is gazing, the main control module sends a wake-up request to the display module; when the main control module determines that the recognition result indicates that the object is not gazing and the time between the current system time and the last time the display module was awakened is greater than a preset time, the main control module sends a sleep request to the display module; the display module responds to the wake-up request and determines that the display state is normal display; and responds to the sleep request and determines that the display state is sleep.
[0213] Here, the preset duration can be set based on experience. It should be noted that a reasonable preset duration is set so that the terminal enters the sleep state appropriately in actual application scenarios. If the preset duration is too short, it will lead to frequent switching between sleep and wake-up states, which is not conducive to reducing power consumption. If the preset duration is too long, it will not be able to maximize the reduction of terminal power consumption.
[0214] Here, the display module may be, for example, a display screen, and the main control module may be, for example, a system on chip (SOC). The main control module and the detection device may communicate using a protocol such as UART.
[0215] Figure 8a is an exemplary terminal workflow diagram provided by an embodiment of the present disclosure. As shown in Figure 8a, the main control module: S31, start system initialization; S32, send a startup instruction to the detection device, and receive a response confirmation from the detection device, and execute S33 in sequence; S33, reset the timer; S34, poll the serial port to see if a recognition result indicating gaze is received, and send a response confirmation to the detection device; if so, return to execute step S33; if not, execute S35; S35, determine whether the preset time is exceeded; if so, execute S36; if not, return to execute S34; S36, the main control SOC enters the sleep state, and sends a sleep request to the display module, and executes S37 in sequence; S37, poll the serial port to see if a recognition result indicating gaze is received, and send a response confirmation to the detection device; if so, execute S38; if not, repeat S37; S38, wake up the main control SOC, send a wake-up request to the display module, and return to step S33.
[0216] Figure 8b is an exemplary workflow diagram of a detection device provided in an embodiment of the present disclosure. As shown in Figure 8b, the detection device: S301, starts system initialization; S302, polls the serial port to receive a startup instruction sent by the main control module, and sends a response confirmation to the main control module, and executes S303 in sequence; S303, identifies whether someone is staring at the screen, and sends the identification result to the terminal.
[0217] In some embodiments, the target neural network model is trained by the following steps:
[0218] Training the teacher network learning model to be trained according to the acquired training data set to obtain a trained teacher network learning model;
[0219] According to the training data set and the trained teacher network learning model, the knowledge distillation training method is used to train the student network learning model to be trained, and the trained student network learning model is obtained as the target neural network model.
[0220] In some embodiments, the training dataset includes a plurality of training images annotated with sample labels;
[0221] Based on the training data set and the trained teacher network learning model, the knowledge distillation training method is used to train the student network learning model to be trained, and the trained student network learning model is obtained as the target neural network model, including:
[0222] Inputting the training image into the trained teacher network learning model to obtain a first output result of the trained teacher network learning model;
[0223] Inputting the training image into the student network learning model to be trained to obtain a second output result of the student network learning model to be trained;
[0224] Determining a first loss function according to the first output result and the second output result;
[0225] Determining a second loss function based on the second output result and the sample label of the training image;
[0226] According to the first loss function and the second loss function, a weighted loss function is obtained;
[0227] The parameters of the student network learning model to be trained are adjusted according to the weighted loss function until the weighted loss function converges, and the trained student network learning model is obtained as the target neural network model.
[0228] In some embodiments, the training dataset is determined by the following steps:
[0229] Obtaining an original data set; the original data set includes multiple initial sample images;
[0230] Performing face recognition on the initial sample image, and if a face is present, determining a first reference frame containing the face and determining key point information of the face;
[0231] Determine the facial posture information based on key point information;
[0232] According to the posture information and the ratio of the first reference frame to the initial sample image, the first reference frame is updated to obtain a second reference frame;
[0233] According to the first reference frame and the second reference frame, marking sample labels for a portion of the sample image of the initial sample image located in the second reference frame, and using the portion of the sample image marked with sample labels as a training image;
[0234] The sample labels include a first category label representing gaze, and a second category label and a third category label representing non-gaze; wherein,
[0235] The first category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is less than or equal to the first set value, the yaw angle of the face is less than or equal to the second set value, and the rotation angle of the face is less than or equal to the third set value;
[0236] The second category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is greater than the first set value, the yaw angle of the face is greater than the second set value, and the rotation angle of the face is greater than the third set value;
[0237] The third category label indicates that the ratio of the second reference frame in the training image to the training image is less than the first preset threshold and greater than the second preset threshold.
[0238] In some embodiments, the posture information includes at least the pitch angle, yaw angle, and rotation angle of the face;
[0239] The first reference frame is updated according to the posture information and the ratio of the first reference frame to the initial sample image to obtain a second reference frame, including:
[0240] When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the yaw angle of the face is less than or equal to a second set value, and the rotation angle of the face is less than or equal to a third set value, the specific coordinate point in the first reference frame is moved according to the position information of the first reference frame within the first set range to obtain the second reference frame.
[0241] In some embodiments, according to the first reference frame and the second reference frame, labeling a sample label for a portion of the sample image where the initial sample image is located in the second reference frame, and using the portion of the sample image with the sample label as a training image includes:
[0242] determining an overlapping area between the second reference frame and the first reference frame based on the position information of the second reference frame and the position information of the first reference frame;
[0243] When a first ratio of the overlapping area to the area of the second reference frame is greater than or equal to a third preset threshold, a first preset range is generated according to the first preset probability, a second preset range is generated according to the second preset probability, a third preset range is generated according to the third preset probability, and a fourth preset range is generated according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; and the sum of the first preset probability, the second preset probability, the third preset probability, and the fourth preset probability is 1;
[0244] When the first ratio is within a first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the first category label, so that the first ratio is within the first preset range and a second ratio between the number of training images marked with the first category label and the total number of all training images marked with the first category label in the training data set is a first preset probability;
[0245] When the first ratio is within a second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the first category label, so that the first ratio is within the second preset range and a second ratio between the number of training images marked with the first category label and the total number of all training images marked with the first category label in the training data set is a second preset probability;
[0246] When the first ratio is within a third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the first category label, so that the first ratio is within the third preset range and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is a third preset probability;
[0247] When the first ratio is within a fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the first category label, so that the first ratio is within the fourth preset range and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is a fourth preset probability;
[0248] The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.
[0249] In some embodiments, the posture information includes at least the pitch angle, yaw angle, and rotation angle of the face;
[0250] The first reference frame is updated according to the posture information and the ratio of the first reference frame to the initial sample image to obtain a second reference frame, including:
[0251] When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is greater than a first set value, the yaw angle of the face is greater than a second set value, and the rotation angle of the face is greater than a third set value, the specific coordinate point in the first reference frame is moved according to the position information of the first reference frame and within a second set range to obtain a second reference frame.
[0252] In some embodiments, according to the first reference frame and the second reference frame, labeling a sample label for a portion of the sample image where the initial sample image is located in the second reference frame, and using the portion of the sample image with the sample label as a training image includes:
[0253] determining an overlapping area between the second reference frame and the first reference frame based on the position information of the second reference frame and the position information of the first reference frame;
[0254] When a third ratio of the overlapping area to the area of the second reference frame is greater than or equal to a third preset threshold, a first preset range is generated according to the first preset probability, a second preset range is generated according to the second preset probability, a third preset range is generated according to the third preset probability, and a fourth preset range is generated according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; and the sum of the first preset probability, the second preset probability, the third preset probability, and the fourth preset probability is 1;
[0255] When the third ratio is within a first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the second category label, so that the third ratio is within the first preset range and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is a first preset probability;
[0256] When the third ratio is within a second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the second category label, so that the third ratio is within the second preset range and a fourth ratio between the number of training images marked with the second category label and the total number of all training images marked with the second category label in the training data set is a second preset probability;
[0257] When the third ratio is within a third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the second category label, so that the third ratio is within the third preset range and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is a third preset probability;
[0258] When the third ratio is within a fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the second category label, so that the third ratio is within the fourth preset range and a fourth ratio between the number of training images marked with the second category label and the total number of all training images marked with the second category label in the training data set is a fourth preset probability;
[0259] The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.
[0260] In some embodiments, the posture information includes at least the pitch angle, yaw angle, and rotation angle of the face;
[0261] The first reference frame is updated according to the posture information and the ratio of the first reference frame to the initial sample image to obtain a second reference frame, including:
[0262] When the ratio of the first reference frame to the initial sample image is less than the first preset threshold and greater than the second preset threshold, the specific coordinate point in the first reference frame is moved according to the position information of the first reference frame and within the third set range to obtain the second reference frame; the first preset threshold is greater than the second preset threshold.
[0263] In some embodiments, according to the first reference frame and the second reference frame, labeling a sample label for a portion of the sample image where the initial sample image is located in the second reference frame, and using the portion of the sample image with the sample label as a training image includes:
[0264] determining an overlapping area between the second reference frame and the first reference frame based on the position information of the second reference frame and the position information of the first reference frame;
[0265] If a fifth ratio of the overlapping area to the area of the second reference frame is less than the third preset threshold, a fifth preset range is generated according to the fifth preset probability, a sixth preset range is generated according to the sixth preset probability, and a seventh preset range is generated according to the seventh preset probability; the fifth preset probability is greater than the sixth preset probability, and the sixth preset probability is greater than the seventh preset probability; and the sum of the fifth preset probability, the sixth preset probability, and the seventh preset probability is 1;
[0266] When the fifth ratio is within a fifth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the fifth ratio is within the fifth preset range and a sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is a fifth preset probability;
[0267] When the fifth ratio is within a sixth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within the sixth preset range and a sixth ratio between the number of training images marked with the third category label and the total number of all training images marked with the third category label in the training data set is a sixth preset probability;
[0268] When the fifth ratio is within a seventh preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within a seventh preset range and the sixth ratio between the number of training images marked with the third category label and the total number of all training images marked with the third category label in the training data set is a seventh preset probability;
[0269] The data within the seventh preset range is smaller than the data within the sixth preset range, and the data within the sixth preset range is smaller than the data within the fifth preset range.
[0270] In some embodiments, the sample labels further include a fourth category label representing non-gaze;
[0271] In the step of obtaining the training data set, when face recognition is performed on the initial sample image to determine that no face exists, the step further includes: marking the initial sample image with a fourth category label, and using the initial sample image marked with the fourth category label as a training image.
[0272] In some embodiments, the sample labels further include a fourth category label representing non-gaze;
[0273] After performing face recognition on the initial sample image to determine the presence of a face and determining a first reference frame containing the face, the method further includes:
[0274] Determining a target center point in the initial sample image according to the size of the initial sample image and within a fourth set range, and determining a third reference frame with the target center point as the center;
[0275] If the third reference frame does not overlap with any first reference frame in the initial sample image, the portion of the initial sample image located in the third reference frame is marked as a fourth category label, and the portion of the sample image marked with the fourth category label is used as a training image.
[0276] In addition, the embodiment of the present disclosure also provides a detection device corresponding to the control method of the monitoring system of the first aspect. Since the principle of solving the problem by the detection device in the embodiment of the present disclosure is similar to the control method of the monitoring system of the first aspect of the embodiment of the present disclosure, the implementation of the detection device can refer to the implementation of the corresponding method of the first aspect, and the repeated parts will not be repeated.
[0277] FIG9 is a schematic diagram of a detection device provided by an embodiment of the present disclosure. As shown in FIG9 , the detection device includes a data processing module 91. The detection device may be, for example, a memory-computing integrated chip integrated with Flash or SRAM.
[0278] The data processing module 91 is configured to acquire a detection image, and perform gaze recognition on the detection image using a pre-trained target neural network model to obtain a recognition result of the detection image; and send the recognition result to the terminal so that the terminal determines the display state at least based on the recognition result.
[0279] In some embodiments, the detection device also includes a model training module 92; the model training module 92 is configured to train the teacher network learning model to be trained based on the acquired training data set to obtain a trained teacher network learning model; based on the training data set and the trained teacher network learning model, the student network learning model to be trained is trained using the knowledge distillation training method to obtain a trained student network learning model as the target neural network model.
[0280] In some embodiments, the training data set includes multiple training images with labeled sample labels; the model training module 92 is specifically configured to input the training image into the trained teacher network learning model to obtain a first output result of the trained teacher network learning model; input the training image into the student network learning model to be trained to obtain a second output result of the student network learning model to be trained; determine a first loss function based on the first output result and the second output result; determine a second loss function based on the second output result and the sample label of the training image; obtain a weighted loss function based on the first loss function and the second loss function; adjust the parameters of the student network learning model to be trained according to the weighted loss function until the weighted loss function converges, and obtain the trained student network learning model as the target neural network model.
[0281] In some embodiments, the detection device further includes a sample generation module 93, which is configured to obtain an original data set; the original data set includes multiple initial sample images; perform face recognition on the initial sample image, and if a face exists, determine a first reference frame containing the face, and determine key point information of the face; determine the posture information of the face based on the key point information; update the first reference frame based on the posture information and the ratio of the first reference frame to the initial sample image to obtain a second reference frame; based on the first reference frame and the second reference frame, mark sample labels for the portion of the sample image of the initial sample image located in the second reference frame, and use the portion of the sample image marked with sample labels as training images; the sample labels include a first category label representing gaze, and A second category label and a third category label characterizing non-gaze; wherein the first category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is less than or equal to the first set value, the yaw angle of the face is less than or equal to the second set value, and the rotation angle of the face is less than or equal to the third set value; the second category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is greater than the first set value, the yaw angle of the face is greater than the second set value, and the rotation angle of the face is greater than the third set value; the third category label indicates that the ratio of the second reference frame in the training image to the training image is less than the first preset threshold and greater than the second preset threshold.
[0282] In some embodiments, the posture information includes at least the pitch angle, yaw angle and rotation angle of the face; the sample generation module 93 is specifically configured to move the specific coordinate point in the first reference frame according to the position information of the first reference frame within the first set range to obtain the second reference frame when the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to the first set value, the yaw angle of the face is less than or equal to the second set value, and the rotation angle of the face is less than or equal to the third set value.
[0283] Furthermore, the sample generation module 93 is specifically configured to determine the overlapping area of the second reference frame and the first reference frame based on the position information of the second reference frame and the position information of the first reference frame; when the first ratio of the overlapping area to the area of the second reference frame is greater than or equal to the third preset threshold, generate the first preset range according to the first preset probability, generate the second preset range according to the second preset probability, generate the third preset range according to the third preset probability, and generate the fourth preset range according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1; when the first ratio is within the first preset range, determine and mark the part of the sample image located in the second reference frame in the initial sample image as the first category label, so that the first ratio is within the first preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the first preset probability; when the first ratio is within the second preset range, determine and mark the part of the sample image located in the second reference frame in the initial sample image as the first category label. Part of the sample images in the reference frame are labeled with the first category, so that the first ratio is within the second preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is a second preset probability; when the first ratio is within the third preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the first category label, so that the first ratio is within the third preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is a third preset probability; when the first ratio is within the fourth preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the first category label, so that the first ratio is within the fourth preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is a fourth preset probability; the data in the first preset range is smaller than the data in the second preset range, the data in the second preset range is smaller than the data in the third preset range, and the data in the third preset range is smaller than the data in the fourth preset range.
[0284] In some embodiments, the posture information includes at least the pitch angle, yaw angle and rotation angle of the face; the sample generation module 93 is specifically configured to move the specific coordinate point in the first reference frame according to the position information of the first reference frame and within the second set range to obtain the second reference frame when the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is greater than a first set value, the yaw angle of the face is greater than a second set value, and the rotation angle of the face is greater than a third set value.
[0285] Furthermore, the sample generation module 93 is specifically configured to determine the overlapping area of the second reference frame and the first reference frame based on the position information of the second reference frame and the position information of the first reference frame; when the third ratio of the overlapping area to the area of the second reference frame is greater than or equal to the third preset threshold, generate the first preset range according to the first preset probability, generate the second preset range according to the second preset probability, generate the third preset range according to the third preset probability, and generate the fourth preset range according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1; when the third ratio is within the first preset range, determine and mark the part of the sample image located in the second reference frame in the initial sample image as the second category label, so that the third ratio is within the first preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the first preset probability; when the third ratio is within the second preset range, determine and mark the part of the sample image located in the second reference frame in the initial sample image as the second category label. Part of the sample images in the reference frame are labeled as the second category, so that the third ratio is within the second preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the second preset probability; when the third ratio is within the third preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the second category label, so that the third ratio is within the third preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the third preset probability; when the third ratio is within the fourth preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the second category label, so that the third ratio is within the fourth preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the fourth preset probability; the data in the first preset range is smaller than the data in the second preset range, the data in the second preset range is smaller than the data in the third preset range, and the data in the third preset range is smaller than the data in the fourth preset range.
[0286] In some embodiments, the posture information includes at least the pitch angle, yaw angle and rotation angle of the face; the sample generation module 93 is specifically configured to move the specific coordinate point in the first reference frame according to the position information of the first reference frame and within a third set range to obtain the second reference frame when the ratio of the first reference frame to the initial sample image is less than a first preset threshold and greater than a second preset threshold; the first preset threshold is greater than the second preset threshold.
[0287] Furthermore, the sample generation module 93 is specifically configured to determine the overlapping area of the second reference frame and the first reference frame based on the position information of the second reference frame and the position information of the first reference frame; when the fifth ratio of the overlapping area to the area of the second reference frame is less than the third preset threshold, generate a fifth preset range according to the fifth preset probability, generate a sixth preset range according to the sixth preset probability, and generate a seventh preset range according to the seventh preset probability; the fifth preset probability is greater than the sixth preset probability, and the sixth preset probability is greater than the seventh preset probability; the sum of the fifth preset probability, the sixth preset probability and the seventh preset probability is 1; when the fifth ratio is within the fifth preset range, determine and mark the part of the sample image located in the second reference frame in the initial sample image as the third category label, so that the number of training images with the fifth ratio within the fifth preset range and marked with the third category label is the same as the number of all training images marked with the third category label in the training data set. The sixth ratio between the total numbers is the fifth preset probability; when the fifth ratio is within the sixth preset range, the part of the sample images in the initial sample image located in the second reference frame is determined and marked as the third category label, so that the sixth ratio is within the sixth preset range, and the sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the sixth preset probability; when the fifth ratio is within the seventh preset range, the part of the sample images in the initial sample image located in the second reference frame is determined and marked as the third category label, so that the sixth ratio is within the seventh preset range, and the sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the seventh preset probability; the data in the seventh preset range is smaller than the data in the sixth preset range, and the data in the sixth preset range is smaller than the data in the fifth preset range.
[0288] In some embodiments, when face recognition is performed on the initial sample image and it is determined that no face exists, the initial sample image is labeled with a fourth category label, and the initial sample image labeled with the fourth category label is used as a training image.
[0289] In some embodiments, the sample label also includes a fourth category label representing non-gaze; the sample generation module 93 is also configured to determine a target center point in the initial sample image according to the size of the initial sample image and in accordance with a fourth set range, and determine a third reference frame with the target center point as the center; if the third reference frame has no overlap with any first reference frame in the initial sample image, then the portion of the sample image of the initial sample image located in the third reference frame is marked as a fourth category label, and the portion of the sample image marked with the fourth category label is used as a training image.
[0290] In addition, the embodiment of the present disclosure also provides a control device for the monitoring system corresponding to the control method of the monitoring system of the second aspect. Since the principle of solving the problem by the control device of the monitoring system in the embodiment of the present disclosure is similar to the control method of the monitoring system of the second aspect of the embodiment of the present disclosure, the implementation of the control device of the monitoring system can refer to the implementation of the corresponding method of the second aspect, and the repeated parts will not be repeated.
[0291] Figure 10 is a schematic diagram of a control device of a monitoring system according to an embodiment of the present disclosure, as shown in Figure 10 , the detection device 101 and the terminal 102. The detection device 101 may be, for example, a memory-computing integrated chip integrated with Flash or SRAM.
[0292] The detection device 101 is configured to obtain a detection image, and perform gaze recognition on the detection image using a pre-trained target neural network model to obtain a recognition result of the detection image; and send the recognition result to the terminal;
[0293] The terminal 102 is configured to determine a display state based on the recognition result.
[0294] In some embodiments, the terminal 102 includes a display module 1022 and a main control module 1021; the main control module 1021 is configured to send a wake-up request to the display module when it is determined that the recognition result indicates that the object is gazing; when it is determined that the recognition result indicates that the object is not gazing, and the current system time is greater than the preset time from the last time the display module was awakened, send a sleep request to the display module; the display module 1022 is configured to respond to the wake-up request and determine that the display state is normal display; and respond to the sleep request and determine that the display state is sleep.
[0295] Figure 11 is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. As shown in Figure 11, the computer device provided in an embodiment of the present disclosure includes: one or more processors 111, memory 112, and one or more I / O interfaces 113. Memory 112 stores one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the control method of the monitoring system as described in any of the above embodiments; one or more I / O interfaces 113 are connected between the processor and memory and are configured to enable information exchange between the processor and memory.
[0296] Among them, the processor 111 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 112 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) 113 is connected between the processor 111 and the memory 112, and can realize information interaction between the processor 111 and the memory 112, including but not limited to a data bus (Bus), etc.
[0297] In some embodiments, the processor 111 , the memory 112 , and the I / O interface 113 are connected to each other via a bus 114 , and further connected to other components of the computing device.
[0298] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium is further provided. The non-transitory computer-readable storage medium stores a computer program, wherein when the program is executed by a processor, the program implements the steps of the control method of the monitoring system in any of the above embodiments.
[0299] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the system of the present disclosure are executed.
[0300] It should be noted that the computer non-transitory readable medium shown in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any non-transitory computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the non-transitory computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination thereof.
[0301] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the aforementioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two connected boxes can actually represent execution in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0302] It is understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Those skilled in the art may make various modifications and improvements without departing from the spirit and substance of the present disclosure, and such modifications and improvements are also considered to be within the scope of protection of the present disclosure.
Claims
1. A control method for a monitoring system, wherein: include: Acquire detection images; Using a pre-trained target neural network model, performing gaze recognition on the detection image to obtain a recognition result of the detection image; The recognition result is sent to a terminal so that the terminal determines a display state based at least on the recognition result.
2. The control method of the monitoring system according to claim 1, wherein: The target neural network model is trained by the following steps: Training the teacher network learning model to be trained according to the acquired training data set to obtain a trained teacher network learning model; According to the training data set and the trained teacher network learning model, the student network learning model to be trained is trained using a knowledge distillation training method to obtain a trained student network learning model as the target neural network model.
3. The control method of the monitoring system according to claim 2, wherein: The training data set includes a plurality of training images annotated with sample labels; The method of training the student network learning model to be trained by using the training method of knowledge distillation according to the training data set and the trained teacher network learning model to obtain the trained student network learning model as the target neural network model includes: Inputting the training image into the trained teacher network learning model to obtain a first output result of the trained teacher network learning model; Inputting the training image into the student network learning model to be trained to obtain a second output result of the student network learning model to be trained; Determine a first loss function according to the first output result and the second output result; Determining a second loss function according to the second output result and the sample label of the training image; Obtaining a weighted loss function according to the first loss function and the second loss function; The parameters of the student network learning model to be trained are adjusted according to the weighted loss function until the weighted loss function converges, thereby obtaining a trained student network learning model as the target neural network model.
4. The control method of the monitoring system according to claim 3, wherein: The training data set is determined by the following steps: Acquire an original data set; the original data set includes a plurality of initial sample images; Performing face recognition on the initial sample image, and if a face exists, determining a first reference frame containing the face, and determining key point information of the face; Determining facial posture information according to the key point information; According to the posture information and the ratio of the first reference frame to the initial sample image, the first reference frame is updated to obtain a second reference frame; According to the first reference frame and the second reference frame, marking the sample label for a portion of the sample image of the initial sample image located in the second reference frame, and using the portion of the sample image marked with the sample label as the training image; The sample labels include a first category label representing gaze, and a second category label and a third category label representing non-gaze; wherein, The first category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the deflection angle of the face is less than or equal to a second set value, or the rotation angle of the face is less than or equal to a third set value; The second category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is greater than the first set value, the deflection angle of the face is greater than the second set value, or the rotation angle of the face is greater than the third set value; The third category label indicates that a ratio of the second reference frame in the training image to the training image is less than a first preset threshold and greater than a second preset threshold.
5. The control method of the monitoring system according to claim 4, wherein: The posture information includes at least the pitch angle, yaw angle and rotation angle of the face; The updating of the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain the second reference frame includes: When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the yaw angle of the face is less than or equal to a second set value, and the rotation angle of the face is less than or equal to a third set value, the specific coordinate points in the first reference frame are moved within a first set range according to the position information of the first reference frame to obtain a second reference frame.
6. The control method of the monitoring system according to claim 5, wherein: The step of marking the sample labels for the portion of the sample images of the initial sample image located in the second reference frame according to the first reference frame and the second reference frame, and using the portion of the sample images marked with the sample labels as the training images, comprises: determining an overlapping area between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame; When the first ratio of the overlapping area to the area of the second reference frame is greater than or equal to a third preset threshold, a first preset range is generated according to a first preset probability, a second preset range is generated according to a second preset probability, a third preset range is generated according to a third preset probability, and a fourth preset range is generated according to a fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1; When the first ratio is within the first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the first preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the first preset probability; When the first ratio is within the second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the first category label, so that the first ratio is within the second preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the second preset probability; When the first ratio is within the third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the third preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is within the third preset probability; When the first ratio is within the fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the fourth preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the fourth preset probability; The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.
7. The control method of the monitoring system according to claim 4, wherein: The posture information includes at least the pitch angle, yaw angle and rotation angle of the face; The updating of the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain the second reference frame includes: When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is greater than a first set value, the deflection angle of the face is greater than a second set value, and the rotation angle of the face is greater than a third set value, the specific coordinate points in the first reference frame are moved within a second set range according to the position information of the first reference frame to obtain a second reference frame.
8. The control method of the monitoring system according to claim 7, wherein: The step of marking the sample labels for the portion of the sample images of the initial sample image located in the second reference frame according to the first reference frame and the second reference frame, and using the portion of the sample images marked with the sample labels as the training images, comprises: determining an overlapping area between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame; When a third ratio of the overlapping area to the area of the second reference frame is greater than or equal to a third preset threshold, a first preset range is generated according to a first preset probability, a second preset range is generated according to a second preset probability, a third preset range is generated according to a third preset probability, and a fourth preset range is generated according to a fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; and the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1; When the third ratio is within the first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a second category label, so that the third ratio is within the first preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the first preset probability; When the third ratio is within the second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the second category label, so that the third ratio is within the second preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the second preset probability; When the third ratio is within the third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a second category label, so that the third ratio is within the third preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the third preset probability; When the third ratio is within the fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a second category label, so that the third ratio is within the fourth preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the fourth preset probability; The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.
9. The control method of the monitoring system according to claim 4, wherein: The posture information includes at least the pitch angle, yaw angle and rotation angle of the face; The updating of the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain the second reference frame includes: When the ratio of the first reference frame to the initial sample image is less than a first preset threshold and greater than a second preset threshold, the specific coordinate points in the first reference frame are moved within a third set range according to the position information of the first reference frame to obtain a second reference frame; and the first preset threshold is greater than the second preset threshold.
10. The control method of the monitoring system according to claim 9, wherein: The step of marking the sample labels for the portion of the sample images of the initial sample image located in the second reference frame according to the first reference frame and the second reference frame, and using the portion of the sample images marked with the sample labels as the training images, comprises: determining an overlapping area between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame; When a fifth ratio of the overlapping area to the area of the second reference frame is less than a third preset threshold, a fifth preset range is generated according to a fifth preset probability, a sixth preset range is generated according to a sixth preset probability, and a seventh preset range is generated according to a seventh preset probability; the fifth preset probability is greater than the sixth preset probability, and the sixth preset probability is greater than the seventh preset probability; and the sum of the fifth preset probability, the sixth preset probability and the seventh preset probability is 1; When the fifth ratio is within the fifth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a third category label, so that the fifth ratio is within the fifth preset range, and a sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the fifth preset probability; When the fifth ratio is within the sixth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within the sixth preset range, and a sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the sixth preset probability; When the fifth ratio is within the seventh preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within the seventh preset range, and the sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the seventh preset probability; The data within the seventh preset range is smaller than the data within the sixth preset range, and the data within the sixth preset range is smaller than the data within the fifth preset range.
11. The control method of the monitoring system according to claim 4, wherein: The sample labels also include a fourth category label representing non-gaze; In the step of obtaining the training data set, when face recognition is performed on the initial sample image to determine that no face exists, the step further includes: The initial sample image is labeled with a fourth category label, and the initial sample image labeled with the fourth category label is used as the training image.
12. The control method of the monitoring system according to claim 4, wherein: The sample labels also include a fourth category label representing non-gaze; After performing face recognition on the initial sample image to determine that a face exists, and determining a first reference frame containing the face, the method further includes: According to the size of the initial sample image and in accordance with a fourth set range, a target center point is determined in the initial sample image, and a third reference frame is determined with the target center point as the center; If the third reference frame has no overlap with any first reference frame in the initial sample image, the portion of the sample image of the initial sample image located in the third reference frame is marked as a fourth category label, and the portion of the sample image marked with the fourth category label is used as the training image.
13. A control method for a monitoring system, wherein: The monitoring system includes a detection device and a terminal; the control method of the monitoring system includes: The detection device acquires a detection image, and uses a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and sends the recognition result to the terminal; The terminal determines a display state based on the recognition result.
14. The control method of the monitoring system according to claim 13, wherein: The terminal includes a display module and a main control module; The terminal determines a display state based on the recognition result, including: The main control module sends a wake-up request to the display module when determining that the recognition result indicates that the object is gazing; When it is determined that the recognition result indicates that there is no object gaze, and the time from the current system time to the last time the display module was awakened is greater than a preset time, sending a sleep request to the display module; The display module responds to the wake-up request to determine that the display state is normal display; and responds to the sleep request to determine that the display state is sleep.
15. A detection device, configured to acquire a detection image, and use a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and send the recognition result to a terminal so that the terminal determines a display state based at least on the recognition result.
16. A control device for a monitoring system, comprising a detection device and a terminal; The detection device is configured to acquire a detection image, and perform gaze recognition on the detection image using a pre-trained target neural network model to obtain a recognition result of the detection image; Sending the recognition result to the terminal; The terminal is configured to determine a display state based on the recognition result.
17. A computer device, wherein: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the control method of the monitoring system as described in any one of claims 1 to 12 are performed, and / or the steps of the control method of the monitoring system as described in any one of claims 13 to 14 are performed.
18. A computer non-transitory readable storage medium, wherein: The computer non-volatile readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the control method of the monitoring system as described in any one of claims 1 to 12, and / or the steps of the control method of the monitoring system as described in any one of claims 13 to 14.
Citation Information
Patent Citations
Model training method, fixation point detection method, medium, device and computing equipment
CN113762393A
Face recognition method, device and equipment based on artificial intelligence and storage medium
CN114743243A
Image recognition method and device
CN115424318A
Eye movement tracking method and device, electronic equipment, storage medium and program product
CN116091541A
Federated teacher-student machine learning
US20220012637A1
