Control method and device of broadcast monitoring system, computer equipment and storage medium

A neural network-based gaze recognition system integrated with storage-computing terminals reduces power consumption by automatically managing display states, addressing the high power usage of smart terminals during monitoring.

CN119068303APending Publication Date: 2024-12-03BOE TECHNOLOGY GROUP CO LTD +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202310636620.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The high power consumption of smart terminals due to prolonged display of monitoring images poses a challenge, affecting system life and efficiency.

Method used

A control method utilizing a pre-trained neural network model for gaze recognition to determine display states, integrating storage and computing, reducing power consumption by automatically switching to sleep mode when no one is looking at the screen and waking up upon detection.

Benefits of technology

This method effectively reduces power consumption and improves recognition accuracy, allowing smart terminals to maintain low power standby and efficiently manage display states based on user interaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068303A_ABST
    Figure CN119068303A_ABST
Patent Text Reader

Abstract

The invention provides a control method and device of a broadcast monitoring system, computer equipment and a storage medium, and belongs to the technical field of image recognition and terminal broadcast monitoring. The control method of the broadcast monitoring system comprises the following steps: acquiring a detection image; performing staring recognition on the detection image by using a pre-trained target neural network model to obtain a recognition result of the detection image; and sending the identification result to a terminal, so that the terminal determines a display state at least based on the identification result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition and terminal monitoring and broadcasting, and in particular relates to a control method, device, computer equipment and storage medium of a monitoring and broadcasting system. Background Art

[0002] With the rapid development of information technology, modern electronic devices are also developing rapidly towards intelligence, lightness and portability. For smart terminals, the display screen displays monitoring images for a long time, which makes the system power consumption larger and affects the system life. Therefore, how to achieve low power consumption operation is a technical problem that needs to be solved in the monitoring field. Summary of the invention

[0003] The present disclosure aims to solve at least one of the technical problems existing in the prior art, and provides a control method, device, computer equipment and storage medium for a monitoring system.

[0004] In the first aspect, the technical solution adopted to solve the technical problem of the present disclosure is a control method of a monitoring system, comprising:

[0005] Acquire detection images;

[0006] Using a pre-trained target neural network model, performing gaze recognition on the detection image to obtain a recognition result of the detection image;

[0007] The recognition result is sent to a terminal so that the terminal determines a display state based at least on the recognition result.

[0008] In some embodiments, the target neural network model is trained by the following steps:

[0009] Training the teacher network learning model to be trained according to the acquired training data set to obtain a trained teacher network learning model;

[0010] According to the training data set and the trained teacher network learning model, the student network learning model to be trained is trained using a knowledge distillation training method to obtain a trained student network learning model as the target neural network model.

[0011] In some embodiments, the training data set includes a plurality of training images annotated with sample labels;

[0012] The method of training the student network learning model to be trained by using the training method of knowledge distillation according to the training data set and the trained teacher network learning model to obtain the trained student network learning model as the target neural network model includes:

[0013] Inputting the training image into the trained teacher network learning model to obtain a first output result of the trained teacher network learning model;

[0014] Inputting the training image into the student network learning model to be trained to obtain a second output result of the student network learning model to be trained;

[0015] Determine a first loss function according to the first output result and the second output result;

[0016] Determining a second loss function according to the second output result and the sample label of the training image;

[0017] Obtaining a weighted loss function according to the first loss function and the second loss function;

[0018] The parameters of the student network learning model to be trained are adjusted according to the weighted loss function until the weighted loss function converges, thereby obtaining a trained student network learning model as the target neural network model.

[0019] In some embodiments, the training data set is determined by the following steps:

[0020] Acquire an original data set; the original data set includes a plurality of initial sample images;

[0021] Performing face recognition on the initial sample image, and if a face exists, determining a first reference frame containing the face, and determining key point information of the face;

[0022] Determining facial posture information according to the key point information;

[0023] According to the posture information and the ratio of the first reference frame to the initial sample image, the first reference frame is updated to obtain a second reference frame;

[0024] According to the first reference frame and the second reference frame, marking the sample label for a portion of the sample image of the initial sample image located in the second reference frame, and using the portion of the sample image marked with the sample label as the training image;

[0025] The sample labels include a first category label representing gaze, and a second category label and a third category label representing non-gaze; wherein,

[0026] The first category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the deflection angle of the face is less than or equal to a second set value, or the rotation angle of the face is less than or equal to a third set value;

[0027] The second category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is greater than the first set value, the deflection angle of the face is greater than the second set value, or the rotation angle of the face is greater than the third set value;

[0028] The third category label indicates that a ratio of the second reference frame in the training image to the training image is less than a first preset threshold and greater than a second preset threshold.

[0029] In some embodiments, the posture information includes at least a pitch angle, a yaw angle, and a rotation angle of the face;

[0030] The updating of the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain the second reference frame includes:

[0031] When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the yaw angle of the face is less than or equal to a second set value, and the rotation angle of the face is less than or equal to a third set value, the specific coordinate points in the first reference frame are moved within a first set range according to the position information of the first reference frame to obtain a second reference frame.

[0032] In some embodiments, the step of marking the sample labels for the portion of the sample images of the initial sample image located in the second reference frame according to the first reference frame and the second reference frame, and using the portion of the sample images marked with the sample labels as the training images, comprises:

[0033] determining an overlapping area between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame;

[0034] When the first ratio of the overlapping area to the area of ​​the second reference frame is greater than or equal to a third preset threshold, a first preset range is generated according to a first preset probability, a second preset range is generated according to a second preset probability, a third preset range is generated according to a third preset probability, and a fourth preset range is generated according to a fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1;

[0035] When the first ratio is within the first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the first preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the first preset probability;

[0036] When the first ratio is within the second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the first category label, so that the first ratio is within the second preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the second preset probability;

[0037] When the first ratio is within the third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the third preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is within the third preset probability;

[0038] When the first ratio is within the fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the fourth preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the fourth preset probability;

[0039] The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.

[0040] In some embodiments, the posture information includes at least a pitch angle, a yaw angle, and a rotation angle of the face;

[0041] The updating of the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain the second reference frame includes:

[0042] When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is greater than a first set value, the deflection angle of the face is greater than a second set value, and the rotation angle of the face is greater than a third set value, the specific coordinate points in the first reference frame are moved within a second set range according to the position information of the first reference frame to obtain a second reference frame.

[0043] In some embodiments, the step of marking the sample labels for the portion of the sample images of the initial sample image located in the second reference frame according to the first reference frame and the second reference frame, and using the portion of the sample images marked with the sample labels as the training images, comprises:

[0044] determining an overlapping area between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame;

[0045] When a third ratio of the overlapping area to the area of ​​the second reference frame is greater than or equal to a third preset threshold, a first preset range is generated according to a first preset probability, a second preset range is generated according to a second preset probability, a third preset range is generated according to a third preset probability, and a fourth preset range is generated according to a fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; and the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1;

[0046] When the third ratio is within the first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a second category label, so that the third ratio is within the first preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the first preset probability;

[0047] When the third ratio is within the second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the second category label, so that the third ratio is within the second preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the second preset probability;

[0048] When the third ratio is within the third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a second category label, so that the third ratio is within the third preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the third preset probability;

[0049] When the third ratio is within the fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a second category label, so that the third ratio is within the fourth preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the fourth preset probability;

[0050] The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.

[0051] In some embodiments, the posture information includes at least a pitch angle, a yaw angle, and a rotation angle of the face;

[0052] The updating of the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain the second reference frame includes:

[0053] When the ratio of the first reference frame to the initial sample image is less than a first preset threshold and greater than a second preset threshold, the specific coordinate points in the first reference frame are moved within a third set range according to the position information of the first reference frame to obtain a second reference frame; and the first preset threshold is greater than the second preset threshold.

[0054] In some embodiments, the step of marking the sample labels for the portion of the sample images of the initial sample image located in the second reference frame according to the first reference frame and the second reference frame, and using the portion of the sample images marked with the sample labels as the training images, comprises:

[0055] determining an overlapping area between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame;

[0056] When a fifth ratio of the overlapping area to the area of ​​the second reference frame is less than a third preset threshold, a fifth preset range is generated according to a fifth preset probability, a sixth preset range is generated according to a sixth preset probability, and a seventh preset range is generated according to a seventh preset probability; the fifth preset probability is greater than the sixth preset probability, and the sixth preset probability is greater than the seventh preset probability; and the sum of the fifth preset probability, the sixth preset probability and the seventh preset probability is 1;

[0057] When the fifth ratio is within the fifth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a third category label, so that the fifth ratio is within the fifth preset range, and a sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the fifth preset probability;

[0058] When the fifth ratio is within the sixth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within the sixth preset range, and a sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the sixth preset probability;

[0059] When the fifth ratio is within the seventh preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within the seventh preset range, and the sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the seventh preset probability;

[0060] The data within the seventh preset range is smaller than the data within the sixth preset range, and the data within the sixth preset range is smaller than the data within the fifth preset range.

[0061] In some embodiments, the sample labels further include a fourth category label representing non-gaze;

[0062] In the step of obtaining the training data set, when face recognition is performed on the initial sample image to determine that no face exists, the step further includes:

[0063] The initial sample image is labeled with a fourth category label, and the initial sample image labeled with the fourth category label is used as the training image.

[0064] In some embodiments, the sample labels further include a fourth category label representing non-gaze;

[0065] After performing face recognition on the initial sample image to determine that a face exists, and determining a first reference frame containing the face, the method further includes:

[0066] According to the size of the initial sample image and in accordance with a fourth set range, a target center point is determined in the initial sample image, and a third reference frame is determined with the target center point as the center;

[0067] If the third reference frame has no overlap with any first reference frame in the initial sample image, the portion of the sample image of the initial sample image located in the third reference frame is marked as a fourth category label, and the portion of the sample image marked with the fourth category label is used as the training image.

[0068] In a second aspect, an embodiment of the present disclosure further provides a control method of a monitoring system, wherein the monitoring system includes a detection device and a terminal; the control method of the monitoring system includes:

[0069] The detection device acquires a detection image, and uses a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and sends the recognition result to the terminal;

[0070] The terminal determines a display state based on the recognition result.

[0071] In some embodiments, the terminal includes a display module and a main control module;

[0072] The terminal determines a display state based on the recognition result, including:

[0073] The main control module sends a wake-up request to the display module when it is determined that the recognition result indicates that the object is staring; and sends a sleep request to the display module when it is determined that the recognition result indicates that the object is not staring and the time from the current system time to the last time the display module was awakened is greater than a preset time.

[0074] The display module responds to the wake-up request to determine that the display state is normal display; and responds to the sleep request to determine that the display state is sleep.

[0075] In the third aspect, the embodiments of the present disclosure also provide a detection device, which is configured to acquire a detection image, and use a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and send the recognition result to a terminal so that the terminal determines a display state at least based on the recognition result.

[0076] In a fourth aspect, the embodiments of the present disclosure further provide a control device of a monitoring system, which includes a detection device and a terminal;

[0077] The detection device is configured to acquire a detection image, and perform gaze recognition on the detection image using a pre-trained target neural network model to obtain a recognition result of the detection image; and send the recognition result to a terminal;

[0078] The terminal is configured to determine a display state based on the recognition result.

[0079] In a fifth aspect, an embodiment of the present disclosure further provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory through the bus, and when the machine-readable instructions are executed by the processor, the steps of the control method of the monitoring system as described in any one of the first aspect above and / or the steps of the control method of the monitoring system as described in any one of the second aspect above are performed.

[0080] In the sixth aspect, the embodiments of the present disclosure further provide a computer non-volatile readable storage medium, wherein a computer program is stored on the computer non-volatile readable storage medium, and when the computer program is executed by a processor, the steps of the control method of the monitoring system as described in any one of the first aspect above, and / or the steps of the control method of the monitoring system as described in any one of the second aspect above are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 A flowchart of a control method of a monitoring system provided by an embodiment of the present disclosure;

[0082] Figure 2 A schematic diagram of a network architecture of an exemplary target neural network model provided in an embodiment of the present disclosure;

[0083] Figure 3 A flowchart of the target neural network model training provided by the embodiment of the present disclosure;

[0084] Figure 4 A schematic diagram of an exemplary network architecture of knowledge distillation provided in an embodiment of the present disclosure;

[0085] Figure 5 A flowchart for generating a training data set provided in an embodiment of the present disclosure;

[0086] Figure 6 A schematic diagram of an exemplary face summary target key points provided in an embodiment of the present disclosure;

[0087] Figure 7A schematic diagram of different facial postures provided by an exemplary embodiment of the present disclosure;

[0088] Figure 8a An exemplary terminal workflow diagram provided for an embodiment of the present disclosure;

[0089] Figure 8b An exemplary detection device workflow diagram provided in an embodiment of the present disclosure;

[0090] Fig. 9 A schematic diagram of a detection device provided in an embodiment of the present disclosure;

[0091] Fig.10 A schematic diagram of a control device of a monitoring system provided by an embodiment of the present disclosure;

[0092] Fig.11 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0094] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantitative restrictions, but indicate that there is at least one. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0095] The "multiple or several" mentioned in this disclosure refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0096] In related technologies, with the in-depth research and popularization of artificial intelligence algorithms represented by deep learning neural networks, smart electronic devices and related application scenarios can be seen everywhere, such as face recognition, voice recognition, smart home, security monitoring, unmanned driving, etc. However, limited by the classic von Neumann computing architecture, data storage and processing are separated, and the computing and storage functions are completed by the central processing unit and memory respectively. The performance gap between the two forms a "memory wall", which consumes a lot of energy in the frequent transmission of computing cores and memory. However, the current storage-computing gradient design can solve this problem. Storage-computing integration refers to combining memory and computing more closely than traditional computer architecture, thereby reducing the overhead caused by memory access and solving the "memory wall" problem. The three major elements of artificial intelligence are computing power, data and algorithms, and storage-computing integration has the characteristics of high computing power, low power consumption and low latency, so storage-computing integration will play a huge role in the future of artificial intelligence. At present, the common storage and computing methods are flash memory (Flash) and static random access memory (Static Random-Access Memory, SRAM) in form. Generally, many national or industry standards associated with smart terminal products have certain standby power consumption requirements. To design an intelligent wake-up function on such products and support 24-hour standby wake-up through voice, personnel detection and other methods, it is necessary to ensure that the entire detection system has low standby power consumption to meet the corresponding energy efficiency standards.

[0097] In view of this, the embodiment of the present disclosure provides a control method for a monitoring system, which substantially eliminates one or more of the problems caused by the limitations and defects of the relevant technology. Specifically, the embodiment of the present disclosure obtains a detection image; uses a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; sends the recognition result to the terminal, so that the terminal determines the display state at least based on the recognition result. That is, the display state of the terminal is controlled using a high-precision gaze recognition result. For example, if no one is looking at the screen for a long time, the terminal automatically enters a sleep state, thereby saving system power consumption; if after entering the sleep state, the recognition result indicates that a person is looking at the screen, the system can be automatically started to restore the normal display of the screen. The present disclosure is applied to intelligent wake-up scenarios in a gaze environment (such as an environment where a person is staring at a display screen). While saving system power consumption, it combines neural network gaze recognition technology to improve the recognition accuracy of people looking at the screen, thereby more accurately controlling the timing of the terminal entering sleep or actively waking up.

[0098] For ease of understanding, a control method for a monitoring system provided by an embodiment of the present disclosure is first described in detail. The control method for the monitoring system may be performed by a detection device with certain computing capabilities, such as a storage-computing integrated chip with integrated Flash or SRAM, which has a faster response speed and reduces the power consumption of the monitoring system. The detection device may be part of the monitoring system.

[0099] Figure 1 A flowchart of a control method of a monitoring system provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, it includes steps S11 to S13, wherein:

[0100] S11, obtaining a detection image.

[0101] In this step, the detection image is an image captured by a shooting device and is used to detect whether someone is staring at the screen. The detection image can be an image captured by the shooting device in real time.

[0102] The shooting device may be, for example, a camera or other device with a picture acquisition function, such as a mobile phone, a vehicle-mounted terminal, a smart home device, a security monitoring device, etc.

[0103] S12. Using the pre-trained target neural network model, perform gaze recognition on the detection image to obtain a recognition result of the detection image.

[0104] Specifically, the detection image can be input into a pre-trained target neural network model, and features of the detection image can be extracted and analyzed to determine whether someone is staring at the screen. The target neural network model can use a lightweight classification model, and the result of its recognition may be that someone is staring at the screen, or it may be that no one is staring at the screen. Here, the screen can be, for example, a terminal display screen or an image acquisition device such as a camera.

[0105] In this step, a lightweight target neural network model is designed and migrated to the integrated storage and computing chip. A low-power integrated storage and computing method is used to assist the target neural network in reasoning, complete the wake-up service of the smart terminal, reduce power consumption, and increase response speed.

[0106] S13. Send the recognition result to the terminal, so that the terminal determines the display state at least based on the recognition result.

[0107] Specifically, the recognition result is sent to the terminal, and the main control module in the terminal can determine the display state of the terminal according to the indication of the recognition result; the display screen in the terminal can respond to the currently determined display state to determine whether to display the picture normally or enter sleep.

[0108] The control method of the monitoring system provided by the present disclosure realizes monitoring by adopting a low-power storage-computing integrated method. The designed lightweight target neural network model is migrated to the storage-computing integrated chip, and the lightweight characteristics are used to reduce the power consumption of the storage-computing integrated chip. At the same time, combined with the neural network gaze recognition technology, the recognition accuracy of people staring at the screen is improved, so as to more accurately control the timing of the terminal entering sleep or actively waking up. When the display screen enters the sleep state, the system power consumption is saved. When the terminal is not actively awakened, it can remain in standby for a long time, thereby solving the problem of high power consumption and reducing a lot of maintenance costs.

[0109] Since the intelligent wake-up application scenario of the present disclosure is limited by the hardware of the storage-computing integrated chip, it is necessary to design a lightweight target neural network model to complete the target detection task. The following is a detailed introduction to the target neural network model training process provided by the embodiment of the present disclosure.

[0110] In some embodiments, Figure 2 A schematic diagram of a network architecture of an exemplary target neural network model provided in an embodiment of the present disclosure is shown in FIG. Figure 2 As shown in the figure, it includes seven convolutional layers conv. For example, the first convolutional layer conv1 uses a convolutional kernel of size 3×3 for convolution, a convolution step length stride of 2, and a padding value of 1; the second convolutional layer conv2 uses a convolutional kernel of size 3×3 for convolution, a convolution step length stride of 2, and a padding value of 1; the third convolutional layer conv3 uses a convolutional kernel of size 3×3 for convolution, a convolution step length stride of 1, and a padding value of 1; the fourth convolutional layer conv4 uses a convolutional kernel of size 3×3 for convolution, a convolution step length stride of 2, and a padding value of 1; the fifth convolutional layer conv5 uses a convolutional kernel of size 7×7 for convolution, a convolution step length stride of 2, and a padding value of 1; the sixth convolutional layer conv6 uses a convolutional kernel of size 7×7 for convolution, a convolution step length stride of 2, and a padding value of 1. The convolution kernel of size 3×3 is used for convolution, the stride of convolution step is 1, and the padding value is 1; the seventh convolution layer conv7 uses the convolution kernel of size 3×3 for convolution, the stride of convolution step is 2, and the padding value is 1.

[0111] The above-mentioned target neural network model meets the lightweight requirements and is migrated to the integrated storage and computing chip. During its application process, it has a faster response speed and lower power consumption.

[0112] Since the model is relatively small, directly training the designed model may not achieve high-precision recognition results. Therefore, the large model distillation method is used to improve performance. MobilNetV3 with a width of 0.5 is used as the large model, that is, the teacher network learning model (teacher model). Finally, the larger teacher model is converted into a smaller student network learning model (student model) as the target neural network model after training.

[0113] In some embodiments, Figure 3 A flowchart of the target neural network model training provided in the embodiment of the present disclosure, such as Figure 3 As shown, the target neural network model is trained by the following steps S21 to S22:

[0114] S21. Train the teacher network learning model to be trained according to the acquired training data set to obtain a trained teacher network learning model.

[0115] Here, the training data set can be a labeled public data set and / or collected video data. However, unlike face recognition, the gaze recognition scenario is based on face recognition, and further uses the information of key points in the face to determine whether the human eye is in a state of staring at the screen. Therefore, compared with the face recognition scenario, the number of positive samples (i.e., gaze samples) in the existing public data set is often small. In order to improve the accuracy of model training, a large amount of sample data support is required. Therefore, the training data set disclosed in the present invention can be based on the labeled public data set and / or collected video data, and the data set after further enriching the sample types can make the obtained training images more diverse. The method of enriching the sample types can be referred to the following steps S211 to S217, and the detailed process is not described here. Among them, the training data set includes multiple training images, and each training image includes a sample label corresponding to it.

[0116] Exemplarily, a training image in a training data set is input into a teacher network learning model to be trained, and an output result corresponding to the training image is obtained, where the output result is a result indicating whether someone is staring at the screen. If the sample label of the training image indicates that someone is staring at the screen, a third loss function is determined based on the output result and the sample label of the training image; the above process is repeated, and the training images in the training data set are traversed in sequence to determine the third loss function, so as to continuously train the teacher network learning model to be trained, and finally a trained teacher network learning model is obtained.

[0117] S22. According to the training data set and the trained teacher network learning model, the student network learning model to be trained is trained using the knowledge distillation training method to obtain the trained student network learning model as the target neural network model.

[0118] Figure 4 A schematic diagram of an exemplary knowledge distillation network architecture provided in an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, it includes a teacher model and a student model. The teacher model includes m network layers; the student model includes n network layers; here, the teacher model and the student model can be homogeneous networks or heterogeneous networks.

[0119] The process of training the student model also uses the hard labels in the training data set (i.e., the sample labels carried by the training images) to train the teacher model, and then uses the soft labels obtained in the teacher model (i.e., the output of the softmax layer in the teacher model) combined with the hard labels in the training data set to jointly determine the loss function of the student model.

[0120] Continue as Figure 4 As shown in the figure, the output result of the mth network layer of the teacher model is processed by the softmax layer to obtain the first output result; the output result of the nth network layer of the student model is processed by the softmax layer to obtain the second output result. The loss function loss is constructed based on the first output result, the second output result and the sample label.

[0121] This embodiment adopts the knowledge distillation method to transform a larger teacher model into a smaller student model while retaining performance close to that of the teacher model, thereby solving the problem of insufficient hardware for deploying the target neural network model at the edge segment.

[0122] The specific training process is shown in steps S221 to S226, where:

[0123] S221. Input the training image into the trained teacher network learning model to obtain the first output result of the trained teacher network learning model.

[0124] Exemplarily, the training image A in the training data set is input into the trained teacher network learning model to obtain a first output result corresponding to the training image A. The first output result is a result output by the trained teacher network learning model indicating whether someone is staring at the screen in the training image A, such as the predicted label of the training image A predicted by the trained teacher network learning model.

[0125] S222. Input the training image into the student network learning model to be trained to obtain a second output result of the student network learning model to be trained.

[0126] Continuing with the above example, the training image A in the training data set is input into the student network learning model to be trained to obtain the second output result corresponding to the training image A. The second output result is the result output by the student network learning model to be trained indicating whether someone is staring at the screen in the training image A, such as the predicted label of the training image A predicted by the student network learning model to be trained.

[0127] It should be noted that the same training image corresponds to a set of first output results and second output results.

[0128] S223. Determine a first loss function according to the first output result and the second output result.

[0129] According to a group of first output results and second output results corresponding to the same training image, the mean square error between the first output result and the second output result is calculated as the first loss function; or according to a group of first output results and second output results corresponding to the same training image, the first loss function is determined using the KLD loss function algorithm (also known as KLDloss).

[0130] S224. Determine a second loss function according to the second output result and the sample label of the training image.

[0131] The second loss function can be determined according to the overlap IOU between the second output result (ie, the predicted label of the training image A predicted by the student network learning model to be trained) and the sample label of the training image (ie, the real hard label).

[0132] S225. Obtain a weighted loss function according to the first loss function and the second loss function.

[0133] According to the preset weights, the first loss function and the second loss function are weighted to obtain a weighted loss function. Here, the preset weights can be determined based on experience, and the embodiments of the present disclosure do not impose specific limitations.

[0134] Obtain the proportion coefficients of the first loss function and the second loss function. In this embodiment, since the weighted loss function is composed of the first loss function and the second loss function, by determining one of the proportion coefficients (α), the other proportion coefficient (ie, 1-α) can be determined. Afterwards, the first loss function L1 and the second loss function L2 are weighted and summed according to the proportion coefficients to determine the weighted loss function. The weighted loss function loss combined with the above description can be specifically expressed as: L = α × L1 + (1-α) × L2.

[0135] S226. Adjust the parameters of the student network learning model to be trained according to the weighted loss function until the weighted loss function converges, and obtain the trained student network learning model as the target neural network model.

[0136] In addition, while lightweight models reduce system power consumption, they are often accompanied by the problem of insufficient detection accuracy. In order to improve the recognition accuracy of the lightweight target neural network model, the present invention improves the model training process, specifically increasing the diversity of training images at the training sample level, using training images corresponding to multiple category labels to train the model, and improving the training accuracy of the target neural network model.

[0137] In some embodiments, the training data set may be a data set that is further enriched with sample types based on the original data set. Figure 5 A flowchart of generating a training data set is provided in an embodiment of the present disclosure, such as Figure 5 As shown, the specific steps of generating diversified training images include S211 to S217, wherein:

[0138] S211. Obtain original data set.

[0139] The original data set includes multiple initial sample images.

[0140] S212, performing face recognition on the initial sample image, if a face exists, executing step S213; if no face exists, executing step S217.

[0141] For example, the face detection network retinaface or the neural network yolov5 and other models can be used to perform face recognition on the initial sample image.

[0142] S213: Determine a first reference frame that includes the face, and determine key point information of the face.

[0143] When it is determined that there is a face in the initial sample image, a first reference frame is marked for the area where the face is located, so that the first reference frame frames the face. The position information of the first reference frame indicates the position of the face in the initial sample image.

[0144] When it is determined that a human face exists in the initial sample image, target key points of the human face are further determined. The key point information indicates the coordinates of the target key points in the initial sample image. Figure 6 A schematic diagram of an exemplary face summary target key points provided in an embodiment of the present disclosure, such as Figure 6 As shown, the target key points here can be customized according to the actual scene. For example, the target key points may include left eye 1, right eye 2, nose 3, left corner of mouth 4, right corner of mouth 5, center of left and right corners of mouth 6 and other key points.

[0145] S214: Determine the posture information of the face according to the key point information.

[0146] Figure 7A schematic diagram of an exemplary embodiment of the present disclosure providing different facial postures, such as Figure 7 As shown, specifically, a posture estimation algorithm (such as the posture estimation algorithm in POSIT in OpenCV) can be used to project the target key points (3D key points) of the six world coordinate systems to the five 2D key points through changes such as rotation and translation, and then estimate the transformation parameters, and finally obtain the posture information of the face in the 2D plane, such as the pitch angle, yaw angle and roll angle of the face.

[0147] The subsequent combination of multi-angle posture information can improve the accuracy of sample labels of training images, thereby improving the accuracy of model training.

[0148] S215 . Update the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain a second reference frame.

[0149] Different posture information and / or different ratios of the first reference frame to the initial sample image correspond to different ways of updating the first reference frame, thereby obtaining different second reference frames. The specific updating method may be, for example, randomly moving specific coordinate points.

[0150] In one embodiment, the posture information includes the pitch angle, yaw angle, and rotation angle of the face.

[0151] When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the yaw angle of the face is less than or equal to a second set value, and the rotation angle of the face is less than or equal to a third set value, the specific coordinate points in the first reference frame are moved within a first set range according to the position information of the first reference frame to obtain a second reference frame.

[0152] Among them, the first preset threshold, the first set value, the second set value and the third set value can all be set according to the actual application scenario. Exemplarily, the first preset threshold is set to 50%, the first set value is set to 25°, the second set value is set to 25°, and the third set value is set to 45°. That is to say, under the conditions that the ratio of the first reference frame to the initial sample image is greater than or equal to 50%, the deflection angle of the face is less than or equal to 25°, the pitch angle of the face is less than or equal to 25°, and the rotation angle of the face is less than or equal to 45°, it is determined that the person is in the state of staring at the screen at this time.

[0153] Here, when the ratio of the first reference frame to the initial sample image is greater than or equal to the first preset threshold, the pitch angle of the face is less than or equal to the first set value, the deflection angle of the face is less than or equal to the second set value, and the rotation angle of the face is less than or equal to the third set value, it can be determined that the sample label to be marked is a first category label, and the first category label is a label used to characterize gaze. It should be noted that this is only a process of determining the sample label to be marked, not a marking process. According to the current judgment that the sample label to be marked is a first category label, the corresponding first set range is determined. The mapping relationship between the first category label and the first set range is preset.

[0154] Afterwards, according to the position information of the first reference frame, the specific coordinate point in the first reference frame is moved according to the first set range to obtain the second reference frame. Here, the first set range corresponding to the first category label may be a pre-set moving range of the specific coordinate point. The specific coordinate point may be a vertex of the outer contour of the first reference frame, such as vertex 1 at the upper left corner, vertex 2 at the upper right corner, vertex 3 at the lower left corner, and vertex 4 at the lower right corner. In this embodiment, vertex 1 at the upper left corner and vertex 4 at the lower right corner may be selected as the specific coordinate points for the diversified update of the reference frame.

[0155] For example, the coordinates of the vertex 1 in the upper left corner are (lt x ,lt y ), the coordinates of vertex 4 in the lower right corner are (rb x , rb y ); Let the coordinates of vertex 1 (lt x ,lt y ) has a random move value of δ x , δ y ; The coordinates of vertex 4 (rb x , rb y )’s random moving value σ x , σ y , the value can be positive or negative. After random movement, the coordinates of the new vertex 1 are (lt1 x , lt1 y ), that is:

[0156] lt1 x =lt x +δ x

[0157] lt1 y =lt y +δ y

[0158] The coordinates of the new vertex 4 are (rb1 x ,rb1 y ), that is:

[0159] rb1 x =rb x +σ x

[0160] rb1 y =rb y +σ y

[0161] Among them, the first setting range includes δ x , δ y , σ x and σ y The range of, for example, δ x ∈[-w1,0],δ y ∈[-h1,0],σ x ∈[0,rw1],σ y ∈[0,h1]. Where w1=(rb x -ltx)×0.8; h1=(rb y -lt y )×0.5. When the specific coordinate point is moved according to the first setting range, the corresponding first reference frame expands outward, and the expansion range is small, so as to ensure a better gaze effect of the face in the second reference frame.

[0162] In one embodiment, the posture information includes the pitch angle, yaw angle, and rotation angle of the face.

[0163] When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is greater than a first set value, the yaw angle of the face is greater than a second set value, and the rotation angle of the face is greater than a third set value, the specific coordinate points in the first reference frame are moved within a second set range according to the position information of the first reference frame to obtain a second reference frame.

[0164] Among them, the first preset threshold, the first set value, the second set value and the third set value can all be set according to the actual application scenario. Exemplarily, the first preset threshold is set to 50%, the first set value is set to 25°, the second set value is set to 25°, and the third set value is set to 45°. That is to say, under the conditions that the ratio of the first reference frame to the initial sample image is greater than or equal to 50%, the deflection angle of the face is greater than 25°, the pitch angle of the face is greater than 25°, and the rotation angle of the face is greater than 45°, it is determined that the person is not in the state of staring at the screen at this time.

[0165] Here, when the ratio of the first reference frame to the initial sample image is greater than or equal to the first preset threshold, the pitch angle of the face is greater than the first set value, the deflection angle of the face is greater than the second set value, and the rotation angle of the face is greater than the third set value, it can be determined that the sample label to be marked is a second category label, and the second category label is a label used to characterize non-gaze. It should be noted that this is only a process of determining the sample label to be marked, not a marking process. According to the current judgment that the sample label to be marked is a second category label, determine the corresponding second set range. The mapping relationship between the second category label and the second set range is preset.

[0166] Afterwards, according to the position information of the first reference frame, the specific coordinate point in the first reference frame is moved according to the second set range to obtain the second reference frame. Here, the second set range corresponding to the second category label may be a pre-set moving range of the specific coordinate point. The specific coordinate point may be a vertex of the outer contour of the first reference frame, such as vertex 1 at the upper left corner, vertex 2 at the upper right corner, vertex 3 at the lower left corner, and vertex 4 at the lower right corner. In this embodiment, vertex 1 at the upper left corner and vertex 4 at the lower right corner may be selected as the specific coordinate points for the diversified update of the reference frame.

[0167] For example, the coordinates of the vertex 1 in the upper left corner are (lt x ,lt y ), the coordinates of vertex 4 in the lower right corner are (rb x , rb y ); Let the coordinates of vertex 1 (lt x ,lt y ) has a random move value of δ x , δ y ; The coordinates of vertex 4 (rb x , rb y )’s random moving value σ x , σ y , the value can be positive or negative. After random movement, the coordinates of the new vertex 1 are (lt1 x , lt1 y ), that is:

[0168] lt1 x =lt x +δ x

[0169] lt1 y =lt y +δ y

[0170] The coordinates of the new vertex 4 are (rb1 x ,rb1 y ), that is:

[0171] rb1 x =rb x +σ x

[0172] rb1 y =rb y +σ y

[0173] The second setting range may be the same as or different from the first setting range. The disclosure takes the second setting range being the same as the first setting range as an example for explanation, and the second setting range includes δ x , δ y , σ x and σ y The range of, for example, δ x ∈[-w1,0],δ y ∈[-h1,0],σ x ∈[0,w1],σ y ∈[0,h1]. Where w1=(rb x -lt x )×0.8; h1=(rb y -lt y )×0.5. When the specific coordinate point is moved according to the second set range, the corresponding first reference frame expands outward, and the expansion range is small, so as to ensure the diversity of samples of non-gaze effects of faces in the second reference frame, thereby improving the accuracy of subsequent determination of positive and negative samples.

[0174] In one embodiment, the posture information includes the pitch angle, yaw angle, and rotation angle of the face.

[0175] When the ratio of the first reference frame to the initial sample image is less than the first preset threshold and greater than the second preset threshold, the specific coordinate points in the first reference frame are moved according to the position information of the first reference frame within a third set range to obtain the second reference frame; the first preset threshold is greater than the second preset threshold.

[0176] The first preset threshold and the second preset threshold can be set according to the actual application scenario. Exemplarily, the first preset threshold is set to 50%, and the second preset threshold can be 0. That is to say, under the condition that the ratio of the first reference frame to the initial sample image is less than 50% and greater than 0, it is determined that the person is in a non-staring screen state at this time.

[0177] Here, when the ratio of the first reference frame to the initial sample image is less than the first preset threshold and greater than the second preset threshold, it can be determined that the sample label to be marked is a third category label, and the third category label is a label used to characterize non-gaze. It should be noted that this is only a process of determining the sample label to be marked, not a marking process. According to the current determination that the sample label to be marked is a third category label, the corresponding third set range is determined. The mapping relationship between the third category label and the third set range is preset.

[0178] Afterwards, according to the position information of the first reference frame, the specific coordinate point in the first reference frame is moved according to the third set range to obtain the second reference frame. Here, the third set range corresponding to the third category label may be a pre-set moving range of the specific coordinate point. The specific coordinate point may be a vertex of the outer contour of the first reference frame, such as vertex 1 at the upper left corner, vertex 2 at the upper right corner, vertex 3 at the lower left corner, and vertex 4 at the lower right corner. In this embodiment, vertex 1 at the upper left corner and vertex 4 at the lower right corner may be selected as the specific coordinate points for the diversified update of the reference frame.

[0179] For example, the coordinates of the vertex 1 in the upper left corner are (lt x ,lt y ), the coordinates of vertex 4 in the lower right corner are (rb x , rb y ); Let the coordinates of vertex 1 (lt x ,lt y ) has a random move value of δ x , δ y ; The coordinates of vertex 4 (rb x , rb y )’s random moving value σ x , σ y , the value can be positive or negative. After random movement, the coordinates of the new vertex 1 are (lt1 x , lt1 y ), that is:

[0180] lt1 x =lt x +δ x

[0181] lt1 y =lt y +δ y

[0182] The coordinates of the new vertex 4 are (rb1 x ,rb1 y ), that is:

[0183] rb1 x =rb x+σ x

[0184] rb1 y =rb y +σ y

[0185] Among them, the third setting range includes δ x , δ y , σ x and σ y The range of, for example, δ x ∈[-w2,-w3],δ y ∈[-h2,-h3],σ x ∈[w3,w2],σ y ∈[h3,h2]. Where w2=(rb x -lt x )×2.5; w3=(rb x -lt x )×0.8; h2=(rb y -lt y )×2.5, h3=(rb y -lt y )×0.8. When the specific coordinate point is moved according to the third setting range, the corresponding first reference frame expands outward, and its expansion range is larger than that of the first setting range, so as to ensure that the non-gaze effect of the face in the second reference frame is better, and on the basis of the non-gaze effect corresponding to the second setting range, the sample diversity of the non-gaze effect of the face is further improved, thereby improving the accuracy of the subsequent determination of positive and negative samples.

[0186] The present disclosure is combined with the above-mentioned embodiments, and refines the gaze and non-gaze states (for example, the state corresponding to the first category label belonging to the gaze state, the state corresponding to the second category label belonging to the non-gaze state, and the state corresponding to the third category label) according to the ratio of the first reference frame to the initial sample image and the posture information of the face, thereby improving the accuracy of subsequent determination of positive and negative samples.

[0187] S216 , according to the first reference frame and the second reference frame, marking sample labels for a portion of the sample images of the initial sample image located in the second reference frame, and using the portion of the sample images marked with sample labels as training images.

[0188] With respect to the specific implementation process of the above-mentioned step S215, it is determined that the sample labels include a first category label representing gaze, and a second category label and a third category label representing non-gaze; wherein the first category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is less than or equal to the first set value, the deflection angle of the face is less than or equal to the second set value, or the rotation angle of the face is less than or equal to the third set value; the second category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is greater than the first set value, the deflection angle of the face is greater than the second set value, or the rotation angle of the face is greater than the third set value; the third category label indicates that the ratio of the second reference frame in the training image to the training image is less than the first preset threshold and greater than the second preset threshold.

[0189] In one embodiment, when the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the deflection angle of the face is less than or equal to a second set value, and the rotation angle of the face is less than or equal to a third set value, the specific process of marking the first category label includes:

[0190] The overlapping area of ​​the second reference frame and the first reference frame may be determined based on the position information of the second reference frame and the position information of the first reference frame.

[0191] It should be noted that, since the second reference frame is obtained by expanding the first reference frame, the larger the overlapping area between the second reference frame and the first reference frame, the greater the proportion of the face in the second reference frame in the entire second reference frame.

[0192] In order to balance the number of positive samples, when the first ratio of the overlapping area to the area of ​​the second reference frame is greater than or equal to the third preset threshold, the first preset range is generated according to the first preset probability, the second preset range is generated according to the second preset probability, the third preset range is generated according to the third preset probability, and the fourth preset range is generated according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1.

[0193] Afterwards, according to the preset range controlled by the above-mentioned preset probability, when the first ratio is within the first preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the first category label, so that the first ratio is within the first preset range, and the second ratio between the number of training images marked with the first category label and the total number of all training images marked with the first category label in the training data set is the first preset probability; when the first ratio is within the second preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the first category label, so that the first ratio is within the second preset range, and the second ratio between the number of training images marked with the first category label and the total number of all training images marked with the first category label in the training data set is The ratio is a second preset probability; when the first ratio is within a third preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the first category label, so that the first ratio is within the third preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the third preset probability; when the first ratio is within a fourth preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the first category label, so that the first ratio is within the fourth preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the fourth preset probability.

[0194] The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.

[0195] Exemplarily, the third preset threshold is set to 0.5; if the first ratio is greater than or equal to 0.5, the first preset range of 0.5 to 0.7 (excluding 0.7) is selected with a probability of 50% (first preset probability), 0.7 to 0.8 (excluding 0.8) is selected with a probability of 30% (second preset probability), 0.8 to 0.9 (excluding 0.9) is selected with a probability of 10% (third preset probability), and 0.9 to 1.0 is selected with a probability of 10% (fourth preset probability). When the first ratio is within the first preset range, part of the sample images located in the second reference frame are marked as the first category label, so that the second ratio between the number of training images with the first ratio within the first preset range and marked with the first category label and the total number of training images marked with the first category label in the training data set is 50%. When the first ratio is within the second preset range, the part of the sample images located in the second reference frame is marked as the first category label, so that the second ratio between the number of training images with the first ratio within the second preset range and the first category label and the total number of training images with the first category label in the training data set is 30%. When the first ratio is within the third preset range, the part of the sample images located in the second reference frame is marked as the first category label, so that the second ratio between the number of training images with the first ratio within the third preset range and the first category label and the total number of training images with the first category label in the training data set is 10%. When the first ratio is within the fourth preset range, the part of the sample images located in the second reference frame is marked as the first category label, so that the second ratio between the number of training images with the first ratio within the fourth preset range and the first category label and the total number of training images with the first category label in the training data set is 10%.

[0196] This embodiment controls the proportion of positive samples corresponding to different first ratios to avoid having facial features on every positive sample training image, thereby improving the diversity of positive samples.

[0197] In one embodiment, when the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is greater than a first set value, the deflection angle of the face is greater than a second set value, and the rotation angle of the face is greater than a third set value, the specific process of marking the second category label includes:

[0198] The overlapping area of ​​the second reference frame and the first reference frame may be determined based on the position information of the second reference frame and the position information of the first reference frame.

[0199] It should be noted that, since the second reference frame is obtained by expanding the first reference frame, the larger the overlapping area between the second reference frame and the first reference frame, the greater the proportion of the face in the second reference frame in the entire second reference frame.

[0200] When the third ratio of the overlapping area to the area of ​​the second reference frame is greater than or equal to the third preset threshold, the first preset range is generated according to the first preset probability, the second preset range is generated according to the second preset probability, the third preset range is generated according to the third preset probability, and the fourth preset range is generated according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1.

[0201] Afterwards, according to the preset range controlled by the above-mentioned preset probability, when the third ratio is within the first preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the second category label, so that the third ratio is within the first preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the first preset probability; when the third ratio is within the second preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the second category label, so that the third ratio is within the second preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is ratio is a second preset probability; when the third ratio is within the third preset range, part of the sample images in the initial sample image located in the second reference frame is determined and marked as the second category label, so that the third ratio is within the third preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the third preset probability; when the third ratio is within the fourth preset range, part of the sample images in the initial sample image located in the second reference frame is determined and marked as the second category label, so that the third ratio is within the fourth preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the fourth preset probability.

[0202] The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.

[0203] Exemplarily, the third preset threshold is set to 0.5; if the third ratio is greater than or equal to 0.5, the first preset range of 0.5 to 0.7 (excluding 0.7) is selected with a probability of 50% (first preset probability), 0.7 to 0.8 (excluding 0.8) is selected with a probability of 30% (second preset probability), 0.8 to 0.9 (excluding 0.9) is selected with a probability of 10% (third preset probability), and 0.9 to 1.0 is selected with a probability of 10% (fourth preset probability). When the third ratio is within the first preset range, part of the sample images located in the second reference frame are marked as second category labels, so that the fourth ratio between the number of training images with the third ratio within the first preset range and marked with the second category label and the total number of training images marked with the second category label in the training data set is 50%. When the third ratio is within the second preset range, some sample images located in the second reference frame are marked as the second category label, so that the fourth ratio between the number of training images with the third ratio within the second preset range and the second category label and the total number of training images with the second category label in the training data set is 30%. When the third ratio is within the third preset range, some sample images located in the second reference frame are marked as the second category label, so that the fourth ratio between the number of training images with the third ratio within the third preset range and the second category label and the total number of training images with the second category label in the training data set is 10%. When the third ratio is within the fourth preset range, some sample images located in the second reference frame are marked as the second category label, so that the fourth ratio between the number of training images with the third ratio within the fourth preset range and the second category label and the total number of training images with the second category label in the training data set is 10%.

[0204] The above is the process of producing negative samples corresponding to the second category label. This embodiment controls the proportion of negative samples corresponding to different third ratios so that the produced training images have a large proportion of human faces and are samples in a non-gaze state, thereby improving the diversity of negative samples.

[0205] In one embodiment, when the ratio of the first reference frame to the initial sample image is less than a first preset threshold and greater than a second preset threshold, the specific process of marking the third category label includes:

[0206] The overlapping area of ​​the second reference frame and the first reference frame may be determined based on the position information of the second reference frame and the position information of the first reference frame.

[0207] When the fifth ratio of the overlapping area to the area of ​​the second reference frame is less than the third preset threshold, the fifth preset range is generated according to the fifth preset probability, the sixth preset range is generated according to the sixth preset probability, and the seventh preset range is generated according to the seventh preset probability; the fifth preset probability is greater than the sixth preset probability, and the sixth preset probability is greater than the seventh preset probability; the sum of the fifth preset probability, the sixth preset probability and the seventh preset probability is 1.

[0208] Afterwards, according to the preset range controlled by the above preset probability, the part of the sample image in the initial sample image located in the second reference frame is marked as the third category label; when the fifth ratio is within the fifth preset range, the part of the sample image in the initial sample image located in the second reference frame is determined and marked as the third category label, so that the fifth ratio is within the fifth preset range, and the sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the fifth preset probability; when the fifth ratio is within the sixth preset range, the part of the sample image in the initial sample image located in the second reference frame is determined and marked as the third category label, so that the sixth ratio is within the sixth preset range. Assume that the sixth ratio between the number of training images marked with the third category label within the range and to the total number of training images marked with the third category label in the training data set is the sixth preset probability; when the fifth ratio is within the seventh preset range, determine and mark some sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within the seventh preset range, and the sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the seventh preset probability; wherein, the data within the seventh preset range is smaller than the data within the sixth preset range, and the data within the sixth preset range is smaller than the data within the fifth preset range.

[0209] Exemplarily, the third preset threshold is set to 0.5; if the fifth ratio is less than 0.5, the fifth preset range of 0.25 to 0.5 (excluding 0.5) is selected according to a probability of 60% (fifth preset probability), 0.15 to 0.25 (excluding 0.25) is selected according to a probability of 30% (sixth preset probability), and 0.01 to 0.15 is selected according to a probability of 10% (seventh preset probability). When the fifth ratio is within the fifth preset range, part of the sample images located in the second reference frame are marked as the third category label, so that the sixth ratio between the number of training images with the fifth ratio within the fifth preset range and the third category label and the total number of training images with the third category label in the training data set is 60%. When the fifth ratio is within the sixth preset range, part of the sample images located in the second reference frame are marked as the third category label, so that the sixth ratio between the number of training images with the fifth ratio within the sixth preset range and the third category label and the total number of training images with the third category label in the training data set is 30%. When the fifth ratio is within the seventh preset range, some sample images located in the second reference frame are marked as third category labels, so that the sixth ratio between the number of training images with the fifth ratio within the seventh preset range and marked with the third category labels and the total number of training images marked with the third category labels in the training data set is 10%.

[0210] The above is the process of producing negative samples corresponding to the third category label. This embodiment controls the ratio of negative samples corresponding to the ratio of different overlapping areas to the area of ​​the second reference frame, so that the produced training images are samples with a small proportion of human faces and in a non-gaze state, thereby improving the diversity of negative samples.

[0211] S217: Mark the initial sample image with a fourth category label, and use the initial sample image marked with the fourth category label as a training image.

[0212] The sample labels also include a fourth category label representing non-gaze.

[0213] In some embodiments, after executing S212, it also includes: according to the size of the initial sample image and in accordance with the fourth set range, a target center point can be determined in the initial sample image, and a third reference frame can be determined with the target center point as the center; if the third reference frame has no overlap with any first reference frame in the initial sample image, then the portion of the sample image of the initial sample image located in the third reference frame is marked as a fourth category label, and the portion of the sample image marked with the fourth category label is used as a training image.

[0214] Here, the target center point is randomly set on the initial sample image to define the third reference frame, and part of the sample image corresponding to the third reference frame without a face is used as a negative sample training image.

[0215] For example, the width of the initial sample image is w', and the height is h'; the coordinates of the randomly generated target center point are (cnew x ,cnew y ), where cnew x ∈[0.05×w', 0.95×w'], cnew y ∈[0.05×h',0.95×h']. The width of the randomly generated third reference frame is wnew and the height is hnew, where wnew∈[w'×0.1,w'×0.9] and hnew∈[h'×0.1,h'×0.9], thereby determining the third reference frame. If the third reference frame does not overlap with any first reference frame in the initial sample image, it means that there is no human face in the third reference frame. Then, the part of the sample image in the initial sample image located in the third reference frame can be marked as the fourth category label, and the part of the sample image marked with the fourth category label can be used as the training image.

[0216] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0217] It should be noted that, in addition to the gaze detection and device wake-up mentioned in the embodiments of this disclosure, portable smart terminal application services in smart homes and wearable devices can adopt this high-efficiency storage and computing method to reduce a lot of costs. Moreover, with the development of storage and computing technology, it can support larger deep learning models and more extensive and universal services; in addition, since different storage and computing platforms support different functions and operators, this disclosure can design and build a network based on the actual hardware platform, and is not limited to the target neural network model proposed in this disclosure.

[0218] In addition, the embodiment of the present disclosure also provides another control method of a monitoring system, the execution subject of the control method of the monitoring system can be a monitoring system, and the monitoring system includes a detection device and a terminal; the detection device can be, for example, a storage and computing integrated chip integrated with Flash or SRAM. The control method of the monitoring system includes:

[0219] The detection device acquires a detection image, and uses a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and sends the recognition result to the terminal.

[0220] The terminal determines the display state based on the recognition result.

[0221] The disclosed embodiment provides a control method for a monitoring system, which uses high-precision gaze recognition results to control the display state of the terminal. For example, if no one is looking at the screen for a long time, the terminal automatically enters a sleep state, thereby saving system power consumption; if after entering the sleep state, the recognition result indicates that a person is looking at the screen, the system can be automatically started to restore the normal display of the screen. The present disclosure is applied to intelligent wake-up scenarios in a gaze environment (for example, an environment where a person is staring at a display screen). While saving system power consumption, it is combined with neural network gaze recognition technology to improve the recognition accuracy of people looking at the screen, thereby more accurately controlling the timing of the terminal entering sleep or actively waking up.

[0222] In some embodiments, the terminal includes a display module and a main control module; the terminal determines the display state based on the recognition result, including:

[0223] When determining that the recognition result indicates that the object is staring, the main control module sends a wake-up request to the display module; when determining that the recognition result indicates that the object is not staring, and the time from the current system time to the last time the display module was awakened is greater than a preset time, the main control module sends a sleep request to the display module; the display module responds to the wake-up request and determines that the display state is normal display; and responds to the sleep request and determines that the display state is sleep.

[0224] Here, the preset duration can be set based on experience. It should be noted that the preset duration is set reasonably so that the terminal can reasonably enter the sleep state in the actual application scenario. If the preset duration is too short, it will cause frequent switching between sleep and wake-up states, which is not conducive to reducing power consumption. If the preset duration is too long, the power consumption of the terminal cannot be reduced to the greatest extent.

[0225] Here, the display module may be, for example, a display screen, and the main control module may be, for example, a system on chip (SOC). The main control module and the detection device may communicate using protocols such as UART.

[0226] Figure 8a An exemplary terminal workflow diagram provided in the embodiment of the present disclosure is as follows: Figure 8aAs shown, the main control module: S31, start system initialization; S32, send a start instruction to the detection device, and receive a response confirmation from the detection device, and execute S33 in sequence; S33, reset the timer; S34, poll the serial port to see if a recognition result indicating gaze is received, and send a response confirmation to the detection device; if so, return to step S33; if not, execute S35; S35, determine whether the preset time is exceeded; if so, execute S36; if not, return to execute S34; S36, the main control SOC enters the sleep state, and sends a sleep request to the display module, and executes S37 in sequence; S37, poll the serial port to see if a recognition result indicating gaze is received, and send a response confirmation to the detection device; if so, execute S38; if not, repeat S37; S38, wake up the main control SOC, send a wake-up request to the display module, and return to step S33.

[0227] Figure 8b An exemplary detection device workflow diagram provided in the embodiment of the present disclosure is as follows: Figure 8b As shown, the detection device: S301, start system initialization; S302, poll the serial port to receive the startup command sent by the main control module, and send a response confirmation to the main control module, and execute S303 in sequence; S303, identify whether someone is staring at the screen, and send the identification result to the terminal.

[0228] In some embodiments, the target neural network model is trained by the following steps:

[0229] Training the teacher network learning model to be trained according to the acquired training data set to obtain a trained teacher network learning model;

[0230] According to the training data set and the trained teacher network learning model, the knowledge distillation training method is used to train the student network learning model to be trained, and the trained student network learning model is obtained as the target neural network model.

[0231] In some embodiments, the training data set includes a plurality of training images annotated with sample labels;

[0232] According to the training data set and the trained teacher network learning model, the student network learning model to be trained is trained using the knowledge distillation training method to obtain the trained student network learning model as the target neural network model, including:

[0233] Inputting the training image into the trained teacher network learning model to obtain a first output result of the trained teacher network learning model;

[0234] Inputting the training image into the student network learning model to be trained to obtain a second output result of the student network learning model to be trained;

[0235] Determine a first loss function according to the first output result and the second output result;

[0236] Determine a second loss function according to the second output result and the sample label of the training image;

[0237] According to the first loss function and the second loss function, a weighted loss function is obtained;

[0238] The parameters of the student network learning model to be trained are adjusted according to the weighted loss function until the weighted loss function converges, and the trained student network learning model is obtained as the target neural network model.

[0239] In some embodiments, the training data set is determined by the following steps:

[0240] Acquire an original data set; the original data set includes multiple initial sample images;

[0241] Performing face recognition on the initial sample image, if a face exists, determining a first reference frame containing the face, and determining key point information of the face;

[0242] Determine the face posture information based on the key point information;

[0243] According to the posture information and the ratio of the first reference frame to the initial sample image, the first reference frame is updated to obtain a second reference frame;

[0244] According to the first reference frame and the second reference frame, marking sample labels for a portion of the sample images of the initial sample image located in the second reference frame, and using the portion of the sample images marked with sample labels as training images;

[0245] The sample labels include a first category label representing gaze, and a second category label and a third category label representing non-gaze; wherein,

[0246] The first category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is less than or equal to the first set value, the deflection angle of the face is less than or equal to the second set value, and the rotation angle of the face is less than or equal to the third set value;

[0247] The second category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is greater than the first set value, the deflection angle of the face is greater than the second set value, and the rotation angle of the face is greater than the third set value;

[0248] The third category label indicates that the ratio of the second reference frame in the training image to the training image is less than the first preset threshold and greater than the second preset threshold.

[0249] In some embodiments, the posture information includes at least the pitch angle, yaw angle, and rotation angle of the face;

[0250] According to the posture information and the ratio of the first reference frame to the initial sample image, the first reference frame is updated to obtain the second reference frame, including:

[0251] When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the yaw angle of the face is less than or equal to a second set value, and the rotation angle of the face is less than or equal to a third set value, the specific coordinate points in the first reference frame are moved within a first set range according to the position information of the first reference frame to obtain a second reference frame.

[0252] In some embodiments, according to the first reference frame and the second reference frame, marking sample labels for a portion of the sample images of the initial sample image located in the second reference frame, and using the portion of the sample images marked with sample labels as training images, includes:

[0253] Determining an overlapping area of ​​the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame;

[0254] When the first ratio of the overlapping area to the area of ​​the second reference frame is greater than or equal to the third preset threshold, a first preset range is generated according to the first preset probability, a second preset range is generated according to the second preset probability, a third preset range is generated according to the third preset probability, and a fourth preset range is generated according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1;

[0255] When the first ratio is within a first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the first category label, so that the first ratio is within the first preset range, and a second ratio between the number of training images marked with the first category label and the total number of all training images marked with the first category label in the training data set is a first preset probability;

[0256] When the first ratio is within a second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the first category label, so that the first ratio is within the second preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is a second preset probability;

[0257] When the first ratio is within a third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the first category label, so that the first ratio is within the third preset range, and a second ratio between the number of training images marked with the first category label and the total number of all training images marked with the first category label in the training data set is a third preset probability;

[0258] When the first ratio is within a fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the first category label, so that the first ratio is within the fourth preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is a fourth preset probability;

[0259] The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.

[0260] In some embodiments, the posture information includes at least the pitch angle, yaw angle, and rotation angle of the face;

[0261] According to the posture information and the ratio of the first reference frame to the initial sample image, the first reference frame is updated to obtain the second reference frame, including:

[0262] When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is greater than a first set value, the yaw angle of the face is greater than a second set value, and the rotation angle of the face is greater than a third set value, the specific coordinate points in the first reference frame are moved within a second set range according to the position information of the first reference frame to obtain a second reference frame.

[0263] In some embodiments, according to the first reference frame and the second reference frame, marking sample labels for a portion of the sample images of the initial sample image located in the second reference frame, and using the portion of the sample images marked with sample labels as training images, includes:

[0264] Determining an overlapping area of ​​the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame;

[0265] When the third ratio of the overlapping area to the area of ​​the second reference frame is greater than or equal to the third preset threshold, a first preset range is generated according to the first preset probability, a second preset range is generated according to the second preset probability, a third preset range is generated according to the third preset probability, and a fourth preset range is generated according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1;

[0266] When the third ratio is within the first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the second category label, so that the third ratio is within the first preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of all training images marked with the second category label in the training data set is a first preset probability;

[0267] When the third ratio is within a second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a second category label, so that the third ratio is within the second preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of all training images marked with the second category label in the training data set is a second preset probability;

[0268] When the third ratio is within a third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the second category label, so that the third ratio is within the third preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is a third preset probability;

[0269] When the third ratio is within a fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the second category label, so that the third ratio is within the fourth preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is a fourth preset probability;

[0270] The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.

[0271] In some embodiments, the posture information includes at least the pitch angle, yaw angle, and rotation angle of the face;

[0272] According to the posture information and the ratio of the first reference frame to the initial sample image, the first reference frame is updated to obtain the second reference frame, including:

[0273] When the ratio of the first reference frame to the initial sample image is less than the first preset threshold and greater than the second preset threshold, the specific coordinate points in the first reference frame are moved according to the position information of the first reference frame within a third set range to obtain the second reference frame; the first preset threshold is greater than the second preset threshold.

[0274] In some embodiments, according to the first reference frame and the second reference frame, marking sample labels for a portion of the sample images of the initial sample image located in the second reference frame, and using the portion of the sample images marked with sample labels as training images, includes:

[0275] Determining an overlapping area of ​​the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame;

[0276] When the fifth ratio of the overlapping area to the area of ​​the second reference frame is less than the third preset threshold, a fifth preset range is generated according to the fifth preset probability, a sixth preset range is generated according to the sixth preset probability, and a seventh preset range is generated according to the seventh preset probability; the fifth preset probability is greater than the sixth preset probability, and the sixth preset probability is greater than the seventh preset probability; the sum of the fifth preset probability, the sixth preset probability and the seventh preset probability is 1;

[0277] When the fifth ratio is within a fifth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the fifth ratio is within the fifth preset range, and a sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is a fifth preset probability;

[0278] When the fifth ratio is within a sixth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within a sixth preset range, and a sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is a sixth preset probability;

[0279] When the fifth ratio is within a seventh preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within a seventh preset range, and the sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is a seventh preset probability;

[0280] The data within the seventh preset range is smaller than the data within the sixth preset range, and the data within the sixth preset range is smaller than the data within the fifth preset range.

[0281] In some embodiments, the sample labels further include a fourth category label representing non-gaze;

[0282] In the step of obtaining the training data set, when face recognition is performed on the initial sample image to determine that no face exists, the step further includes: marking the initial sample image with a fourth category label, and using the initial sample image marked with the fourth category label as a training image.

[0283] In some embodiments, the sample labels further include a fourth category label representing non-gaze;

[0284] After performing face recognition on the initial sample image to determine that a face exists, and determining a first reference frame containing the face, the method further includes:

[0285] According to the size of the initial sample image and in accordance with a fourth set range, a target center point is determined in the initial sample image, and a third reference frame is determined with the target center point as the center;

[0286] If the third reference frame has no overlap with any first reference frame in the initial sample image, the portion of the sample image in the initial sample image located in the third reference frame is marked as a fourth category label, and the portion of the sample image marked with the fourth category label is used as a training image.

[0287] In addition, the embodiments of the present disclosure also provide a detection device corresponding to the control method of the monitoring system of the first aspect. Since the principle of solving the problem by the detection device in the embodiments of the present disclosure is similar to the control method of the monitoring system of the first aspect of the embodiments of the present disclosure, the implementation of the detection device can refer to the implementation of the corresponding method of the first aspect, and the repeated parts will not be repeated.

[0288] Fig. 9 A schematic diagram of a detection device provided in an embodiment of the present disclosure, such as Fig. 9 As shown, the detection device includes a data processing module 91. The detection device can be, for example, a storage and computing integrated chip integrating Flash or SRAM.

[0289] The data processing module 91 is configured to acquire a detection image, and use a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and send the recognition result to the terminal so that the terminal determines the display state at least based on the recognition result.

[0290] In some embodiments, the detection device also includes a model training module 92; the model training module 92 is configured to train the teacher network learning model to be trained according to the acquired training data set to obtain a trained teacher network learning model; according to the training data set and the trained teacher network learning model, the student network learning model to be trained is trained using the knowledge distillation training method to obtain the trained student network learning model as the target neural network model.

[0291] In some embodiments, the training data set includes multiple training images with labeled sample labels; the model training module 92 is specifically configured to input the training image into a trained teacher network learning model to obtain a first output result of the trained teacher network learning model; input the training image into a student network learning model to be trained to obtain a second output result of the student network learning model to be trained; determine a first loss function based on the first output result and the second output result; determine a second loss function based on the second output result and the sample labels of the training image; obtain a weighted loss function based on the first loss function and the second loss function; adjust the parameters of the student network learning model to be trained according to the weighted loss function until the weighted loss function converges, and obtain the trained student network learning model as the target neural network model.

[0292] In some embodiments, the detection device further includes a sample generation module 93, which is configured to obtain an original data set; the original data set includes multiple initial sample images; perform face recognition on the initial sample image, and if a face exists, determine a first reference frame containing the face, and determine key point information of the face; determine the posture information of the face according to the key point information; update the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain a second reference frame; mark sample labels for part of the sample images of the initial sample image located in the second reference frame according to the first reference frame and the second reference frame, and use the part of the sample images marked with sample labels as training images; the sample labels include a first category label representing gaze, and A second category label and a third category label characterizing non-gaze; wherein the first category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is less than or equal to the first set value, the yaw angle of the face is less than or equal to the second set value, and the rotation angle of the face is less than or equal to the third set value; the second category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is greater than the first set value, the yaw angle of the face is greater than the second set value, and the rotation angle of the face is greater than the third set value; the third category label indicates that the ratio of the second reference frame in the training image to the training image is less than the first preset threshold and greater than the second preset threshold.

[0293] In some embodiments, the posture information includes at least the pitch angle, yaw angle and rotation angle of the face; the sample generation module 93 is specifically configured to move the specific coordinate point in the first reference frame within a first set range to obtain the second reference frame according to the position information of the first reference frame when the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the yaw angle of the face is less than or equal to a second set value, and the rotation angle of the face is less than or equal to a third set value.

[0294] Further, the sample generation module 93 is specifically configured to determine the overlapping area of ​​the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame; when the first ratio of the overlapping area to the area of ​​the second reference frame is greater than or equal to the third preset threshold, generate the first preset range according to the first preset probability, generate the second preset range according to the second preset probability, generate the third preset range according to the third preset probability, and generate the fourth preset range according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1; when the first ratio is within the first preset range, determine and mark the part of the sample image located in the second reference frame in the initial sample image as the first category label, so that the first ratio is within the first preset range, and the second ratio between the number of training images marked with the first category label and the total number of all training images marked with the first category label in the training data set is the first preset probability; when the first ratio is within the second preset range, determine and mark the part of the sample image located in the second reference frame in the initial sample image as the first category label. Part of the sample images in the reference frame are labeled with the first category, so that the first ratio is within the second preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the second preset probability; when the first ratio is within the third preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the first category label, so that the first ratio is within the third preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the third preset probability; when the first ratio is within the fourth preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the first category label, so that the first ratio is within the fourth preset range, and the second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the fourth preset probability; the data in the first preset range is smaller than the data in the second preset range, the data in the second preset range is smaller than the data in the third preset range, and the data in the third preset range is smaller than the data in the fourth preset range.

[0295] In some embodiments, the posture information includes at least the pitch angle, yaw angle and rotation angle of the face; the sample generation module 93 is specifically configured to move the specific coordinate point in the first reference frame within a second set range to obtain the second reference frame according to the position information of the first reference frame when the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is greater than a first set value, the yaw angle of the face is greater than a second set value, and the rotation angle of the face is greater than a third set value.

[0296] Further, the sample generation module 93 is specifically configured to determine the overlapping area of ​​the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame; when the third ratio of the overlapping area to the area of ​​the second reference frame is greater than or equal to the third preset threshold, generate the first preset range according to the first preset probability, generate the second preset range according to the second preset probability, generate the third preset range according to the third preset probability, and generate the fourth preset range according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1; when the third ratio is within the first preset range, determine and mark the part of the sample image located in the second reference frame in the initial sample image as the second category label, so that the third ratio is within the first preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the first preset probability; when the third ratio is within the second preset range, determine and mark the part of the sample image located in the second reference frame in the initial sample image as the second category label. Part of the sample images in the reference frame are labeled as the second category, so that the third ratio is within the second preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the second preset probability; when the third ratio is within the third preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the second category label, so that the third ratio is within the third preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the third preset probability; when the third ratio is within the fourth preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the second category label, so that the third ratio is within the fourth preset range, and the fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the fourth preset probability; the data in the first preset range is smaller than the data in the second preset range, the data in the second preset range is smaller than the data in the third preset range, and the data in the third preset range is smaller than the data in the fourth preset range.

[0297] In some embodiments, the posture information includes at least the pitch angle, yaw angle and rotation angle of the face; the sample generation module 93 is specifically configured to move the specific coordinate points in the first reference frame within a third set range according to the position information of the first reference frame to obtain the second reference frame when the ratio of the first reference frame to the initial sample image is less than a first preset threshold and greater than a second preset threshold; the first preset threshold is greater than the second preset threshold.

[0298] Further, the sample generation module 93 is specifically configured to determine the overlapping area of ​​the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame; when the fifth ratio of the overlapping area to the area of ​​the second reference frame is less than the third preset threshold, generate the fifth preset range according to the fifth preset probability, generate the sixth preset range according to the sixth preset probability, and generate the seventh preset range according to the seventh preset probability; the fifth preset probability is greater than the sixth preset probability, and the sixth preset probability is greater than the seventh preset probability; the sum of the fifth preset probability, the sixth preset probability and the seventh preset probability is 1; when the fifth ratio is within the fifth preset range, determine and mark the part of the sample image located in the second reference frame in the initial sample image as the third category label, so that the number of training images with the fifth ratio within the fifth preset range and marked with the third category label is equal to all the training images marked with the third category label in the training data set. The sixth ratio between the total numbers is the fifth preset probability; when the fifth ratio is within the sixth preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the third category label, so that the sixth ratio is within the sixth preset range, and the sixth ratio between the number of training images marked with the third category label and the total number of all training images marked with the third category label in the training data set is the sixth preset probability; when the fifth ratio is within the seventh preset range, part of the sample images in the initial sample image located in the second reference frame are determined and marked as the third category label, so that the sixth ratio is within the seventh preset range, and the sixth ratio between the number of training images marked with the third category label and the total number of all training images marked with the third category label in the training data set is the seventh preset probability; the data in the seventh preset range is smaller than the data in the sixth preset range, and the data in the sixth preset range is smaller than the data in the fifth preset range.

[0299] In some embodiments, when face recognition is performed on the initial sample image and it is determined that no face exists, the initial sample image is labeled with a fourth category label, and the initial sample image labeled with the fourth category label is used as a training image.

[0300] In some embodiments, the sample label also includes a fourth category label that characterizes non-gaze; the sample generation module 93 is also configured to determine a target center point in the initial sample image according to the size of the initial sample image and within a fourth set range, and to determine a third reference frame centered on the target center point; if the third reference frame has no overlap with any first reference frame in the initial sample image, then the portion of the sample image of the initial sample image that is located in the third reference frame is marked as a fourth category label, and the portion of the sample image marked with the fourth category label is used as a training image.

[0301] In addition, the embodiments of the present disclosure also provide a control device for the monitoring system corresponding to the control method for the monitoring system of the second aspect. Since the principle of solving the problem by the control device for the monitoring system in the embodiments of the present disclosure is similar to the control method for the monitoring system of the second aspect of the embodiments of the present disclosure, the implementation of the control device for the monitoring system can refer to the implementation of the corresponding method of the second aspect, and the repeated parts will not be repeated.

[0302] Fig.10 A schematic diagram of a control device of a monitoring system provided in an embodiment of the present disclosure, such as Fig.10 As shown, the detection device 101 and the terminal 102. The detection device 101 can be, for example, a storage and computing integrated chip integrating Flash or SRAM.

[0303] The detection device 101 is configured to obtain a detection image, and use a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and send the recognition result to the terminal;

[0304] The terminal 102 is configured to determine a display state based on the recognition result.

[0305] In some embodiments, the terminal 102 includes a display module 1022 and a main control module 1021; the main control module 1021 is configured to send a wake-up request to the display module when it is determined that the recognition result indicates that the object is staring; when it is determined that the recognition result indicates that the object is not staring, and the current system time is greater than a preset time from the last time the display module was awakened, send a sleep request to the display module; the display module 1022 is configured to respond to the wake-up request and determine that the display state is normal display; and respond to the sleep request and determine that the display state is sleep.

[0306] Fig.11 The following is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. Fig.11 As shown, the embodiment of the present disclosure provides a computer device including: one or more processors 111, a memory 112, and one or more I / O interfaces 113. The memory 112 stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the control method of the monitoring system in any of the above embodiments; the one or more I / O interfaces 113 are connected between the processor and the memory, and are configured to implement information interaction between the processor and the memory.

[0307] Among them, the processor 111 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 112 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 113 is connected between the processor 111 and the memory 112, and can realize information interaction between the processor 111 and the memory 112, including but not limited to a data bus (Bus), etc.

[0308] In some embodiments, the processor 111 , the memory 112 , and the I / O interface 113 are connected to each other via a bus 114 , and further connected to other components of the computing device.

[0309] According to an embodiment of the present disclosure, a non-transitory computer readable storage medium is also provided. The non-transitory computer readable storage medium stores a computer program, wherein when the program is executed by a processor, the steps in the control method of the monitoring system in any of the above embodiments are implemented.

[0310] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the system of the present disclosure are executed.

[0311] It should be noted that the computer non-transient readable medium shown in the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the above two. The computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable non-transient storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable non-transient storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0312] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the aforementioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two connected boxes can actually represent that they are executed in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0313] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A control method for a monitoring system, wherein: include: Acquire detection images; Using a pre-trained target neural network model, performing gaze recognition on the detection image to obtain a recognition result of the detection image; The recognition result is sent to a terminal so that the terminal determines a display state based at least on the recognition result.

2. The control method of the monitoring system according to claim 1, wherein: The target neural network model is trained by the following steps: Training the teacher network learning model to be trained according to the acquired training data set to obtain a trained teacher network learning model; According to the training data set and the trained teacher network learning model, the student network learning model to be trained is trained using a knowledge distillation training method to obtain a trained student network learning model as the target neural network model.

3. The control method of the monitoring system according to claim 2, wherein: The training data set includes a plurality of training images annotated with sample labels; The method of training the student network learning model to be trained by using the training method of knowledge distillation according to the training data set and the trained teacher network learning model to obtain the trained student network learning model as the target neural network model includes: Inputting the training image into the trained teacher network learning model to obtain a first output result of the trained teacher network learning model; Inputting the training image into the student network learning model to be trained to obtain a second output result of the student network learning model to be trained; Determine a first loss function according to the first output result and the second output result; Determining a second loss function according to the second output result and the sample label of the training image; Obtaining a weighted loss function according to the first loss function and the second loss function; The parameters of the student network learning model to be trained are adjusted according to the weighted loss function until the weighted loss function converges, thereby obtaining a trained student network learning model as the target neural network model.

4. The control method of the monitoring system according to claim 3, wherein: The training data set is determined by the following steps: Acquire an original data set; the original data set includes a plurality of initial sample images; Performing face recognition on the initial sample image, and if a face exists, determining a first reference frame containing the face, and determining key point information of the face; Determining facial posture information according to the key point information; According to the posture information and the ratio of the first reference frame to the initial sample image, the first reference frame is updated to obtain a second reference frame; According to the first reference frame and the second reference frame, marking the sample label for a portion of the sample image of the initial sample image located in the second reference frame, and using the portion of the sample image marked with the sample label as the training image; The sample labels include a first category label representing gaze, and a second category label and a third category label representing non-gaze; wherein, The first category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the deflection angle of the face is less than or equal to a second set value, or the rotation angle of the face is less than or equal to a third set value; The second category label indicates that when the ratio of the second reference frame in the training image to the training image is greater than or equal to the first preset threshold, the pitch angle of the face is greater than the first set value, the deflection angle of the face is greater than the second set value, or the rotation angle of the face is greater than the third set value; The third category label indicates that a ratio of the second reference frame in the training image to the training image is less than a first preset threshold and greater than a second preset threshold.

5. The control method of the monitoring system according to claim 4, wherein: The posture information includes at least the pitch angle, yaw angle and rotation angle of the face; The updating of the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain the second reference frame includes: When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is less than or equal to a first set value, the yaw angle of the face is less than or equal to a second set value, and the rotation angle of the face is less than or equal to a third set value, the specific coordinate points in the first reference frame are moved within a first set range according to the position information of the first reference frame to obtain a second reference frame.

6. The control method of the monitoring system according to claim 5, wherein: The step of marking the sample labels for the portion of the sample images of the initial sample image located in the second reference frame according to the first reference frame and the second reference frame, and using the portion of the sample images marked with the sample labels as the training images, comprises: determining an overlapping area between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame; When the first ratio of the overlapping area to the area of ​​the second reference frame is greater than or equal to a third preset threshold, a first preset range is generated according to a first preset probability, a second preset range is generated according to a second preset probability, a third preset range is generated according to a third preset probability, and a fourth preset range is generated according to a fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1; When the first ratio is within the first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the first preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the first preset probability; When the first ratio is within the second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the first category label, so that the first ratio is within the second preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the second preset probability; When the first ratio is within the third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the third preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is within the third preset probability; When the first ratio is within the fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a first category label, so that the first ratio is within the fourth preset range, and a second ratio between the number of training images marked with the first category label and the total number of training images marked with the first category label in the training data set is the fourth preset probability; The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.

7. The control method of the monitoring system according to claim 4, wherein: The posture information includes at least the pitch angle, yaw angle and rotation angle of the face; The updating of the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain the second reference frame includes: When the ratio of the first reference frame to the initial sample image is greater than or equal to a first preset threshold, the pitch angle of the face is greater than a first set value, the deflection angle of the face is greater than a second set value, and the rotation angle of the face is greater than a third set value, the specific coordinate points in the first reference frame are moved within a second set range according to the position information of the first reference frame to obtain a second reference frame.

8. The control method of the monitoring system according to claim 7, wherein: The step of marking the sample labels for the portion of the sample images of the initial sample image located in the second reference frame according to the first reference frame and the second reference frame, and using the portion of the sample images marked with the sample labels as the training images, comprises: determining an overlapping area between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame; When a third ratio of the overlapping area to the area of ​​the second reference frame is greater than or equal to a third preset threshold, a first preset range is generated according to a first preset probability, a second preset range is generated according to a second preset probability, a third preset range is generated according to a third preset probability, and a fourth preset range is generated according to a fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; and the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1; When the third ratio is within the first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a second category label, so that the third ratio is within the first preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the first preset probability; When the third ratio is within the second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the second category label, so that the third ratio is within the second preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the second preset probability; When the third ratio is within the third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a second category label, so that the third ratio is within the third preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the third preset probability; When the third ratio is within the fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a second category label, so that the third ratio is within the fourth preset range, and a fourth ratio between the number of training images marked with the second category label and the total number of training images marked with the second category label in the training data set is the fourth preset probability; The data within the first preset range is smaller than the data within the second preset range, the data within the second preset range is smaller than the data within the third preset range, and the data within the third preset range is smaller than the data within the fourth preset range.

9. The control method of the monitoring system according to claim 4, wherein: The posture information includes at least the pitch angle, yaw angle and rotation angle of the face; The updating of the first reference frame according to the posture information and the ratio of the first reference frame to the initial sample image to obtain the second reference frame includes: When the ratio of the first reference frame to the initial sample image is less than a first preset threshold and greater than a second preset threshold, the specific coordinate points in the first reference frame are moved within a third set range according to the position information of the first reference frame to obtain a second reference frame; and the first preset threshold is greater than the second preset threshold.

10. The control method of the monitoring system according to claim 9, wherein: The step of marking the sample labels for the portion of the sample images of the initial sample image located in the second reference frame according to the first reference frame and the second reference frame, and using the portion of the sample images marked with the sample labels as the training images, comprises: determining an overlapping area between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame; When a fifth ratio of the overlapping area to the area of ​​the second reference frame is less than a third preset threshold, a fifth preset range is generated according to a fifth preset probability, a sixth preset range is generated according to a sixth preset probability, and a seventh preset range is generated according to a seventh preset probability; the fifth preset probability is greater than the sixth preset probability, and the sixth preset probability is greater than the seventh preset probability; and the sum of the fifth preset probability, the sixth preset probability and the seventh preset probability is 1; When the fifth ratio is within the fifth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as a third category label, so that the fifth ratio is within the fifth preset range, and a sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the fifth preset probability; When the fifth ratio is within the sixth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within the sixth preset range, and a sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the sixth preset probability; When the fifth ratio is within the seventh preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as the third category label, so that the sixth ratio is within the seventh preset range, and the sixth ratio between the number of training images marked with the third category label and the total number of training images marked with the third category label in the training data set is the seventh preset probability; The data within the seventh preset range is smaller than the data within the sixth preset range, and the data within the sixth preset range is smaller than the data within the fifth preset range.

11. The control method of the monitoring system according to claim 4, wherein: The sample labels also include a fourth category label representing non-gaze; In the step of obtaining the training data set, when face recognition is performed on the initial sample image to determine that no face exists, the step further includes: The initial sample image is labeled with a fourth category label, and the initial sample image labeled with the fourth category label is used as the training image.

12. The control method of the monitoring system according to claim 4, wherein: The sample labels also include a fourth category label representing non-gaze; After performing face recognition on the initial sample image to determine that a face exists, and determining a first reference frame containing the face, the method further includes: According to the size of the initial sample image and in accordance with a fourth set range, a target center point is determined in the initial sample image, and a third reference frame is determined with the target center point as the center; If the third reference frame has no overlap with any first reference frame in the initial sample image, the portion of the sample image of the initial sample image located in the third reference frame is marked as a fourth category label, and the portion of the sample image marked with the fourth category label is used as the training image.

13. A control method for a monitoring system, wherein: The monitoring system includes a detection device and a terminal; the control method of the monitoring system includes: The detection device acquires a detection image, and uses a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and sends the recognition result to the terminal; The terminal determines a display state based on the recognition result.

14. The control method of the monitoring system according to claim 13, wherein: The terminal includes a display module and a main control module; The terminal determines a display state based on the recognition result, including: The main control module sends a wake-up request to the display module when determining that the recognition result indicates that the object is gazing; When it is determined that the recognition result indicates that there is no object gaze, and the time from the current system time to the last time the display module was awakened is greater than a preset time, sending a sleep request to the display module; The display module responds to the wake-up request to determine that the display state is normal display; and responds to the sleep request to determine that the display state is sleep.

15. A detection device, configured to acquire a detection image, and use a pre-trained target neural network model to perform gaze recognition on the detection image to obtain a recognition result of the detection image; and send the recognition result to a terminal so that the terminal determines a display state based at least on the recognition result.

16. A control device for a monitoring system, comprising a detection device and a terminal; The detection device is configured to acquire a detection image, and perform gaze recognition on the detection image using a pre-trained target neural network model to obtain a recognition result of the detection image; and send the recognition result to a terminal; The terminal is configured to determine a display state based on the recognition result.

17. A computer device, wherein: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the control method of the monitoring system as described in any one of claims 1 to 12 are performed, and / or the steps of the control method of the monitoring system as described in any one of claims 13 to 14 are performed.

18. A computer non-transitory readable storage medium, wherein: The computer non-volatile readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the control method of the monitoring system as described in any one of claims 1 to 12, and / or the steps of the control method of the monitoring system as described in any one of claims 13 to 14.

Citation Information

Patent Citations

  • Face recognition model training method, face recognition method and device

    CN111582141A

  • Agricultural pest image recognition method based on CNN few samples

    CN113177612A

  • Screen control method and device of electronic equipment, electronic equipment and storage medium

    CN113885708A

  • Knowledge distillation-based coal gangue detection method and device

    CN116012635A

  • Eye movement tracking method and device, electronic equipment, storage medium and program product

    CN116091541A