Method for determining a nose swab collection action in an antigen detection process and storage medium

Abstract

This application relates to the field of image recognition technology, and particularly to a method for determining the nasal swab collection action and a storage medium during antigen detection. The method includes the following steps: Obtain N frame video images P of a target object within T0 time during antigen detection; Traverse P, if there is a face in P i , append the face region image in P i to P 1 , and obtain M frame face region images P 1 ; Use a trained first neural network to determine whether P 1 j is a valid collection image, and obtain the number Q of A valid collection pictures during antigen detection; Traverse Q, if Q x ≥Q0, then T 1 x =T 1 x‑1 +T1; if Q x <Q0, then T 1 x =0; if T 1 x during antigen detection are all less than T, indicate non-compliance with the operation standard on the mobile terminal; otherwise, indicate compliance with the operation standard. The present invention realizes an objective determination of whether the nasal swab collection action meets the operation standard.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method and storage medium for determining a nasal swab collection action during an antigen detection process. Background Art

[0002] With the advent of home testing kits, ordinary people can easily perform home testing for some diseases. For example, with the advent of home testing kits for COVID-19, ordinary people can conduct nucleic acid tests at home. Using nasal swabs to collect specimens is a relatively common collection method. Whether the tester's collection action meets the operating standards during the collection process directly affects the accuracy of the antigen test results. However, the tester often cannot make a relatively objective judgment on whether their collection action meets the operating standards. Therefore, how to objectively determine whether the nasal swab collection action meets the operating standards is a technical problem that needs to be solved urgently. Summary of the Invention

[0003] The purpose of the present invention is to provide a method and storage medium for determining the nasal swab collection action during antigen detection, so as to objectively determine whether the nasal swab collection action meets the operating standards.

[0004] According to a first aspect of the present invention, a method for determining a nasal swab collection action during an antigen detection process is provided, comprising the following steps:

[0005] S100, using the camera of the mobile terminal to obtain N frames of video images P = (P1, P2, ..., P N ), P i is the i-th frame of the video image, the value of i ranges from 1 to N, N is a positive integer, T0 is the total duration of the antigen detection process, T0>0s;

[0006] S200, traverse P, if P i If there is a face in P i The face area image is appended to P 1 , get M frames of face area image P 1 =(P 1 1, P 1 2,…,P 1 M ), P 1 Initialized to Null, P 1 j is the face region image of the jth frame, the value of j ranges from 1 to M, M is a positive integer, M≤N;

[0007] S300, using the trained first neural network to determine P 1 jWhether it is a valid collected image, obtaining the number Q of A valid collected pictures during the antigen detection process Q=(Q1, Q2, …, Q A ); Q x is the number of valid collected images within the x-th preset time period during the antigen detection process, where the value range of x is from 1 to A, A is a positive integer, the duration of the preset time period is T1, and T1 < T0; the valid collection means that the nasal swab collection action conforms to the operation standard;

[0008] S400, traverse Q, if Q x ≥Q0, then T 1 x =T 1 x-1 +T1; if Q x <Q0, then T 1 x =0; T 1 x is the cumulative continuous valid collection duration of the x-th preset time period during the antigen detection process, T 1 x-1 is the cumulative continuous valid collection duration of the (x - 1)-th preset time period during the antigen detection process, Q0 is a preset quantity threshold, and T 1 0 = 0s;

[0009] S500, if T 1 x during the antigen detection process are all less than T, then present a first prompt message indicating non-compliance with the operation standard on the mobile terminal; if there exists T 1 x greater than or equal to T during the antigen detection process, then present a second prompt message indicating compliance with the operation standard on the mobile terminal; T is the shortest time that the nasal swab is required to stay in the nose in the operation standard of antigen detection.

[0010] According to the second aspect of the present invention, there is provided a non-transitory computer-readable storage medium, in which at least one instruction or at least one program segment is stored, and the at least one instruction or the at least one program segment is loaded and executed by a processor to perform the method described in the first aspect of the present invention.

[0011] Compared with the prior art, the present invention has obvious beneficial effects. By means of the above technical solutions, a method for determining the nasal swab collection action during antigen detection and a storage medium provided by the present invention can achieve considerable technical progressiveness and practicability, and have wide industrial utilization value. It has at least the following beneficial effects:

[0012] The present invention divides the antigen detection process of the target object into multiple preset time periods. If Q x≥Q0, indicating that the number of valid collected images in the preset time period exceeds the preset number threshold, and the preset time period is determined to be a valid collection time period; by judging whether the cumulative continuous valid collection time during the antigen test process exceeds the minimum time required by the nasal swab to stay in the nose in the operation standard, it is possible to judge whether the target subject's nasal swab collection action during the antigen test process meets the operation standard;

[0013] The present invention uses a video image captured by a mobile terminal's camera to ultimately display on the mobile terminal whether the target subject's nasal swab collection action meets the operating standards. This method of obtaining the determination result is relatively simple and convenient for the target subject, making it more easily accepted.

[0014] The present invention determines whether each face area image is a valid captured image based on the first trained neural network. The present invention generally makes automatic judgments, and the judgment results are not mixed with subjective factors of the target object. The judgment results are relatively objective and have high credibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 This is a flowchart of a method for determining nasal swab collection action during antigen detection provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0018] According to a first aspect of the present invention, a method for determining the nasal swab collection action during antigen detection is provided. Figure 1 As shown, the following steps are included:

[0019] S100, using the camera of the mobile terminal to obtain N frames of video images P = (P1, P2, ..., P N ), P iis the i-th frame of video image, the value of i ranges from 1 to N, N is a positive integer, T0 is the total duration of the antigen detection process, T0>0s.

[0020] It is understandable that the target object is a person who undergoes an antigen test, and the target object uses a home test kit to perform an antigen test. As an example, the home test kit is a home test kit for the new coronavirus. Before the target object undergoes the antigen test, the position of the camera of the mobile terminal is fixed and the target object's face is photographed. As a result, the camera of the mobile terminal can capture the process of the target object undergoing the antigen test. Optionally, the camera of the mobile terminal captures 25 frames of video images or 30 frames of video images per second. Optionally, the processor or server of the mobile terminal judges whether the nasal swab collection action of the target object meets the operating standards based on the captured video. It should be understood that the mobile terminal is a mobile phone, notebook or tablet computer, etc.

[0021] It can be understood that the mobile terminal camera captures N frames of video images within the time T0, and then N / T0 is the sampling frequency of the mobile terminal camera.

[0022] S200, traverse P, if P i If there is a face in P i The face area image is appended to P 1 , get M frames of face area image P 1 =(P 1 1, P 1 2,…,P 1 M ), P 1 Initialized to Null, P 1 j is the face area image of the jth frame, the value range of j is 1 to M, M is a positive integer, M≤N.

[0023] It is understood that the face detection method can be used to identify whether a face exists in each video frame in P. Furthermore, the face region can be marked with a rectangular frame in the video image where a face exists. The face region image can be extracted by extracting the region marked with the rectangular frame in the video image where a face exists. It is understood that each video frame containing a face corresponds to a face region image. The face detection method is an existing technology and is optional. The face detection method is a face detection method based on a neural network.

[0024] It can be understood that the video images in P above may either be video images with faces or images without faces. Therefore, the number M of face region images obtained is less than or equal to the number N of video images in P. It can be understood that if all the video images in P are video images with faces, then M = N; if the video images in P have faces starting from the 3rd frame video image, then M = N - 2.

[0025] It can be understood that the target object can only perform the nasal swab collection action when there is a face in the video image. Therefore, analyzing only the video images with faces can greatly reduce the later calculation amount without affecting the accuracy of the later determination result.

[0026] S300, use the trained first neural network to determine whether P 1 j is a valid collection image, and obtain the number Q = (Q1, Q2,..., Q A ) of A valid collection pictures during the antigen detection process; Q x is the number of valid collection images in the xth preset time period during the antigen detection process, where the value range of x is from 1 to A, A is a positive integer, the duration of the preset time period is T1, and T1 < T0; the valid collection means that the nasal swab collection action meets the operation standard.

[0027] Among them, the training process of the first neural network further includes:

[0028] S310, obtain n antigen detection videos V = {V1, V2,..., V n}), V y is the yth antigen detection video, where the value range of y is from 1 to n, n is a positive integer, and V y includes multiple frames of video images with faces;

[0029] S320, manually label the categories of the face region images corresponding to V y , and the categories include positive samples and negative samples;

[0030] S330, use the manually labeled face region images as training samples to train the first neural network.

[0031] It is understood that a binary classification problem can be solved using a neural network, and determining the type of facial region image in each frame is a binary classification problem. There are two types of facial region images: one is a valid collection image in which the nasal swab collection action meets the operating standards, that is, the target subject places the nasal swab in the nose and the depth of the swab is sufficient to effectively collect the sample. It is understood that the facial region image corresponding to the valid collection image includes not only the human face but also the nasal swab and the hand used to hold the nasal swab, and the depth of the nasal swab entering the nose is sufficient to effectively collect the sample; the other is an invalid collection image in which the nasal swab collection action does not meet the operating standards. It is understood that the facial region image corresponding to the invalid collection image may not include the nasal swab and the hand used to hold the nasal swab, or may include the nasal swab and the hand used to hold the nasal swab, but the depth of the nasal swab entering the nose is insufficient to effectively collect the sample. Optionally, the neural network is a BP neural network or a convolutional neural network.

[0032] It is understandable that in order for the neural network to have the ability to classify facial area images, the neural network needs to be trained, and the cross entropy loss function is used during the training process. In order to obtain a neural network with a high accuracy, the construction of training samples is particularly important. Preferably, a training sample is constructed based on n antigen detection videos V. It is understandable that an antigen detection video is a video obtained by a tester recording the antigen detection process. An antigen detection video includes multiple frames of video images, such as a sampling frequency of 25 frames per second. The number of video images is the total duration of the video multiplied by 25. According to the method of step S200, the facial area image corresponding to each antigen detection video can be obtained.

[0033] As a preferred example, the training process of the neural network adopts a supervised learning method. After obtaining each antigen detection video V y After the corresponding face area image is obtained, each antigen detection video V is manually labeled. y The corresponding face area images are labeled. There are two types of labels: positive samples, which correspond to valid collected images; and negative samples, which correspond to invalid collected images.

[0034] As an experiment, 321 antigen detection videos (n=321) were obtained. These 321 antigen detection videos included 571,535 video frames. By collecting samples every 15 frames, the number of sampled video frames was 38,102, and the number of video frames with detected faces was 9,851. The facial region images corresponding to these 9,851 video frames were used for manual annotation. After annotation, 1,866 positive samples and 1,864 negative samples were obtained, resulting in a total of 3,730 positive and negative samples. Of the total number of samples, 2,983 were used as training samples, and the remaining 747 were used as test samples. The training samples included 1,491 positive samples and 1,492 negative samples; the test samples included 373 positive samples and 374 negative samples. The training results of this experiment are shown in Table 1:

[0035] Table 1

[0036] loss Accuracy Accuracy Recall F1 training set 0.0092 0.9970 0.9973 0.9966 0.9970 Test set 0.0677 0.9839 0.9788 0.9893 0.9840

[0037] Table 1 shows that the neural network trained based on the antigen detection video can achieve a high accuracy in determining positive and negative samples. The accuracy of the neural network obtained according to the above example is 0.9839.

[0038] It is understandable that after the neural network training is completed, the trained neural network can be used to realize the P 1 Determine whether each frame of face area image is a valid captured image.

[0039] The antigen detection process of the target object is divided into multiple preset time periods for counting the number of valid collected images. As an example, the total duration of the antigen detection process of the target object is T0 = 10s, and the preset time period is T1 = 0.4s. Starting from the start time of the antigen detection process, the number of valid collected pictures is counted every 0.4s. If the acquisition frequency of the mobile terminal camera is 25 frames per second, that is, the number of valid collected images is counted every 10 frames of video images, the number of 25 valid collected images in the antigen detection process can be obtained, which are the numbers of the 1st, 2nd, ..., 25th 0.4s valid collected images in the antigen detection process.

[0040] As another example, the total duration of the antigen detection process of the target object is T0 = 10s, and the preset time period is T1 = 2s. Starting from the start time of the antigen detection process, the number of valid collected pictures is counted every 2s. If the acquisition frequency of the mobile terminal camera is 25 frames per second, that is, the number of valid collected images is counted every 50 frames of video images, the number of 5 valid collected images in the antigen detection process can be obtained, which are the numbers of the 1st, 2nd, ..., 5th 2s valid collected images in the antigen detection process.

[0041] S400 traverses Q. If Q x ≥ Q0, then T 1 x = T 1 x-1 + T1; if Q x < Q0, then T 1 x = 0; T 1 x is the cumulative continuous effective acquisition duration in the x-th preset time period during the antigen detection process, T 1 x-1 is the cumulative continuous effective acquisition duration in the (x - 1)-th preset time period during the antigen detection process, Q0 is the preset quantity threshold, T 1 0 = 0s.

[0042] For the x-th preset time period during the antigen detection process, Q x being greater than or equal to the preset quantity threshold Q0 includes two cases. The first case is that there are Q0 consecutive frames of effective acquisition images in the x-th preset time period. The second case is that although there are Q0 frames of effective acquisition images in the x-th preset time period, these Q0 frames of effective acquisition images are not consecutive; it should be understood that generally, the process of the target object undergoing antigen detection is a continuous process. The reason for the second case often is not due to the interruption of the sampling process, but other factors, such as the misjudgment of the trained neural network. Therefore, compared with judging whether there are Q0 consecutive frames of effective acquisition images, only judging whether Q x is greater than or equal to the preset quantity threshold Q0 can improve the accuracy of the judgment result.

[0043] Preferably, set the duration T1 of the preset time period < T, where T is the shortest time required for the nasal swab to stay in the nose in the operation standard of antigen detection. For example, T = 2s. Small-scale experiments show that when T1 is set to be relatively small, the accuracy of determining the x-th preset time period as an effective acquisition time period under the condition of Q x ≥ Q0 is higher.

[0044] Preferably, Q0 satisfies the following conditions:

[0045] Q0 = roundup(k * T1 * N / T0)+ N0

[0046] Q0 < T1 * N / T0

[0047] where, N0 is the preset number of frames, N0 ≥ 0 and N0 is an integer; k is the error tolerance coefficient, 0 < k < 1, and roundup is the ceiling function.

[0048] Preferably, the value range of k is 0.8 ≤ k < 1. Small-scale experiments have shown that when the value of k is set within 0.8 ≤ k < 1, the accuracy of the final determination is relatively high.

[0049] It can be understood that when Q x ≥ Q0, it indicates that the x-th preset time period is a valid acquisition time period. Then, the cumulative continuous valid acquisition duration T 1 x of the x-th preset time period is the sum of the cumulative continuous valid acquisition duration T 1 x-1 of the (x - 1)-th preset time period and T1; when Q x < Q0, it indicates that the (x - 1)-th preset time period is an invalid acquisition time period, indicating that the cumulative continuous valid acquisition before the x-th preset time period is interrupted. Then, the cumulative continuous valid acquisition duration T 1 x of the x-th preset time period is 0s.

[0050] S500, if T 1 x is less than T during the antigen detection process, a first prompt message indicating non-compliance with the operation standard will be presented on the mobile terminal; if there is a T 1 x greater than or equal to T during the antigen detection process, a second prompt message indicating compliance with the operation standard will be presented on the mobile terminal; T is the shortest time required for the nasal swab to stay in the nose in the operation standard of antigen detection.

[0051] It can be understood that in order to achieve effective acquisition of the sample, there may also be certain requirements for the stay time T of the nasal swab in the nose in the operation standard, T > 0s; as long as the cumulative continuous valid acquisition duration of one preset time period during the antigen detection process is greater than or equal to T, it can be determined that the antigen detection process complies with the operation standard; then, a prompt message indicating compliance with the operation standard can be presented on the mobile terminal. Preferably, it can be presented in the form of green font on the mobile terminal or in the form of a green background on the mobile terminal, and a prompt sound indicating compliance with the operation standard is also emitted on the mobile terminal.

[0052] It can be understood that if T 1 x is less than T during the antigen detection process, it indicates that the antigen detection process of the target object does not comply with the operation standard. Then, a prompt message indicating non-compliance with the operation standard can be presented on the mobile terminal. Preferably, it can be presented in the form of red font on the mobile terminal or in the form of a red background on the mobile terminal, and a prompt sound indicating non-compliance with the operation standard is also emitted on the mobile terminal.

[0053] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded by a processor and executes the method for determining the nasal swab collection action during the antigen detection process described in an embodiment of the present invention.

[0054] Although some specific embodiments of the present invention have been described in detail by way of example, it will be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for determining nasal swab collection action during antigen detection, characterized in that: It includes the following steps: S100, using the camera of the mobile terminal to obtain N frames of video images P = (P1, P2, ..., P N ), P i is the i-th frame of the video image, the value of i ranges from 1 to N, N is the total number of frames in the antigen detection process, N is a positive integer, T0 is the total duration of the antigen detection process, T0>0s; S200, traverse P, if P i If there is a face in P i The face area image is appended to P 1 , get M frames of face area image P 1 =(P 1 1, P 1 2,…,P 1 M ), P 1 Initialized to Null, P 1 j is the face region image of the jth frame, the value of j ranges from 1 to M, M is a positive integer, M≤N; S300, use the trained first neural network to determine P 1 j whether it is a valid acquisition image, and obtain the number Q=(Q1, Q2,..., Q A ) of A valid acquisition pictures during the antigen detection process; Q x is the number of valid acquisition images within the x-th preset time period during the antigen detection process, where the value range of x is from 1 to A, A is a positive integer, the duration of the preset time period is T1, and T1 < T0; the valid acquisition means that the nasal swab collection action conforms to the operation standard; S400, traverse Q, if Q x ≥Q0, then T 1 x =T 1 x-1 +T1; if Q x <Q0, then T 1 x =0; T 1 x is the cumulative continuous effective collection duration in the x-th preset time period during the antigen detection process, T 1 x-1 is the cumulative continuous effective collection duration in the (x - 1)-th preset time period during the antigen detection process, Q0 is the preset quantity threshold, T 1 0 = 0s; S500, if the antigen test process T 1 x are all less than T, then the first prompt information indicating that the operation standard is not met is presented on the mobile terminal; if T 1 x If it is greater than or equal to T, a second prompt message indicating that the operating standards are met is presented on the mobile terminal; T is the minimum time that the nasal swab stays in the nose required by the operating standards of the antigen test.

2. The method for determining nasal swab collection action during antigen detection according to claim 1, characterized in that: The training process of the first neural network in S300 includes: S310, obtain n antigen detection videos V = {V1, V2, ..., V n }, V y is the yth antigen detection video, the value of y ranges from 1 to n, n is a positive integer, V y including multiple frames of video images containing human faces; S320, V y Manually marking the categories of the corresponding face region images, wherein the categories include positive samples and negative samples; S330, using the face region image after manual marking as a training sample to train the first neural network.

3. The method for determining nasal swab collection action during antigen detection according to claim 1, characterized in that: Q0 satisfies the following conditions: Q0 = roundup(k * T1 * N / T0) + N0 Q0 < T1 * N / T0 Where, N0 is a preset number of frames, N0 ≥ 0 and N0 is an integer; k is a fault tolerance coefficient, 0 < k < 1, and roundup is a ceiling function.

4. The method for determining nasal swab collection action during antigen detection according to claim 3, characterized in that: The value range of k is: 0.8 ≤ k < 1.

5. The method for determining nasal swab collection action during antigen detection according to claim 1, characterized in that: T1 < T.

6. The method for determining nasal swab collection action during antigen detection according to claim 1, characterized in that: In S500, the first prompt information is presented on the mobile terminal in the form of red font or in the form of a red background on the mobile terminal; the second prompt information is presented on the mobile terminal in the form of green font or in the form of a green background on the mobile terminal.

7. The method for determining nasal swab collection action during antigen detection according to claim 6, characterized in that: In S500, if T2 is less than T during the antigen detection process, a first prompt sound indicating non-compliance with the operation standard is also emitted on the mobile terminal; if there is T2 greater than or equal to T during the antigen detection process, a second prompt sound indicating compliance with the operation standard is also emitted on the mobile terminal.

8. The method for determining nasal swab collection action during antigen detection according to claim 1, characterized in that: The first neural network is a convolutional neural network.

9. The method for determining nasal swab collection action during antigen detection according to claim 1, characterized in that: In the step S200, the face detection method based on the second neural network is used to determine P i Whether there is a face in the image.

10. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by a processor to implement the method according to any one of claims 1-9.

Images