Machine vision-based answer sheet recognition method, network device and storage medium

Abstract

The application provides a machine vision-based answer sheet recognition method, network equipment and a storage medium, wherein the method comprises: acquiring a scene picture containing an answer sheet; performing regional blocking on the scene picture, and the blocked regions have overlapping regions; identifying an answer sheet region of interest of each blocked region through an answer sheet detection model; processing the answer sheet region of interest of the blocked region through a non-maximum value transplantation method; and performing answer recognition on the processed answer sheet region of interest through an answer classification detection model. The answer sheet according to the embodiment of the application can be recognized at a long distance, and the recognition accuracy is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication equipment, and particularly relates to a question card recognition method based on machine vision, network equipment and a storage medium. BACKGROUND

[0002] With the continuous development of computer vision technology and Internet technology, compared with the traditional chalk classroom interactive teaching mode, the real-time interactive smart classroom mode begins to rise, which provides a new teaching interactive experience for teachers and students. However, the existing feedback device smart classroom, pad smart classroom and other smart classroom interactive modes mostly have problems such as difficult to manage, high hardware cost, poor system stability and the like.

[0003] In the smart classroom interactive mode based on machine vision, first, the teacher's questions in the classroom are displayed through a smart screen, and then the students give answers by holding up the selected question cards. During the answering process, the camera on the smart screen can capture the pictures of the question cards held up by the students in real time, and the machine vision program of the smart screen can intelligently recognize the card information held up by each student, and then the answering results of each question are presented on the smart screen, so that the teacher and the students can real-time focus on the answering situation of each question. This smart classroom interactive mode effectively reduces the hardware cost of terminal interaction, and the teacher can give real-time explanation according to the students' answering situation.

[0004] However, according to the inventors' knowledge, the existing smart classroom has problems of long card distance and high card misrecognition. SUMMARY

[0005] In order to overcome the problems in the related art, the present application provides a question card recognition method based on machine vision, network equipment and a storage medium.

[0006] According to an embodiment of the present application, a question card recognition method based on machine vision is provided, which comprises the following steps.

[0007] Obtaining a scene picture containing a question card;

[0008] Dividing the scene picture into regions, and the divided regions have overlapping regions;

[0009] Identifying the question card region of interest of each divided region through a question card detection model;

[0010] Processing the question card region of interest of the divided region through a non-maximum value transplantation method;

[0011] Identifying the answer of the processed question card region of interest through a question classification detection model.

[0012] Preferably, each sub-block region of the answer sheet region of interest is linearly enlarged before being identified by the answer sheet detection model.

[0013] Preferably, the sub-block region of the answer sheet region of interest is processed by a non-maximum suppression method, comprising:

[0014] Two sets A and B are set, the set A is initially used to place the sub-block region of the answer sheet region of interest, and the set B is an empty set;

[0015] The sub-block region is identified by the answer sheet detection model to identify each sub-block region of the answer sheet region of interest, and the corresponding confidence value;

[0016] The sub-block region of the answer sheet region of interest with the highest confidence value is selected and placed in the set B;

[0017] The selected sub-block region of the answer sheet region of interest with the highest confidence value is intersected with the remaining sub-block region of the answer sheet region of interest in the set A, and if the intersection region is greater than a set threshold, the sub-block region of the answer sheet region of interest in the intersection set A is deleted.

[0018] Further, the new sub-block region of the answer sheet region of interest with the highest confidence value is selected from the set A and placed in the set B, and is intersected with the remaining sub-block region of the answer sheet region of interest in the set A, and if the intersection region is greater than a set threshold, the sub-block region of the answer sheet region of interest in the intersection set A is deleted, and the iteration selection is terminated until the set A is empty.

[0019] Further, a human body detection model is used to identify the scene picture to obtain a human body region of interest, a face detection model is used to identify the human body region of interest to obtain a face region of interest, and the human body region of interest is intersected with the answer sheet region to make association.

[0020] Further, the human body region of interest is intersected with the answer sheet region of interest to make association, comprising:

[0021] A threshold of the area intersection ratio of the human body region of interest and the answer sheet region of interest is set in advance;

[0022] The coordinates of the answer sheet in the human body region of interest and the coordinates of the answer sheet region of interest are obtained;

[0023] The human body region of interest and the answer sheet region of interest whose difference between the coordinates of the answer sheet in the human body region of interest and the coordinates of the answer sheet region of interest is within a set acceptance range are selected;

[0024] Intersect the screened answer sheet region of interest with the corresponding answer sheet region in the human region of interest;

[0025] Associate the human region of interest with the answer sheet region of interest with the largest intersection ratio exceeding the area intersection threshold.

[0026] Preferably, the answerer information is associated with the answer sheet, including:

[0027] The answer sheet is provided with a framed region, and the framed region of interest is identified by a graphic detection model;

[0028] The answerer information in the framed region of interest is identified by an answerer information identification model and is associated.

[0029] Preferably, the processed answer sheet region of interest is subjected to color interference screening by an answer classification detection model before the processed answer sheet region of interest is subjected to answer identification.

[0030] The negative samples are trained by the answer classification detection model;

[0031] The pixels of the answer sheet region of interest are extracted as a number of center clusters by clustering, and the color of the answer sheet is determined by calculating the distance relationship between the center cluster and the lab color space.

[0032] If the color of the answer sheet is inconsistent with the set color of the answer sheet, it is determined that the answer sheet region of interest is not an answer sheet; if it is consistent, it is determined to be an answer sheet.

[0033] The embodiments of the present application also provide a network device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to execute the above-mentioned answer sheet identification method based on machine vision.

[0034] The embodiments of the present application finally provide a storage medium having computer program instructions stored thereon, and the program instructions are executed by a processor to implement the above-mentioned answer sheet identification method based on machine vision.

[0035] The technical scheme provided by the embodiments of the present application can include the following beneficial effects:

[0036] In the embodiments of the present application, the answer sheet can be identified at a long distance, and the identification accuracy is high.

[0037] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments that are consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0039] Figure 1 is a flowchart of an embodiment of the present application;

[0040] Figure 2 is an example of a scene picture area blocking diagram of the present application;

[0041] Figure 3 is a flowchart of an embodiment of the present application;

[0042] Figure 4 is a flowchart of a non-maximum value transplantation method processing method of the present application;

[0043] Figure 5 is a diagram of a person form region of interest and answer card association method of the present application. DETAILED DESCRIPTION

[0044] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is presented with reference to the drawings, wherein like reference numerals represent similar or analogous elements throughout the several views. The following description of exemplary embodiments is not representative of all embodiments consistent with the present application. Rather, it is merely an example of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0045] The embodiments of the present application are to solve the problems in the background art, and propose a machine vision-based answer card recognition method, as shown in Figure 1 The embodiment one includes:

[0046] 101: Obtain a scene picture containing an answer card;

[0047] The scene picture is captured by a camera in the scene, and the scene picture requires the answerer to hold up the answer card. The camera here can be independently set up, or integrated on other devices. The scene picture here can be a picture of a teacher-student interactive answer in a smart classroom.

[0048] 103: Regionally block the scene picture, and the blocked regions have overlapping regions;

[0049] Because the target is small and different from detection at a long distance in the scene, the scene picture is divided into multiple regions, and the blocked regions have overlapping regions. As Figure 2As shown, the card is divided into two blocks, with the bottom edge as the reference, one from left to a position 25% away from the right edge (i.e. 75% of the total bottom edge from left to right), and the other from right to a position 25% away from the left edge. Therefore, the overlapping area is from a position 25% of the total bottom edge from left to right to a position 75% of the total bottom edge from left to right.

[0050] 105: Identify the answer card region of interest in each sub-block region through the answer card detection model;

[0051] Since the proportion of the answer card in the scene picture is very small, a deep learning model specifically for small target detection can be selected as the answer card detection model to identify the sub-block region of the scene picture. For example, the yolo series model. Because the identified sub-block region picture is smaller than the original scene picture, the sample and detection picture input into the answer card detection model can be enlarged by linear scaling. In this way, the answer card information in the picture will be correspondingly obtained, which is conducive to the identification of the answer card. Especially for cards that are not easy to identify at a distance, they are more likely to be identified. In addition, since the test taker holds the answer card at multiple angles, the classroom lighting varies day and night, and the student card may be blocked, etc. Image data enhancement methods such as picture rotation, image brightness and contrast change, random partial blocking, mosaic enhancement, etc. are needed to increase the data samples for training the answer card detection model.

[0052] 107: Process the answer card region of interest in the sub-block region through the non-maximum value transplantation method;

[0053] The non-maximum value transplantation method is used to process the answer card region of interest in the sub-block region, as shown in Figure 4 , which includes:

[0054] 1071: Set two sets A and B, the initial set A is used to place the answer card region of interest in the sub-block region, and the set B is an empty set;

[0055] 1073: The sub-block region identifies the answer card region of interest in each sub-block region through the answer card detection model, and the corresponding confidence value;

[0056] The confidence value here can be obtained in the process of identifying the answer card region of interest in each sub-block region through the answer card detection model in step 105. As long as the confidence of each sub-block region of the answer card region of interest is cached accordingly, it can be obtained.

[0057] 1075: Select the sub-block region of the answer card region of interest with the highest confidence value and place it in set B;

[0058] 1077: Intersect the selected answer sheet region of interest area of the highest confidence value of the slice area with the remaining slice area of the answer sheet region of interest area in set A, and if the intersection area is greater than a set threshold, the answer sheet region of interest area of the slice area in set A is deleted;

[0059] 1079: Iterate steps 1075-1077 until set A is empty.

[0060] The actual operation does not necessarily follow the steps, for example: the threshold of the intersection area is not necessarily after step 1071 and step 1073, and step 1071 and step 1073 also have no front and back, which can be step 1071 in front of step 1073, also can be at the same time, also can be step 1073 in front of step 1071. Here is only an example, other steps have the same situation, as long as the same purpose and effect are achieved, they are within the scope of protection of the present application, and will not be described here.

[0061] 201: Recognize the answer of the processed answer sheet region of interest through the answer classification detection model.

[0062] For example, the answer sheet contains multiple-choice options A, B, C, D, and judgment √, X, etc. The answer options are classified. Considering the target multiple nature and real-time nature of the answer sheet that needs to be recognized, a lightweight deep learning model needs to be selected, such as mobilenet, shufflenet, efficientnet, mobileone, etc. The lightweight deep learning model needs to meet the following two conditions: (1) The deep learning model itself occupies a small space, usually within a few megabytes; (2) The deep learning model training sample is small, usually below ten thousand.

[0063] In order to eliminate the interference of different color objects in the scene on the detection of the answer sheet, before step 109 recognition, the processed answer sheet region of interest can be screened for color interference, which specifically includes:

[0064] (1) Training the negative sample through the answer classification detection model;

[0065] (2) Extracting a number of center clusters from the pixels of the answer sheet region of interest through clustering, and determining the color of the answer sheet by calculating the distance relationship between the center cluster and the lab color space;

[0066] (3) If the color of the answer sheet is inconsistent with the set answer sheet color, it is determined that the answer sheet region of interest is not an answer sheet; if it is consistent, it is determined to be an answer sheet.

[0067] Here step (1) and step (2) have no precedence. It should be noted that here a central cluster represents the mean point of all pixel points in each cluster, such as the card target area has red and purple, which will have a red central cluster and a purple central cluster. The lab color space is a color-opposite space with dimension l representing brightness, a and b representing color-opposite dimensions. Calculate the average of the Euclidean distance of all pixel points from each central cluster to obtain the nearest central cluster as the color category label of the card target.

[0068] The application embodiment answer sheet can be recognized at a long distance, and the recognition accuracy is high.

[0069] Embodiment two: as Figure 3 On the basis of the above embodiment, step 109 can be added: a human body detection recognition model is used to recognize the scene picture to obtain a human body region of interest, a human face detection model is used to recognize the human body region of interest to obtain a human face region of interest, and the human body region of interest and the answer sheet region are intersected to make association.

[0070] The human body detection recognition model is used to recognize the scene picture to obtain a human body region of interest, the human face detection model is used to recognize the human body region of interest to obtain a human face region of interest, and the human body region of interest and the answer sheet are associated, as shown in Figure 5 , specifically comprising:

[0071] 1091: A proportion threshold of area intersection of the human body region of interest and the answer sheet region of interest is set in advance;

[0072] 1093: The coordinates of the answer sheet in the human body region of interest and the coordinates of the answer sheet region of interest are obtained;

[0073] 1095: The human body region of interest and the answer sheet region of interest whose coordinate difference between the answer sheet in the human body region of interest and the answer sheet region of interest is within a set acceptance range are screened out;

[0074] The coordinate difference within the set acceptance range represents a detection error, and is considered to belong to the same position.

[0075] 1097: The screened answer sheet region of interest is intersected with the corresponding answer sheet region in the human body region;

[0076] 1099: The human body region of interest and the answer sheet region of interest are associated in the proportion of area intersection exceeding the area intersection threshold and the maximum intersection proportion.

[0077] Through the above operation, the association of human body-human face-answer sheet can be realized, so that the association of the answer person and the answer sheet is realized.

[0078] In the embodiment of the present application, the method for associating the answerer with the answer card can also include:

[0079] The answer card is provided with a framed area, and the framed area of interest is identified by a graphic detection model;

[0080] The answerer information in the framed area of interest is identified by an answerer information identification model and is associated.

[0081] Here, since the framed area on the card is small, the graphic detection model used can be a deep learning model for small target detection, such as the yolo series. The frame here can be a circular frame, a rectangular frame, a triangular frame, etc., which can be selected according to user preference.

[0082] The content in the framed area of interest can be numbers, symbols, or two-dimensional codes, etc., according to the need of pre-setting. In this way, the answerer information identification model can be any model that can correspondingly identify the information in the frame. A lightweight classification and identification model can be generally used, such as mobilenet, shufflenet, efficientnet, etc.

[0083] 203: Statistics of the scores of each answerer.

[0084] The embodiment of the present application also provides a network device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to perform the above-mentioned answer card identification method based on machine vision. The network device here can be a smart screen, and the camera used to collect the scene picture can be a camera integrated on the smart screen, or a camera separate from the smart screen.

[0085] Finally, the embodiment of the present application also provides a storage medium having computer program instructions stored thereon, characterized in that the program instructions are executed by a processor to implement the above-mentioned answer card identification method based on machine vision.

[0086] It should be understood that the present application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0087] The machine-readable storage medium described above can be any electronic, magnetic, optical, or other physical storage device that contains or stores information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be a Random Access Memory (RAM), an electrically programmable memory, a flash memory, a storage drive, a solid-state drive, any type of storage disk (e.g., a hard disk drive, a floppy disk, a magnetic disk, an optical disk, a DVD, etc.), or the like. The machine-readable storage medium can also be a combination of these or other storage devices.

[0088] The systems, apparatuses, modules, or units illustrated by the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an e-mail device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0089] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, a disk memory, a CD-ROM, an optical memory, etc.) containing computer-usable program code.

[0090] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and a combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0091] Moreover, these computer program instructions can also be stored in a computer-readable memory capable of directing the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocksFigure 1 the function specified in the one or more blocks.

[0092] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide processes for implementing the flow Figure 1 the flow or flows and / or blocks Figure 1 the function specified in the one or more blocks.

[0093] The above merely provides the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A machine vision-based answer card recognition method, characterized in that, include: Get the scene image containing the answer cards; The scene image is divided into regions, and the divided regions have overlapping areas; The answer card detection model identifies the regions of interest for each segmented area of ​​the answer card; The human figure detection and recognition model is used to identify the human figure region of interest in the scene image, and the face detection model is used to identify the human figure region of interest to obtain the face region of interest. The intersection of the human figure region of interest and the answer card area is used to make a correlation. The regions of interest in the answer cards of the segmented regions are processed using a non-maximum transplantation method; The answer identification is performed on the region of interest of the processed answer card using a question classification and detection model. The regions of interest on the answer cards in the segmented regions are processed using a non-maximum transplantation method, including: Set up two sets A and B. Initially, set A is used to place the answer cards of interest in the segmented regions, and set B is an empty set. The block region identifies the region of interest of the answer card in each block region and its corresponding confidence value through the answer card detection model; Select the answer card regions of interest from the segment with the highest confidence value and place them in set B; Take the intersection of the region of interest of the answer card of the selected segment with the highest confidence value and the region of interest of the answer card of the remaining segment in set A. If the intersection area is greater than a set threshold, delete the region of interest of the answer card of the segment in set A that has the intersection. The association is established by taking the intersection of the region of interest of the human figure and the region of interest of the answer card, including: A threshold is pre-set for the area overlap ratio between the human figure's region of interest and the answer card's region of interest. Obtain the coordinates of the answer card within the region of interest of the humanoid figure, and the coordinates of the region of interest of the answer card. Filter out regions of interest for both the human figure and the answer card, where the difference between the coordinates of the answer card and the region of interest for the answer card is within a set acceptable range. Intersect the selected answer card interest area with the corresponding answer card area in the human figure area; Associate the regions of interest of the human figure with the regions of interest of the answer sheet whose intersection ratio exceeds the area intersection threshold and has the largest intersection ratio; Before identifying answers using the answer classification and detection model on the processed answer card's region of interest, color interference screening is performed on the processed answer card's region of interest, including: Negative samples are trained using a question-and-answer classification detection model; The pixels in the region of interest of the answer sheet are clustered to extract several central clusters. The color of the answer sheet is determined by calculating the distance between the central clusters and the lab color space. If the color of the answer card is different from the set answer card color, then the area of ​​interest of the answer card is determined to be not an answer card; if they are the same, then it is determined to be an answer card.

2. The machine vision-based answer card recognition method according to claim 1, characterized in that, Before identifying the region of interest (ROI) of the answer card in each block region using the answer card detection model, the ROI of the answer card in each block region is linearly magnified.

3. The machine vision-based answer card recognition method according to claim 1, characterized in that, The iteration selects the answer card regions of interest with the highest confidence from set A and places them in set B. The intersection of these intersection regions with the answer card regions of interest of the remaining blocks in set A is then taken. If the intersection region is larger than a set threshold, the answer card regions of interest of the blocks in set A that are the intersection regions are deleted. The iteration ends when set A is an empty set.

4. The machine vision-based answer card recognition method according to claim 3, characterized in that, Linking respondent information with answer sheets includes: The answer card has a framed area, and the framed area of ​​interest is identified by an image detection model. The system identifies and associates the corresponding respondent information within the bounded area of ​​interest using a respondent information recognition model.

5. A network device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it performs the machine vision-based answer card recognition method as described in any one of claims 1-4.

6. A storage medium storing computer program instructions thereon, characterized in that, When the program instructions are executed by the processor, they are used to implement the machine vision-based answer card recognition method according to any one of claims 1-4.

Images