Data annotation system and method based on human-computer collaborative interaction

Abstract

This solution belongs to the field of human-computer interaction, and specifically involves a data annotation system based on human-computer collaborative interaction, including: an information collection end, an information processing end, and an information output end; the information collection end is used to collect image information to be annotated, facial information recorded by the camera, typing information recorded by the keyboard, and cursor information recorded by the mouse; the information processing end is used to receive facial information when the information entry person logs in to the data annotation system based on human-computer collaborative interaction, and compare the received facial information with the pre-stored login standard to determine whether it is a real login by the person himself; after successful login, the cursor information is received, the position information in the cursor information is processed to obtain the corresponding image to be annotated, and then the image to be annotated is loaded as an annotated image, that is, a number of recommended starting points on the image to be annotated are obtained by extracting boundary features, and a number of recommended starting points form a recommended starting point set. This solution can solve the problem of low manual annotation rate while ensuring low-difficulty operation of the system.

Description

Technical Field

[0001] This solution belongs to the field of human-computer interaction, and specifically relates to a data annotation system and method based on human-computer collaborative interaction. Background Art

[0002] With the widespread application of artificial intelligence and machine learning, more and more companies and organizations need a large amount of high-quality data to train and improve their models and algorithms. Data annotation is the process of processing unprocessed primary data including voice, pictures, text, video, etc. and converting them into machine-recognizable information.

[0003] Traditional data annotation is completely done manually. In the specific operation process of existing manual data annotation tools, the annotator needs to use the human eye to identify the objects to be annotated in the image to be annotated, select the annotation box after the selection is completed, and then add the preset label / attribute to the object in the annotation box as the result. The selection process usually requires the annotator to determine the starting point of the annotation box by human eyes, and then move the cursor to the starting point through the mouse / keyboard and confirm the starting point; then move the cursor to the end point by operating the mouse / keyboard to complete the construction of the annotation box; finally, add the preset label / attribute to the object in the annotation box by operating the mouse / keyboard as the result, thus completing the standard operation of an annotation box.

[0004] On the basis of the above technologies, the current data annotation has added the assistance of annotation tools, which greatly improves the annotation efficiency. Different annotation tools have different assistance methods. Some annotation tools will first automatically input the image to be annotated to the auxiliary annotation model formed based on training to obtain the preliminary annotation results completed by the machine, and then manually review and annotate it; some annotation tools will first classify the image to be annotated according to the objects to be annotated in the image to be annotated, establish unique recognition rules according to the objects to be annotated, and then process and output the results according to the corresponding recognition rules according to the classification of the image to be annotated, and finally hand over the recognition rule processing results to manual review and annotation; some annotation tools simplify the annotation operation by formulating special annotation actions, thereby speeding up the annotation speed, such as first inputting a specific track or pattern formed by sliding the cursor (equivalent to simplifying the two key operations of determining the starting point and determining the end point into one long press operation), and then the machine directly processes it into the annotation range through the track or pattern, but in essence, there is no obvious difference from traditional data annotation (that is, in essence, it still requires moving to the starting point, determining the starting point, moving to the end point, determining the end point, and adding annotation types to the annotation box).

[0005] However, in the above operation process, although the auxiliary labeling model can help humans complete part of the labeling work, the accuracy of its labeling results is not high. The reason is that manual labeling itself is to obtain a labeling model with a high recognition success rate, so the labeling results obtained in this way require more time to modify and correct when manually reviewing the labeling. This method has not solved the problem of low manual data labeling rate; although the recognition rule processing has a higher accuracy rate than the preliminary labeling results obtained by the auxiliary labeling model, the preliminary labeling results obtained by processing are very few (the actual image is much more complicated than the pre-set recognition rules), and cannot completely label all objects to be labeled. More complete labeling work is still required when manually reviewing the labeling. This method has not been well solved the problem of low manual data labeling rate.

[0006] Although the method of formulating special annotation actions to simplify the annotation operation can theoretically reduce the time required for annotation (optimizing the two typing operations into one long press operation), more precise mouse operations are required to click to determine the positions of the starting and end points when completing the formulated special annotation actions (if the long press and mouse movement are performed with one hand, it will increase the difficulty of moving the mouse). If the click or drag is inaccurate, it is necessary to re-mark. This method actually increases the difficulty of manual data annotation operations. Summary of the invention

[0007] The purpose of this solution is to provide a data annotation system based on human-computer collaborative interaction to solve the problem of low manual annotation rate while ensuring low-difficulty operation of the system.

[0008] The basic solution provided by the present invention is a data annotation system based on human-computer collaborative interaction, including: an information collection end, an information processing end and an information output end;

[0009] The information collection terminal is used to collect the image information to be annotated, the facial information recorded by the camera, the typing information recorded by the keyboard, and the cursor information recorded by the mouse;

[0010] The information processing end is used to receive facial information when the information inputter logs in to the data annotation system based on human-computer collaborative interaction, compare the received facial information with the pre-stored login standard, and determine whether it is the real login of the person; after successful login, the cursor information is received, the position information in the cursor information is processed to obtain the corresponding image to be annotated, and then the image to be annotated is loaded as an annotated image, that is, a number of recommended starting points on the image to be annotated are obtained by boundary feature extraction, and the recommended starting points form a recommended starting point set, and the image to be annotated and the recommended starting point set are integrated into image loading information, and then the image loading information is transmitted to the information output end and displayed; then the facial information is received, and the changes in the gaze direction and pupil size of the eyes are obtained according to the facial information analysis, and then the starting point range is determined according to the changes in the gaze direction and pupil size of the eyes, all the recommended starting points within the starting point range are formed into a recommended set, and the point closest to the geometric center of the recommended set is used as the selected starting point, and then the recommended set and the selected starting point are sent to the information output end, and after receiving the recommended set, the information output end highlights the recommended set and flashes the selected starting point;

[0011] The information processing end is further used to receive input information when highlighting the recommended set, wherein the input information includes determining the key combination information and the direction information, wherein the determining the key combination information is defined as the first information and the direction information is defined as the second information; if the first information is received, the selected starting point is transmitted to the information output end as the fixed starting point, and the information output end turns off the recommended set and only highlights the fixed starting point after receiving the fixed starting point; if the second information is received, the selected starting point is adjusted to the recommended starting point in the corresponding direction according to the second information, and the recommended starting point is used as the pending starting point, and the pending starting point is sent to the information output end, and the information output end turns off the selected starting point in the highlighted recommended set and flashes the pending starting point after receiving the pending starting point, and when the input information is not received for a preset time period, or when the current eye gaze direction change and pupil size change overlap with the initial starting point range less than a preset value, the pending starting point is used as the fixed starting point;

[0012] The information processing end is also used to determine the end point range according to the current eye gaze direction change and pupil size change when the fixed starting point is highlighted, and then write the geometric center point of the end point range as the selected end point into the end point range, and transmit the end point range to the information output end, and the information output end will display the received end point range on the corresponding area; receive cursor information, the cursor information includes click information and position information, the click information is defined as the third information, and the position information is defined as the fourth information; when the third information received is a single click, the fourth information is used as the fixed end point, and the line segment formed by connecting the fixed starting point and the fixed end point is used as a diagonal line to form a marked range, and the marked range is transmitted to the information output end, and the information output end displays the marked range as a solid color line frame after receiving it, and pops up a preset mark type list box;

[0013] The information processing end is also used to receive typed information or cursor information when a preset annotation type list box pops up, process the typed information or cursor information into an annotation type, record the annotation type and annotation range as a complete annotation, and pass the annotation type to the information output end. After the information output end receives the annotation type, it displays the annotation type on the annotation range; after the information processing end records the complete annotation, when the received cursor information is to click the confirmation completion button displayed on the data output end, all complete annotations are processed into an annotation set, the image to be annotated is set as an annotated image, and the annotation set is associated with the annotated image, and finally a completed annotation signal is sent to the information output end. After the information output end receives the completed annotation signal, the currently displayed annotated image is closed.

[0014] The working principle of the present invention is:

[0015] The data annotation system based on human-computer collaborative interaction decomposes the manual data annotation action into four stages: data loading, data identification, data annotation, and adding annotation types; the data annotation process is decomposed into four steps: moving to the starting point, confirming the starting point, moving to the end point, and confirming the end point. This solution mainly solves the problem of low rate of manual data annotation from the data annotation aspect, as follows:

[0016] (1) Data loading continues to establish the existing recognition rule method to obtain the recommended starting point set for preprocessing. The implementation principle is: classify the image to be labeled according to the objects that need to be labeled in the image to be labeled, and then establish the recognition rules according to the objects that need to be labeled. In the operation of moving to the starting point step of data annotation, different from the traditional method of using a mouse as an annotation tool, this solution uses a method of adding a camera to collect facial information. When the information inputter moves to the starting point step in the data annotation stage, no matter what tool is used, the cooperation of the eyes is indispensable. This solution adds a high-precision camera to the information input end to collect the facial information of the information inputter, and judges the changes in the screen position and gaze range of the information inputter through the changes in the eye gaze direction and pupil size in the facial information. At the same time, the screen size, eye and screen distance are combined to further accurately determine the gaze range. This method can reduce the steps of moving to the starting point in the data annotation stage. At the same time, the speed of the eyes is faster than the hand, and the speed of moving to the starting point is also faster.

[0017] The reason why many tools do not adopt the gaze range obtained by the above method is that, on the one hand, the cost of high-precision cameras is higher than that of the mouse currently used; on the other hand, the gaze range obtained is not an accurate point, but a small area, which cannot be used directly as the starting point of annotation. The reason why few tools adopt the rule recognition method is that, on the one hand, this method obtains few results and can only complete the annotation of some objects that need to be annotated; on the other hand, this method still has a small number of erroneous results, which need to be manually modified and corrected.

[0018] This solution combines the above two methods. After determining the gaze range, the rule recognition method is used to supplement the problem of precise positioning. The camera provides a small area that the information inputter is looking at, and the rule recognition method provides a recommended starting point within the small range. The two steps of the information inputter looking at the screen and operating the mouse to move to the starting point and clicking the starting point are simplified to the information inputter looking at the screen and using the keyboard to select a starting point in the small range. This method of determining the starting point is faster, easier, and simpler than the traditional method of using a mouse, which increases the speed of data annotation.

[0019] (2) In the data annotation stage, the steps of moving to the end point and confirming the end point are different from the traditional method of using the mouse as an annotation tool. Since this solution does not completely rely on the mouse in the step of determining the starting point, the mouse can be moved to the end point during the process of confirming the starting point. After confirming the starting point with the keyboard, the mouse can be directly clicked to determine the end point position to form the annotation range. This method directly eliminates the step of operating the mouse to move to the end point, further improving the data annotation rate.

[0020] (3) During the step of confirming the starting point in the data annotation stage, if the person who entered the information does not operate the system or directly confirms the end point position after the preset time, the system will automatically complete the step of confirming the starting point and confirm the pending starting point before confirming the end point as the fixed starting point. This method directly eliminates the operation of confirming the starting point during the data annotation process, further improving the data annotation rate; at the same time, even if the person who entered the information missed the step of confirming the starting point after selecting the starting point and directly confirmed the end point, this autonomous processing method also supplements the completion of the operation and improves the experience of human-computer collaborative interaction.

[0021] In summary: This solution increases the data annotation rate and improves the experience of human-computer collaborative interaction by adding camera images to collect facial information and combining it with preprocessing to obtain the system recommended starting point.

[0022] Furthermore, the information processing end is used to receive facial messages and compare them with pre-stored login standards to determine whether it is the real person when the information inputter logs in to the data annotation system based on human-computer collaborative interaction; the data annotation system based on human-computer collaborative interaction is preset with a login oral text set, and the login oral text set consists of several oral texts and corresponding standard lip shapes. After confirming that it is the real person through facial message comparison, an oral text and the corresponding standard lip shape are randomly extracted from the login oral text set, and the text content is passed to the information output end for display, and then the mouth periorbital information in the collected facial information is compared with the standard lip shape to determine whether it is a real person logging in; if it is determined that it is not the real person, a login failure message is displayed.

[0023] This real person identification login method saves the need to enter account and password when logging into the data annotation system based on human-computer collaborative interaction. This makes it convenient for employees to log in quickly and avoids the trouble caused by the information inputter forgetting the password. The login time can also be used as the clock-in time, which improves the experience of human-computer collaborative interaction.

[0024] Furthermore, the information processing end is used to receive cursor information instead of typing information when highlighting the recommended set, and the cursor information includes click information and position information, the click information is defined as the fifth information, and the position information is defined as the sixth information. If the fifth information received is a double-click, the sixth information received at this time is directly used as a fixed starting point, and the fixed starting point is sent to the information output end. After receiving the fixed starting point, the information output end extinguishes the recommended set and only highlights the fixed starting point.

[0025] The purpose of this is to use the above method to supplement when the recommended starting point given by the system is not enough to cover all the objects that need to be labeled on the image to be labeled. Although this conventional annotation method does not increase the data annotation rate, it is only used for supplementary annotation and the usage rate is not high, so overall it has little impact on solving the problem of low data annotation rate.

[0026] Furthermore, the preset annotation type list includes: a new type text box, several default annotation type options and a new annotation type option for the current image to be annotated; the new type text box is used to receive a new description tag input by the user; the default annotation type option is the default annotation type in the default annotation type list. If more than 80% of the images in the same category of labeled images have the same annotation type, this annotation type is set as the default annotation type. The default annotation type list consists of several default annotation types in the same category of annotated images; the new annotation type option is: before the data annotation of all objects in the image to be annotated is completed, the confirmed annotation type is entered from the new type text box.

[0027] Furthermore, the information processing end is used to receive typed information or cursor information when the annotation type list box pops up, and the specific process of processing the typed information or cursor information into an annotation type is as follows: the received typed information or cursor information includes text information of the typed information, confirmation information of the typed information, click information of the cursor information, and position information of the cursor information. The confirmation information of the typed information is defined as the 7th information, the typed text information is defined as the 8th information, the click information of the cursor information is defined as the 9th information, and the position information of the cursor information is defined as the 10th information; if the 7th information is received after the 8th information is received, the 8th information is processed as a newly added annotation type, and the newly added annotation type is added to the preset annotation type list, the annotation type and the annotation range are recorded as a complete annotation, and the annotation type is passed to the information output end. After the information output end receives the annotation type, it displays the annotation type on the annotation range; if the received information is the 9th information, the annotation type is determined by the relative position relationship between the 10th information and the annotation type list, and the annotation type is passed to the information output end for display on the annotation range.

[0028] When setting the default annotation type, only the annotation types with a probability of 80% or more in the same type of labeled images are used. This is to avoid the situation where uncommon annotation types are also set as the default annotation type, to ensure the simplicity of the default annotation type list as much as possible, so that the information inputter can spend less time and more conveniently when selecting the annotation type. The addition of the annotation type is added to the preset annotation type list so that when annotating the same image to be annotated, a new annotation type does not need to be repeatedly entered, and can be used directly later, which increases the experience of human-computer collaborative interaction.

[0029] Further, the information processing end is used to receive facial information and process the facial information into a working state. When the working state is determined to be a fatigue state, the fatigue state is recorded in the work fatigue log, and the fatigue signal is transmitted to the information output end. After the information output end receives the fatigue signal, a health reminder box pops up, and a three-second countdown is displayed in the prompt box; after the countdown ends, the facial information is received to determine whether the facial information meets the health indication action requirements. If it meets the requirements, an instruction to close the health reminder box is transmitted to the information output end. After the information output end receives the instruction to close the health reminder box, the health reminder box is closed; if the frequency of increase of fatigue state data bars in the work fatigue log exceeds a preset warning line, the working state is determined to be a tired state, all data is immediately saved, the countdown to the preset rest time is started, and a forced rest command and the preset rest time are transmitted to the information output end. After receiving the forced rest command, the information output end locks the screen, pops up a forced rest health reminder, and displays a rest reminder box of the preset rest time countdown process, and issues a forced rest voice prompt. After the countdown ends, the rest reminder box is closed and the screen lock is released;

[0030] Furthermore, the working status includes the eye blinking frequency in the facial information, the huffing frequency in the facial information, the nodding arc in the facial information and the gaze range obtained in the facial information. When the increase in the eye blinking frequency and the huffing frequency exceeds the set fatigue limit, the working status is judged as a fatigue state; when the gaze range or the nodding arc exceeds the fatigue limit, the working status is judged as a tired state, and it is directly assumed that the increase in the fatigue state frequency exceeds the preset warning line.

[0031] The recorded facial information is convenient for monitoring the physical condition of the information inputter, so as to prevent or prepare for unexpected situations. The eye blinking frequency and facial coughing frequency recorded in the facial information recorded by the camera reflect the fatigue level of the information inputter during work; the recorded facial nodding arc and the degree of pupil change of the eye reflect the degree of fatigue of the information inputter during work. The combination of the four information can determine whether the physical condition of the information inputter can continue to work with high quality, and even determine whether the information inputter can continue to work in a healthy manner. If it is just fatigue work, only appropriate rest time is needed, so that the information inputter can continue to work after a period of rest, which is conducive to ensuring the quality of work and providing a humane and good working environment for the information inputter; if there is fatigue work, especially when the pupil is dilated, the information inputter is at great risk of sudden death. At this time, the people around and the superior leaders need to pay attention in time, and the information inputter is forced to rest, which can avoid accidents to a certain extent. At the same time, when accidents occur, timely response and emergency measures can be taken, increasing the possibility of saving lives.

[0032] Furthermore, when the information processing end is used to pop up a health reminder box, the nose tip in the received facial information is set as a movement point, and the horizontal continuous turning line formed by the rapid left and right swing of the nose tip is set as an end mark, and the movement trajectory drawn by the movement point is used to judge whether the head movement of the person entering the information meets the health indication action requirements; when the entered facial information does not meet the health indication action requirements, a re-movement instruction is sent to the information output end, and after the information output end receives the re-movement instruction, a prompt box for moving as required is popped up, and a three-second countdown is displayed in the prompt box for moving as required. After the countdown ends, the facial information is entered again for judgment; until the entered facial information meets the health indication action requirements, an instruction to close the health reminder box is transmitted to the information output end, and the information output end closes the health reminder box after receiving the instruction to close the health reminder box;

[0033] By taking head movements that meet the requirements of health instructions to close the health reminder box, the fatigue of the information inputter can be eliminated by shoulder and neck exercises. Adjusting the state through exercise can make the mind clearer and work more energetic. At the same time, shoulder and neck exercises can protect the health of the shoulders and neck and reduce the negative impact of desk work on the body. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Label the swim lane flow chart for this solution data;

[0035] Figure 2 A swimlane flow chart for monitoring the fatigue status of information inputters for this program;

[0036] Figure 3 A swimlane flow chart for monitoring the fatigue status of information inputters for this program, which increases the frequency beyond the preset warning line. DETAILED DESCRIPTION

[0037] Example 1

[0038] Basically as attached Figure 1 As shown, the data labeling system based on human-computer collaborative interaction includes: an information collection end, an information processing end, and an information output end (essentially a display screen).

[0039] The information collection terminal is used to collect the image information to be annotated, the facial information recorded by the camera, the typing information recorded by the keyboard, and the cursor information recorded by the mouse. In the specific embodiment, the information collection terminal is essentially a peripheral device of a PC. The information collection of the image to be annotated is essentially the image information to be annotated received through the network card, the facial information is collected through the camera, and the typing information and the cursor information are respectively as mentioned above, that is, the information collected by the keyboard and the mouse.

[0040] The information processing end (essentially a personal PC) is used to receive facial information when the information inputter logs in to the data annotation system based on human-computer collaborative interaction, compare the received facial information with the pre-stored login standard, and determine whether it is the real login of the person; after successful login, the cursor information is received, the position information in the cursor information is processed to obtain the corresponding image to be annotated, and then the image to be annotated is loaded as an annotated image, that is, a number of recommended starting points on the image to be annotated are obtained by boundary feature extraction, and a number of recommended starting points form a recommended starting point set, and the image to be annotated and the recommended starting point set are integrated into image loading information, and then the image loading information is transmitted to the information output end and displayed; then the facial information is received, and the changes in the gaze direction and pupil size of the eyes are obtained according to the facial information analysis, and then the starting point range is determined according to the changes in the gaze direction and pupil size of the eyes, and all recommended starting points within the starting point range are formed into a recommended set, and the point closest to the geometric center of the recommended set is used as the selected starting point, and then the recommended set and the selected starting point are sent to the information output end. After receiving the recommended set, the information output end highlights the recommended set and flashes the selected starting point;

[0041] The information processing end is further used to receive input information when highlighting the recommended set, wherein the input information includes determining the key combination information and the direction information, wherein the determining the key combination information is defined as the first information and the direction information is defined as the second information; if the first information is received, the selected starting point is transmitted to the information output end as the fixed starting point, and the information output end turns off the recommended set and only highlights the fixed starting point after receiving the fixed starting point; if the second information is received, the selected starting point is adjusted to the recommended starting point in the corresponding direction according to the second information, and the recommended starting point is used as the pending starting point and sent to the information output end, and the information output end turns off the selected starting point in the highlighted recommended set and flashes the pending starting point after receiving the pending starting point, and after the input information is not received for a preset time or the current eye gaze direction change and pupil size change have a degree of overlap with the initial starting point range that is lower than a preset value, the pending starting point is used as the fixed starting point;

[0042] The information processing end is also used to determine the end point range according to the current eye gaze direction change and pupil size change when the fixed starting point is highlighted, and then write the geometric center point of the end point range as the selected end point into the end point range, and transmit the end point range to the information output end, and the information output end will display the received end point range on the corresponding area; receive cursor information, the cursor information includes click information and position information, the click information is defined as the third information, and the position information is defined as the fourth information; when the third information received is a single click, the fourth information is used as the fixed end point, and the line segment formed by connecting the fixed starting point and the fixed end point is used as a diagonal line to form a marked range, and the marked range is transmitted to the information output end, and the information output end displays the marked range as a solid color line frame after receiving it, and pops up a preset mark type list box;

[0043] The information processing end is also used to receive typed information or cursor information when a preset annotation type list box pops up, process the typed information or cursor information into an annotation type, record the annotation type and annotation range as a complete annotation, and pass the annotation type to the information output end. After the information output end receives the annotation type, it displays the annotation type on the annotation range; after the information processing end records the complete annotation, when the received cursor information is to click the confirmation completion button displayed on the data output end, all complete annotations are processed into an annotation set, the image to be annotated is set as an annotated image, and the annotation set is associated with the annotated image, and finally a completed annotation signal is sent to the information output end. After the information output end receives the completed annotation signal, the currently displayed annotated image is closed.

[0044] The information processing end is used to receive facial messages and compare them with pre-stored login standards when the information inputter logs in to the data annotation system based on human-computer collaborative interaction to determine whether it is the real person; the data annotation system based on human-computer collaborative interaction has a preset login oral text set, which consists of several oral texts and corresponding standard lip shapes. After confirming that it is the real person through facial message comparison, an oral text and the corresponding standard lip shape are randomly extracted from the login oral text set, and the text content is passed to the information output end for display, and then the mouth periorbital information in the collected facial information is compared with the standard lip shape to determine whether it is a real person logging in; if it is determined that it is not the real person, a login failure message is displayed.

[0045] The information processing end is used to receive cursor information instead of typing information when highlighting a recommended set. The cursor information includes click information and position information. The click information is defined as the fifth information, and the position information is defined as the sixth information. If the fifth information received is a double-click, the sixth information received at this time is directly used as a fixed starting point, and the fixed starting point is sent to the information output end. After receiving the fixed starting point, the information output end extinguishes the recommended set and only highlights the fixed starting point.

[0046] The preset annotation type list includes a new type text box, several default annotation type options and a new annotation type option for the current image to be annotated. The new type text box is used to receive a new description label input by a user; the default annotation type option is a default annotation type in the default annotation type list. If more than 80% of the images in the same category of labeled images have the same description label, this description label is set as the default annotation type. The default annotation type list consists of several default annotation types in the same category of labeled images; the new annotation type option is a annotation type entered and confirmed in the new type text box before the data annotation of all objects in the image to be annotated is completed.

[0047] The information processing end is used to receive typed information or cursor information when the annotation type list box pops up. The specific process of processing the typed information or cursor information into the annotation type is as follows: the received typed information or cursor information includes text information of the typed information, confirmation information of the typed information, click information of the cursor information, and position information of the cursor information. The confirmation information of the typed information is defined as the 7th information, the typed text information is defined as the 8th information, the click information of the cursor information is defined as the 9th information, and the position information of the cursor information is defined as the 10th information; if the 7th information is received after the 8th information is received, the 8th information is processed as the annotation type, and the annotation type is added as a newly added annotation type option, the annotation type and the annotation range are recorded as a complete annotation, and the annotation type is passed to the information output end. After the information output end receives the annotation type, it displays the annotation type on the annotation range; if the received information is the 9th information, the annotation type is determined by the relative position relationship between the 10th information and the annotation type list, and the annotation type is passed to the information output end for display on the annotation range.

[0048] Specific usage: In this embodiment, Zhang San and Li Si are used as information inputters.

[0049] Li Si opened the data annotation system based on human-computer collaborative interaction, and the face-scanning login interface was displayed on the computer screen. The top of the screen showed "Please dictate the following: Work hard and be fulfilled every day." So Li Si put his face into the frame and said the dictation, but because Li Si spoke very quickly and his lip shape did not change much, the computer screen showed a login failure message and exited the login interface. Li Si had to reopen the data annotation system based on human-computer collaborative interaction, and the face-scanning login interface was displayed on the computer screen. The top of the screen showed "Please dictate the following: Work hard, I am the best." This time, Li Si adjusted his state, spoke at a normal speaking speed and increased the lip shape changes when dictating, and successfully logged into the data annotation system based on human-computer collaborative interaction.

[0050] Zhang San opened the data annotation system based on human-computer collaborative interaction, and the face recognition login interface was displayed on the computer screen. The top of the screen showed "Please orally state the following: The weather is really nice today, and it is another happy day at work." So Zhang San put his face in the viewfinder and spoke the orally.

[0051] After successfully logging in, Zhang San opens the first image to be annotated (the information processing end has completed loading), and the system recommends the starting point that has been annotated on the image.

[0052] After making a subjective judgment, Zhang San plans to start the first annotation from the upper left corner, so he moves his eyes to the upper left corner of the screen. At this time, the system-recommended starting point in a small area in the upper left corner of the screen is highlighted, and a selected starting point flashes in the middle of these highlighted recommended starting points. This selected starting point is the starting point Zhang San wants; at the same time, Zhang San moves the mouse to the end point he wants with his right hand and clicks the mouse. A solid-line annotation range box appears on the screen, and a list of annotation types pops up; the annotation type Zhang San wants to enter is in the third row, so he puts the cursor in the third row and clicks the mouse. At this time, the annotation type is displayed next to the annotation range box, and Zhang San has completed the first annotation.

[0053] For the second marking, Zhang San started marking from the upper right corner of the screen, so he moved his eyes to look at the upper right corner of the screen. At this time, the system-recommended starting point in a small area in the upper right corner of the screen was highlighted, and a selected starting point was flashing in the middle of these highlighted recommended starting points, but the starting point Zhang San wanted was to the upper left of this selected starting point; Zhang San pressed the up arrow key on the keyboard with his left hand, the flashing starting point went out, and the flashing point became the nearest point above the just-selected starting point, which was the pending starting point generated after Zhang San's selection; however, this pending starting point was not the starting point Zhang San wanted, and the starting point was still to the upper left of the pending starting point, so Zhang San pressed the arrow keys on the keyboard several times with his left hand to adjust the position of the pending starting point. The pending starting point changed its flashing position in sequence with the arrow keys pressed by Zhang San, and finally arrived at the starting point Zhang San wanted ; Zhang San then pressed the OK key on the keyboard with his left hand; when Zhang San found the starting point he wanted and operated the keyboard with his left hand, he also found the end point he wanted, so he operated the mouse with his right hand while operating the keyboard with his left hand, and moved the cursor displayed on the computer to the end point he wanted; After Zhang San pressed the OK key on the keyboard with his left hand, he immediately right-clicked the mouse, and a solid-line annotation range box appeared on the screen, and a list of annotation types popped up; however, the displayed list did not contain the annotation type that Zhang San wanted, so Zhang San entered the annotation type he wanted in the text input box at the top of the list (that is, the new type text box), and pressed the Enter key to confirm, and the list disappeared, and the annotation type entered by Zhang San was displayed next to the annotation range box, and Zhang San completed the second annotation.

[0054] For the third annotation, Zhang San started to annotate from the middle of the screen, so he moved his eyes to the middle of the screen. At this time, the system-recommended starting point in a small area in the middle left of the screen was highlighted, and a selected starting point flashed in the middle of these highlighted recommended starting points. This selected starting point was not the point Zhang San wanted, and the point Zhang San wanted was not displayed on the screen; so Zhang San used his right hand to operate the mouse to move the cursor to the point he wanted, and double-clicked the mouse, then the highlighted recommended set went out, and a highlighted fixed starting point appeared under the cursor; Zhang San moved the mouse again, moved the cursor to the end point he wanted, and clicked the mouse, then a solid line annotation range box appeared on the screen, and a list of annotation types popped up; the annotation type Zhang San wanted was the same as the second annotation type, and this same annotation type was displayed in the last line of the annotation type list, so he moved the cursor to the last line of the annotation type list and clicked the mouse. At this time, the annotation type was displayed next to the annotation range box, and Zhang San completed the third annotation.

[0055] Zhang San completed several more annotations. At this time, Zhang San thought that all the objects that needed to be labeled on this image had been labeled, so he pressed the button in the upper left corner of the screen to complete the annotation, and the display interface immediately closed the current annotated image (before closing the current annotated image, the information processing end has processed all complete annotations into annotation sets, set the image to be labeled as an already labeled image, and associated the annotation set with the already labeled image).

[0056] Example 2

[0057] Basically as attached Figure 2 , Figure 3 As shown, the only difference from Example 1 is that:

[0058] The information processing end is used to receive facial information and process the facial information into a working state. When the working state is determined to be a fatigue state, the fatigue state is recorded in the work fatigue log, and the fatigue signal is transmitted to the information output end. After receiving the fatigue signal, the information output end pops up a health reminder box, and a three-second countdown is displayed in the prompt box; after the countdown ends, the facial information is received to determine whether the facial information meets the health indication action requirements. If it meets the requirements, the health reminder box closing instruction is transmitted to the information output end. After receiving the health reminder box closing instruction, the information output end closes the health reminder box; if the frequency of increase of fatigue state data bars in the work fatigue log exceeds the preset warning line, the working state is determined to be a tired state, all data is saved immediately, the preset rest time countdown is started, and the forced rest command and the preset rest time are transmitted to the information output end. After receiving the forced rest command, the information output end locks the screen, pops up a forced rest health reminder, and displays a rest reminder box of the preset rest time countdown process, and issues a forced rest voice prompt. After the countdown ends, the rest reminder box is closed and the screen lock is released;

[0059] The working status includes the eye blinking frequency, the lunging frequency, the nodding arc and the gaze range obtained from the facial information. When the increase in the eye blinking frequency and the lunging frequency exceeds the set fatigue limit, the working status is judged as a fatigue state; when the gaze range or the nodding arc exceeds the fatigue limit, the working status is judged as a tired state, and it is directly assumed that the increase in the fatigue state frequency exceeds the preset warning line.

[0060] When the information processing end is used to pop up a health reminder box, the nose tip in the received facial information is set as a movement point, and the horizontal continuous turning line formed by the rapid left and right swing of the nose tip is set as the end mark, and the movement trajectory drawn by the movement point is used to judge whether the head movement of the information inputter meets the health indication action requirements; when the input facial information does not meet the health indication action requirements, a re-movement instruction is sent to the information output end, and the information output end pops up a prompt box for moving as required after receiving the re-movement instruction, and displays a three-second countdown in the prompt box for moving as required. After the countdown ends, the facial information is input again for judgment; until the input facial information meets the health indication action requirements, the health reminder box closing instruction is transmitted to the information output end, and the health reminder box is closed after the information output end receives the health reminder box closing instruction;

[0061] Specific usage: In this embodiment, Zhang San and Li Si are used as information inputters, and Li Si is a colleague of Zhang San at the adjacent workstation.

[0062] On the first day, because Zhang San did not get enough sleep and rest last night, he was not energetic when he was working today, blinking and yawning many times. The more Zhang San worked, the more tired he became. He blinked and yawned more and more frequently. Suddenly, the system popped up a prompt box, reminding him that he was in a tired working state, and displayed a 3-second countdown, so Zhang San started the health exercises that the company had trained in advance; but because Zhang San's movements were not large enough and not standard enough, the computer screen popped up a prompt box to exercise as required, and at the same time, a three-second countdown was displayed in the prompt box to exercise as required; he had to do the health exercises more vigorously and seriously after the countdown ended. After finishing the health exercises this time, the screen closed the health reminder box. Because Zhang San moved his shoulders and neck twice, and the second time he did the health exercises seriously, he was very awake at this moment, feeling full of energy, and then Zhang San devoted himself to work seriously and actively.

[0063] The next day, because Zhang San had not rested well for two consecutive days, even though he had done a health exercise, he soon fell into a state of mental fatigue, and his yawning and blinking continued to worsen. Suddenly, the screen was locked and Zhang San could not continue working. At this time, the computer sounded a reminder for Zhang San to take a break, and the screen displayed a health reminder and a countdown for the break time. Li Si heard the health reminder sound from the computer, turned around and focused on Zhang San's physical condition. He found that Zhang San was just tired and suggested that Zhang San take a break. Zhang San listened to the health reminder sound and Li Si's advice, and immediately got up and left his workstation to rest in the lounge. When he came back, he found that the countdown was about to end, so he sat in front of the computer and waited for the countdown to end. Because he had a good rest, the previous fatigue was swept away at this moment. After the countdown ended, the health reminder disappeared, the screen was unlocked, and Zhang San was full of energy and devoted to work.

[0064] Zhang San had not had a good rest for several days. One day, when he was working, he suddenly felt uncomfortable in his heart and his vision began to blur. Suddenly, the display screen was locked. At this time, the computer sounded to remind Zhang San to take a break. At the same time, the screen displayed a health reminder and a countdown to the rest time appeared. Li Si heard the health reminder sound from the computer and turned to focus on Zhang San's physical condition. He found that Zhang San had signs of fainting, so he stood up and caught the unconscious Zhang San before he fell. Then he gently put him on the ground and started first aid, and shouted for help. Thanks to Li Si's timely rescue, Zhang San was rescued on the way to the hospital.

[0065] The above is only an embodiment of the present invention, and the common knowledge such as the known specific structure and characteristics in the scheme is not described in detail here. It should be pointed out that for those skilled in the art, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. Data annotation system based on human-computer collaborative interaction, characterized by: include: Information collection end, information processing end and information output end; The information collection terminal is used to collect the image information to be annotated, the facial information recorded by the camera, the typing information recorded by the keyboard, and the cursor information recorded by the mouse; The information processing end is used to receive facial information when the information inputter logs in to the data annotation system based on human-computer collaborative interaction, compare the received facial information with the pre-stored login standard, and determine whether it is the real login of the person himself; After successful login, the cursor information is received, the position information in the cursor information is processed to obtain the corresponding image to be annotated, and then the image to be annotated is loaded as an annotated image, that is, a number of recommended starting points on the image to be annotated are obtained by boundary feature extraction, and a number of recommended starting points form a recommended starting point set, and the image to be annotated and the recommended starting point set are integrated into image loading information, and then the image loading information is passed to the information output end and displayed; Then, the facial information is received, and changes in the gaze direction and pupil size of the eyes are obtained according to the facial information analysis. Then, the starting point range is determined according to the changes in the gaze direction and pupil size of the eyes. All recommended starting points within the starting point range are formed into a recommendation set, and the point closest to the geometric center of the recommendation set is used as the selected starting point. Then, the recommendation set and the selected starting point are sent to the information output end. After receiving the recommendation set, the information output end highlights the recommendation set and flashes the selected starting point. The information processing end is further used to receive input information when highlighting the recommended set, the input information includes determining the key combination information and the direction information, the determining the key combination information is defined as the first information, and the direction information is defined as the second information; if the first information is received, the selected starting point is transmitted to the information output end as the fixed starting point, and after receiving the fixed starting point, the information output end turns off the recommended set and only highlights the fixed starting point; If the second information is received, the selected starting point is adjusted to the recommended starting point in the corresponding direction according to the second information, and the recommended starting point is used as the pending starting point, and the pending starting point is sent to the information output terminal. After receiving the pending starting point, the information output terminal turns off the selected starting point in the highlighted recommendation set and flashes the pending starting point. When the input information is not received for more than a preset time, or the overlap between the current eye gaze direction change and the pupil size change and the initial starting point range is lower than a preset value, the pending starting point is used as the fixed starting point; The information processing end is also used to determine the end point range according to the current eye gaze direction change and pupil size change when the fixed starting point is highlighted, and then write the geometric center point of the end point range as the selected end point into the end point range, and transmit the end point range to the information output end, and the information output end will display the received end point range on the corresponding area; receive cursor information, the cursor information includes click information and position information, the click information is defined as the third information, and the position information is defined as the fourth information; when the third information received is a single click, the fourth information is used as the fixed end point, and the line segment formed by connecting the fixed starting point and the fixed end point is used as a diagonal line to form a marked range, and the marked range is transmitted to the information output end, and the information output end displays the marked range as a solid color line frame after receiving it, and pops up a preset mark type list box; The information processing end is also used to receive typed information or cursor information when a preset annotation type list box pops up, process the typed information or cursor information into an annotation type, record the annotation type and annotation range as a complete annotation, and pass the annotation type to the information output end. After the information output end receives the annotation type, it displays the annotation type on the annotation range; after the information processing end records the complete annotation, when the received cursor information is to click the confirmation completion button displayed on the data output end, all complete annotations are processed into an annotation set, the image to be annotated is set as an annotated image, and the annotation set is associated with the annotated image, and finally a completed annotation signal is sent to the information output end. After the information output end receives the completed annotation signal, the currently displayed annotated image is closed.

2. The data annotation system based on human-computer collaborative interaction according to claim 1 is characterized in that: The information processing end is used to receive facial messages when the information inputter logs in to the data annotation system based on human-computer collaborative interaction, and first compare the facial information with the pre-stored login standard to determine whether it is the person himself; the data annotation system based on human-computer collaborative interaction is preset with a login oral text set, which consists of a number of oral texts and corresponding lip shape standards. After confirming that it is the person himself through facial message comparison, a spoken text and the corresponding lip shape standard are randomly extracted from the login oral text set, and the text content is transmitted to the information output end for display, and then the periorbital information in the collected facial information is compared with the lip shape standard to determine whether it is a real person logging in; If it is determined that the user is not the real person, a login failure message will be displayed.

3. The data annotation system based on human-computer collaborative interaction according to claim 2 is characterized in that: The information processing end is used to receive cursor information instead of typing information when highlighting a recommended set. The cursor information includes click information and position information. The click information is defined as the fifth information, and the position information is defined as the sixth information. If the fifth information received is a double-click, the sixth information received at this time is directly used as a fixed starting point, and the fixed starting point is sent to the information output end. After receiving the fixed starting point, the information output end extinguishes the recommended set and only highlights the fixed starting point.

4. The data annotation system based on human-computer collaborative interaction according to claim 3 is characterized in that: The preset annotation type list includes a new type text box, several default annotation type options and a new annotation type option for the current image to be annotated. The new type text box is used to receive a new description label input by a user; the default annotation type option is a default annotation type in the default annotation type list. If more than 80% of the images in the same category of labeled images have the same description label, this description label is set as the default annotation type. The default annotation type list consists of several default annotation types in the same category of labeled images; the new annotation type option is a annotation type entered and confirmed in the new type text box before the data annotation of all objects in the image to be annotated is completed.

5. The data annotation system based on human-computer collaborative interaction according to claim 4 is characterized in that: The information processing end is used to receive typed information or cursor information when the annotation type list box pops up. The specific process of processing the typed information or cursor information into the annotation type is as follows: the received typed information or cursor information includes text information of the typed information, confirmation information of the typed information, click information of the cursor information, and position information of the cursor information. The confirmation information of the typed information is defined as the 7th information, the typed text information is defined as the 8th information, the click information of the cursor information is defined as the 9th information, and the position information of the cursor information is defined as the 10th information; if the 7th information is received after the 8th information is received, the 8th information is processed as the annotation type, and the annotation type is added as a newly added annotation type option, the annotation type and the annotation range are recorded as a complete annotation, and the annotation type is passed to the information output end. After the information output end receives the annotation type, it displays the annotation type on the annotation range; if the received information is the 9th information, the annotation type is determined by the relative position relationship between the 10th information and the annotation type list, and the annotation type is passed to the information output end for display on the annotation range.

6. The data annotation system based on human-computer collaborative interaction according to claim 5 is characterized in that: The information processing end is used to receive facial information and process the facial information into a working state. When the working state is determined to be a fatigue state, the fatigue state is recorded in the work fatigue log, and the fatigue signal is transmitted to the information output end. After receiving the fatigue signal, the information output end pops up a health reminder box, and a three-second countdown is displayed in the prompt box; after the countdown ends, the facial information is received to determine whether the facial information meets the health indication action requirements. If it meets the requirements, the health reminder box closing instruction is transmitted to the information output end. After receiving the health reminder box closing instruction, the health reminder box is closed; If the frequency of increase of fatigue status data bars in the work fatigue log exceeds the preset warning line, the work status will be judged as a tired state, all data will be saved immediately, the countdown to the preset rest time will be started, and the forced rest command and the preset rest time will be transmitted to the information output end. After receiving the forced rest command, the information output end will lock the screen, pop up a forced rest health reminder, and display the rest reminder box of the preset rest time countdown process. At the same time, a forced rest voice reminder will be issued. After the countdown is over, the rest reminder box will be closed and the screen lock will be released.

7. The data annotation system based on human-computer collaborative interaction according to claim 6 is characterized in that: The working status includes the eye blinking frequency, the lunging frequency, the nodding arc and the gaze range obtained from the facial information. When the increase in the eye blinking frequency and the lunging frequency exceeds the set fatigue limit, the working status is judged as a fatigue state; when the gaze range or the nodding arc exceeds the fatigue limit, the working status is judged as a tired state, and it is directly assumed that the increase in the fatigue state frequency exceeds the preset warning line.

8. The data annotation system based on human-computer collaborative interaction according to claim 7 is characterized in that: When the information processing end is used to pop up a health reminder box, the nose tip in the received facial information is set as the movement point, and the horizontal continuous turning line formed by the rapid left and right swing of the nose tip is set as the end mark. The movement trajectory drawn by the movement point is used to judge whether the head movement of the person entering the information meets the health indication action requirements; when the entered facial information does not meet the health indication action requirements, a re-movement instruction is sent to the information output end. After the information output end receives the re-movement instruction, a prompt box for moving as required is popped up, and a three-second countdown is displayed in the prompt box for moving as required. After the countdown ends, the facial information is entered again for judgment; until the entered facial information meets the health indication action requirements, an instruction to close the health reminder box is passed to the information output end. The information output end closes the health reminder box after receiving the instruction to close the health reminder box.

9. A data annotation method based on human-computer collaborative interaction, characterized in that: A data annotation system based on human-computer collaborative interaction as described in any one of claims 1 to 8 is used.

Images