Model training method, device, electronic device and storage medium
Through the web-based image annotation method, the problem of time-consuming and labor-consuming image annotation in the prior art is solved, and the image shortcut annotation and model quality improvement is achieved, personalized needs are met, and historical model data management is supported.
Patent Information
- Application Number
- CN202210259714.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-16
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-03-16
AI Technical Summary
The existing image annotation methods are time-consuming and labor-intensive, difficult to meet personalized needs, and lack flexible web-based full-function image annotation tools, making it difficult to modularly embed management systems.
Provide a web-based image annotation method, which allows content information to be marked by obtaining training image data sets, supports keyboard or mouse operation behavior judgment, realizes label area overlay and browser window size change operations, and allows users to adjust the number of samples and label content. Supports model management and report generation.
It realizes the quick behavior labeling and modification of images, improves model quality, meets personalized needs, and supports orderly management of historical model data.
Smart Images

Figure CN114663975B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a model training method, device, electronic equipment and storage medium. Background Art
[0002] The method of image annotation and model training by professional technicians cannot meet the personalized needs and efficiency requirements of many schools.
[0003] Disadvantages of existing image annotation: The model in which professional engineers manually annotate images using client-side annotation tools and then conduct training is accurate but time-consuming and labor-intensive. The client software is inconvenient to use and difficult to modularly embed into the management system.
[0004] Currently, there are few full-featured and flexible image annotation tools based on the web. Most image annotation work needs to be implemented with the help of clients, such as LabelImg. Web-based image annotation solutions are more flexible to use and more agile to deploy and upgrade. Summary of the invention
[0005] The present invention provides a model training method, device, electronic device and storage medium, which can quickly annotate and modify the behavior of images, allowing users to adjust the number of samples for training according to actual conditions, and at the same time verify the effect of the model, conveniently and efficiently improve the quality of the model, and orderly manage the historical model data of specific users to meet personalized needs.
[0006] In a first aspect, an embodiment of the present invention provides a model training method, the training method comprising:
[0007] Acquire a training image data set; wherein the training image data set includes classroom student images;
[0008] Annotating the images in the data set with content information; wherein the content information includes student behavior types;
[0009] The annotated data set is trained to obtain an image recognition model; wherein the model training process includes adjusting the model, and the adjustment includes adjusting the number of images in the image data set and adjusting the content information of the image annotations. Optionally, the training method also includes model management, and the model management includes:
[0010] Get user-level permissions;
[0011] Retrieve, view or optimize historical models according to the user level permissions.
[0012] Optionally, providing a web-based image annotation function to annotate content information of images in the data set includes:
[0013] Determine keyboard or mouse operation behavior;
[0014] performing operations on the image according to the judgment result;
[0015] The operations include:
[0016] The annotation area superposition operation during annotation includes: after listening to the pressing of the keyboard w key, setting the edit area frame element to a non-editable and mouse-transmissible state and setting the level to a low level; after the frame selection is completed, generating a new editable frame element and setting the newly generated edit frame element or the selected element to a high level;
[0017] The browser window size change operation includes: adding a page size change monitoring function at the beginning of the page life cycle, reloading the image when the page size changes, obtaining the existing size information of the image, comparing it with the original proportion of the image to obtain the zoom factor, and reloading the data and page elements with the zoom factor.
[0018] In a second aspect, an embodiment of the present invention provides an image recognition method, wherein the recognition includes:
[0019] Acquire a training image data set; wherein the training image data set includes classroom student images;
[0020] Annotating the images in the data set with content information; wherein the content information includes student behavior types;
[0021] The annotated data set is trained to obtain an image recognition model; wherein the model training process includes adjusting the model, and the adjustment includes adjusting the number of images in the image data set and adjusting the content information of the image annotations;
[0022] Acquire a test image dataset; wherein the test image dataset includes classroom student images;
[0023] The test image data set is identified by using the image recognition model to identify the behavior type of the students in the image and mark it; wherein the process of identifying the test image data set includes viewing the identified and marked test images.
[0024] Optionally, the identification method further includes generating an identification report, and the generating of the identification report includes:
[0025] Obtain statistical data of various behaviors generated by the uploaded historical training set and prediction set; wherein the statistical data includes AP, recognition number and annotation number of various behaviors;
[0026] The data is exported in TXT format.
[0027] In a third aspect, an embodiment of the present invention provides a model training device, the training device comprising:
[0028] A data acquisition module is used to acquire a training image data set; wherein the training image data set includes classroom student images;
[0029] An image annotation module annotates the images in the data set with content information; wherein the content information includes student behavior types;
[0030] The model training module trains the annotated data set to obtain an image recognition model; wherein the model training process includes adjusting the model, and the adjustment includes adjusting the number of images in the image data set and adjusting the content information of the image annotations.
[0031] Optionally, the training device further comprises:
[0032] The model management module obtains user-level permissions and retrieves, views or optimizes historical models according to the user-level permissions.
[0033] Optionally, the training device further comprises:
[0034] The report generation module obtains the statistical data of various behaviors generated by the uploaded historical training set and prediction set and exports the data in TXT format; wherein the statistical data includes the AP, recognition number and annotation number of various behaviors.
[0035] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the program, the method described in any one of the first aspects is implemented.
[0036] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of the first aspects.
[0037] Beneficial Effects
[0038] The present invention provides a model training method, device, electronic device and storage medium, wherein the training method first obtains a training image data set; wherein the training image data set includes classroom student images; then the images in the data set are annotated with content information; wherein the content information includes student behavior types; finally, the annotated data set is trained to obtain an image recognition model; wherein the model training process includes adjusting the model, and the adjustment includes adjusting the number of images in the image data set and adjusting the content information of the image annotations. The image can be quickly annotated and modified for behavior, allowing the user to adjust the number of samples for training according to actual conditions, and at the same time, the effect of the model can be verified, and the quality of the model can be improved conveniently and efficiently. The historical model data of a specific user can also be managed in an orderly manner to meet personalized needs.
[0039] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements.
[0041] Figure 1 A flow chart of a model training method according to an embodiment of the present invention is shown;
[0042] Figure 2 A flow chart showing an image recognition method according to an embodiment of the present invention is shown;
[0043] Figure 3 A schematic diagram showing the structure of a model training device according to an embodiment of the present invention is shown;
[0044] Figure 4 A structural diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0045] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.
[0046] It should be noted that the description of the embodiments of the present invention is only for more clearly illustrating the technical solutions of the embodiments of the present invention, and does not constitute a limitation on the technical solutions provided by the embodiments of the present invention.
[0047] Figure 1 A flow chart of a model training method according to an embodiment of the present invention is shown; see Figure 1 , the training method comprises:
[0048] S20, obtaining a training image dataset;
[0049] Wherein, the training image data set includes classroom student images;
[0050] The images are obtained through video recording or other image acquisition tools and packaged and uploaded in zip format.
[0051] S40, annotating the images in the data set with content information; wherein the content information includes the student behavior type; the image annotation is constructed using a div element;
[0052] Specifically, the image annotation function can be used independently through component development, or embedded in other systems developed with the react framework; secondary development based on the react-moveable plug-in realizes the image annotation function, box selection, dragging, and resizing, and also has keyboard shortcuts to start annotation, delete, cancel, and confirm operations. The annotation box and annotation information can adapt to the size of the browser window;
[0053] The react-moveable plug-in itself is mainly used for dragging and moving list data and changing the size and position of a single element. Through secondary development, the following functions are added to meet the functional requirements of the module: annotation area overlay operation, rendering logic of multiple annotation elements, real-time response to zooming when the browser window size changes, and adding styles for the active state of the annotation box selection;
[0054] Among them, the main implementation logic of the annotation area overlay operation during annotation is to monitor the keyboard w key being pressed, set the edit area box element to a non-editable and mouse-penetrable state and set the level to a low level. When the box selection is completed, generate a new editable box element and set the newly generated edit box element or the selected element to a high level.
[0055] The main implementation logic of real-time response and scaling of browser window size changes is to add a page size change monitoring function at the beginning of the page life cycle, reload the image when the page size changes, obtain the existing size information of the image, compare it with the original proportion of the image to obtain the scaling factor, and reload the data and page elements with the scaling factor.
[0056] S60, training the labeled data set to obtain an image recognition model;
[0057] The model training process includes adjusting the model, and the adjustment includes adjusting the number of images in the image data set and adjusting the content information of the image annotations;
[0058] Specifically, the user adjusts the number of samples for training according to actual conditions, re-labels the images in the samples, adds new annotation content or deletes the annotated content, and increases or decreases the number of annotation types.
[0059] The training method also includes model management, and the model management includes:
[0060] Get user-level permissions;
[0061] Retrieve, view or optimize historical models according to the user level permissions.
[0062] The embodiment of the present invention provides a model training method, which first obtains a training image data set; wherein the training image data set includes classroom student images; then the images in the data set are annotated with content information; wherein the content information includes student behavior types; finally, the annotated data set is trained to obtain an image recognition model; wherein the model training process includes adjusting the model, and the adjustment includes adjusting the number of images in the image data set and adjusting the content information of the image annotations. The image can be quickly annotated and modified, allowing the user to adjust the number of samples for training according to the actual situation. At the same time, the effect of the model can be verified, and the quality of the model can be improved conveniently and efficiently. The historical model data of a specific user can also be managed in an orderly manner to meet personalized needs.
[0063] Figure 2 A flowchart of an image recognition method according to an embodiment of the present invention is shown; see Figure 2 , the identification method comprises:
[0064] S20, obtaining a training image dataset;
[0065] Wherein, the training image data set includes classroom student images;
[0066] S40, annotating the images in the data set with content information;
[0067] Wherein, the content information includes student behavior type;
[0068] S60, training the labeled data set to obtain an image recognition model;
[0069] The model training process includes adjusting the model, and the adjustment includes adjusting the number of images in the image data set and adjusting the content information of the image annotations.
[0070] S70, obtaining a test image dataset; wherein the test image dataset includes classroom student images;
[0071] The images are obtained through video recording or other image acquisition tools and packaged and uploaded in zip format.
[0072] S80, using the image recognition model to recognize the test image dataset, identify the behavior type of the students in the image and mark it;
[0073] The process of performing recognition on the test image data set includes viewing the recognized and labeled test images.
[0074] The identification method further includes generating an identification report, and the generating of the identification report includes:
[0075] Obtain statistical data of various behaviors generated by the uploaded historical training set and prediction set, wherein the statistical data includes AP, recognition number, and annotation number of various behaviors;
[0076] Export the data in TXT format;
[0077] Among them, AP:
[0078] The intersection-over-union (IOU) measures the degree of overlap between two detection boxes (for target detection);
[0079] True Positive (TP): the number of detection boxes with IoU>IoU threshold (usually 0.5);
[0080] False Positive (FP): the number of detection boxes with IoU <= IoU threshold;
[0081] Precision: TP / (TP+FP);
[0082] Recall: TP / (TP+FN);
[0083] The PR curve is the precision-recall curve, with recall as the horizontal axis and precision as the vertical axis;
[0084] AP is to calculate the area under the PR curve of a certain type of behavior;
[0085] The number of predictions refers to the number of predictions of a certain type of behavior in the test set;
[0086] The number of annotations refers to the number of manual annotations for a certain type of behavior in the training set.
[0087] The recognition report can be used to verify the effect of the model, thereby improving the quality of the model conveniently and efficiently.
[0088] The beneficial effects of the present invention are described below with a preferred embodiment;
[0089] Obtain the uploaded classroom student image zip collected by video or other image acquisition tools, select the image set that has been uploaded, select a specific photo in the image set, and the web program obtains the existing annotation information of the image from the server for display; wherein the existing annotation information includes the image number and size;
[0090] After acquisition, adjust the annotations. Press the w key to start the selection. After the selection is completed, the preset behavior name appears. Press Enter to select or Esc to cancel the selection. Click the picture box or double-click a behavior name in the annotation list to select it. The eight direction points can adjust the box size. Drag to adjust the position. Delete to delete the selected box. The behavior names include stand up, writing, student_look book, student_raise hand, no listen, and point blackboard.
[0091] After all images in the image data set are annotated, click Start Training to obtain the recognition model. During the training process, you can also adjust the training data set, increase or delete the number of samples in the data set, modify the annotation content of the annotated images, and increase or modify the type of annotation behavior;
[0092] Get the zip of classroom student images for testing, use the trained recognition model to recognize the images in the test data set and annotate the student behavior types in the images. During the recognition process, you can view the recognized images to view the predicted annotation information.
[0093] After all test data sets are identified, select the training set and test set and click Generate Report to export the report in TXT format. The report content includes the AP, predicted number, and labeled number of each behavior.
[0094] Analyze the report to verify the effect of the recognition model. If the effect is not good, you can use the model management function to call out the generated recognition model and adjust it to achieve better results.
[0095] The embodiments of the present invention can quickly annotate and modify the behavior of images, allowing users to adjust the number of samples for training according to actual conditions. At the same time, the effect of the model can be verified, and the quality of the model can be improved conveniently and efficiently. The historical model data of specific users can also be managed in an orderly manner to meet personalized needs.
[0096] Based on the same inventive concept, an embodiment of the present invention also provides a model training device, which can be used to implement a model training method described in the above embodiment, as described in the following embodiment: Since the principle of solving the problem by the model training device is similar to that of a model training method, the implementation of a model training device can refer to the implementation of a model training method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0097] Figure 3 FIG. 4 is a block diagram showing a model training device according to an embodiment of the present invention. Figure 3 As shown, the training device comprises:
[0098] The data acquisition module 20 acquires a training image data set; wherein the training image data set includes classroom student images;
[0099] An image annotation module 40 annotates the images in the data set with content information; wherein the content information includes student behavior types;
[0100] The model training module 60 trains the annotated data set to obtain an image recognition model; wherein the model training process includes adjusting the model, and the adjustment includes adjusting the number of images in the image data set and adjusting the content information of the image annotations.
[0101] The embodiment of the present invention provides a model training device, which first obtains a training image data set through a data acquisition module 20; wherein the training image data set includes classroom student images; then, through an image annotation module 40, the images in the data set are annotated with content information; wherein the content information includes student behavior types; finally, through a model training module 60, the annotated data set is trained to obtain an image recognition model; wherein the model training process includes adjusting the model, and the adjustment includes adjusting the number of images in the image data set and adjusting the content information of the image annotations. The above technical solution can be used to quickly annotate and modify the behavior of images, allowing users to adjust the number of samples for training according to actual conditions. At the same time, the effect of the model can be verified, and the quality of the model can be improved conveniently and efficiently. The historical model data of specific users can also be managed in an orderly manner to meet personalized needs.
[0102] The training device also includes:
[0103] Model management module 70, after obtaining user-level permissions, retrieves, views or optimizes historical models according to the user-level permissions;
[0104] The report generation module 80 obtains the statistical data of various behaviors generated by the uploaded historical training set and prediction set and exports the data in TXT format; wherein the statistical data includes the AP, recognition number and annotation number of various behaviors.
[0105] The model management module 70 can be used to verify the effect of the model and improve the model quality conveniently and efficiently; the report generation module 80 can be used to manage the historical model data of a specific user in an orderly manner to meet personalized needs.
[0106] An embodiment of the present invention further provides a computer electronic device, Figure 4 FIG. 1 shows a schematic diagram of the structure of an electronic device to which an embodiment of the present invention can be applied. Figure 4 As shown, the computer electronic device includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403. In the RAM 403, various programs and data required for system operation are also stored. The CPU 401, the ROM 402 and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0107] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.
[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0109] The units or modules involved in the embodiments of the present invention may be implemented by software or hardware. The units or modules described may also be provided in a processor, for example, may be described as: a processor including a data acquisition module 20, an image annotation module 40 and a model training module 60, wherein the names of these modules do not constitute a limitation on the modules themselves in certain circumstances, for example, the model training module 60 may also be described as "a model training module 60 for training the annotated data set to obtain an image recognition model".
[0110] As another aspect, the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in a model training device described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into an electronic device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute a model training method described in the present invention.
[0111] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.
Claims
1. A model training method, It is characterized in that The training method includes: Acquire a training image data set; wherein the training image data set includes classroom student images; Annotating the images in the data set with content information; wherein the content information includes student behavior types; The annotated data set is trained to obtain an image recognition model; wherein the model training process includes adjusting the model, and the adjustment includes adjusting the number of images in the image data set and adjusting the content information of the image annotations. The step of labeling the images in the data set with content information includes: Determine keyboard or mouse operation behavior; performing operations on the image according to the judgment result; The operations include: The annotation area superposition operation during annotation includes: after listening to the pressing of the keyboard w key, setting the edit area frame element to a non-editable and mouse-transmissible state and setting the level to a low level; after the frame selection is completed, generating a new editable frame element and setting the newly generated edit frame element or the selected element to a high level; The browser window size change operation includes: adding a page size change monitoring function at the beginning of the page life cycle, reloading the image when the page size changes, obtaining the existing size information of the image, comparing it with the original proportion of the image to obtain the zoom factor, and reloading the data and page elements with the zoom factor.
2. The training method according to claim 1, It is characterized in that The training method also includes model management, and the model management includes: Get user-level permissions; Retrieve, view or optimize historical models according to the user level permissions.
3. An image recognition method, It is characterized in that The identification method comprises: Acquire a training image data set; wherein the training image data set includes classroom student images; Annotating the images in the data set with content information; wherein the content information includes student behavior types; The annotated data set is trained to obtain an image recognition model; wherein the model training process includes adjusting the model, and the adjustment includes adjusting the number of images in the image data set and adjusting the content information of the image annotations; Acquire a test image dataset; wherein the test image dataset includes classroom student images; The test image dataset is identified by using the image recognition model, and the behavior type of the students in the image is identified and annotated; wherein viewing the identification process of the test image dataset includes viewing the identified and annotated test images, Among them, the content information annotation of the images in the data set includes: judging the operation behavior of the keyboard or mouse; operating the image according to the judgment result; wherein the operation includes: the annotation area superposition operation during annotation, and the implementation of the annotation area superposition operation during annotation includes: after listening to the pressing of the keyboard w key, setting the editing area frame element to a non-editable and mouse-penetrable state and setting the level to a low level, after the box selection is completed, generating a new editable frame element and setting the newly generated editing frame element or the selected element to a high level; browser window size change operation, the implementation of the browser window size change operation includes: adding a page size change listening function at the beginning of the page life cycle, reloading the picture when the page size changes, obtaining the existing size information of the picture, comparing it with the original ratio of the picture to obtain the scaling factor, and reloading the data and page elements with the scaling factor.
4. The identification method according to claim 3, It is characterized in that The identification method further includes generating an identification report, wherein the generating of the identification report includes: Obtain statistical data of various behaviors generated by the uploaded historical training set and prediction set; wherein the statistical data includes AP, recognition number and annotation number of various behaviors; The data is exported in TXT format.
5. A model training device, It is characterized in that The training device comprises: A data acquisition module is used to acquire a training image data set; wherein the training image data set includes classroom student images; An image annotation module annotates the images in the data set with content information; wherein the content information includes student behavior types; A model training module is used to train the labeled data set to obtain an image recognition model; wherein the model training process includes adjusting the model, and the adjustment includes adjusting the number of images in the image data set and adjusting the content information of the image annotations. Among them, the content information annotation of the images in the data set includes: judging the operation behavior of the keyboard or mouse; operating the image according to the judgment result; wherein the operation includes: the annotation area superposition operation during annotation, and the implementation of the annotation area superposition operation during annotation includes: after listening to the pressing of the keyboard w key, setting the editing area frame element to a non-editable and mouse-penetrable state and setting the level to a low level, after the box selection is completed, generating a new editable frame element and setting the newly generated editing frame element or the selected element to a high level; browser window size change operation, the implementation of the browser window size change operation includes: adding a page size change listening function at the beginning of the page life cycle, reloading the picture when the page size changes, obtaining the existing size information of the picture, comparing it with the original ratio of the picture to obtain the scaling factor, and reloading the data and page elements with the scaling factor.
6. The training device according to claim 5, It is characterized in that The training device also includes: The model management module obtains user-level permissions and retrieves, views or optimizes historical models according to the user-level permissions.
7. The training device according to claim 6, It is characterized in that The training device also includes: The report generation module obtains the statistical data of various behaviors generated by the uploaded historical training set and prediction set and exports the data in TXT format; wherein the statistical data includes the AP, recognition number and annotation number of various behaviors.
8. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory. It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
