Quick image recognition method and system based on yov8 architecture

Abstract

The invention belongs to the technical field of computer vision, and particularly provides a quick image recognition method and system based on a yolov8 architecture, and the method comprises the steps: modifying a yolov8 serial network architecture into a parallel network architecture, and enabling image data to pass through a network of a teacher model and a student model during forward propagation; combining the positive sample intervals of the student model and the teacher model into a unified positive sample interval based on a yov8 positive sample distribution method; designing a loss function to obtain an optimized knowledge distillation model; and quickly outputting an image recognition result based on the optimized knowledge distillation model. Aiming at a current leading target detection model, namely, the yolov8, an innovative knowledge distillation system is provided, and aims at further optimizing the performance of the yolov8 through the system and expanding the application potential of the yolov8 in a resource-constrained environment. The yolov8 not only maintains the rapid and accurate target detection capability, but also realizes better balance between computing resource consumption and model performance.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a method and system for rapid image recognition based on a yolov8 architecture. Background Art

[0002] In the field of computer vision, the object detection task is one of the core issues. The YOLO (YouOnlyLookOnce) series of models are well-known for their fast and accurate object detection capabilities. However, as the model size increases, the demand for computing resources also increases. Knowledge distillation technology transfers the knowledge of a large, complex model (teacher model) to a small, simple model (student model), improving its performance while maintaining a small model size, and has become an important research direction for model optimization.

[0003] In traditional technologies, deep learning models, especially large models, face challenges in deploying in resource-constrained environments. In addition, although small models consume less resources, their performance is usually lower than that of large models. Since YOLOv8 has a high detection speed, it is suitable for scenarios that require real-time feedback, such as video surveillance and autonomous driving. In the current environment where autonomous driving is in full swing, how to further optimize the performance of YOLOv8 while maintaining the algorithm model's ability to detect targets quickly and accurately, and expand its application potential in resource-constrained environments. Summary of the invention

[0004] The present disclosure aims to solve at least one of the technical problems existing in the prior art, and proposes a method and system for rapid image recognition based on the yolov8 architecture.

[0005] In a first aspect, the present disclosure provides a method for rapid image recognition based on the yolov8 architecture, comprising the following steps: S1, the serial network architecture of yolov8 is modified into a parallel network architecture, so that the image data passes through the network of the teacher model and the student model during forward propagation; S2, based on the positive sample allocation method of yolov8, combines the positive sample intervals of the student model and the teacher model into a unified positive sample interval; S3, design the loss function to obtain the optimized knowledge distillation model; S4, after training based on the optimized knowledge distillation model, quickly outputs the image recognition results.

[0006] Preferably, the S1 specifically includes: Design the network sublayer Empty for image data stream separation and the network sublayer Detect_Distill for image data stream fusion; Modify the number of channels and step size of the corresponding network layer so that the network can run normally.

[0007] Preferably, the S2 specifically includes: Expand the original sample interval containing only two areas to a sample interval of four areas; Different distillation strategies are implemented in these four areas.

[0008] Preferably, the S3 specifically includes: The loss function design includes the sum of parts, namely classification loss loss_cls, prediction box loss loss_box and distribution focus loss loss_dfl; where C_cls, C_box, and C_dfl are the coefficients of these parts, which are 0.7, 1.1, and 1.0 respectively.

[0009] Preferably, the classification loss is calculated as follows: Assume S_cls is the student's prediction for the classification after sigmoid, T_cls is the teacher's prediction for the classification after sigmoid, BCE stands for binary cross entropy, gt_cls is the ground truth, S_cls_t is the student's prediction after sigmoid_T, and T_cls_t is the teacher's prediction after sigmoid_T, area_i is the mask of each region, where sigmoid_T is defined as follows: The loss_classification of region 1 is, where a and b range from 0 to 1: The loss_classification of region 2 is: The loss_classification of region 3 is: The loss_classification of region 4 is: The weights for region 1 are: The weights for region 2 are: The weights for region 3 are: The weights for region 4 are: The sum of these weights is: The loss of the final classification part is: .

[0010] Preferably, the prediction box loss is calculated as follows: Assume S_box is the student's prediction of the bounding box and T_box is the teacher's prediction of the bounding box, where CIOU stands for Complete Intersection Over Union, and gt_box is still the ground truth; The bounding box loss is calculated as follows: The loss_bounding_box for region 1 is: The loss_bounding_box for region 2 is: The loss_bounding_box for region 3 is: The loss of the final prediction box is: .

[0011] Preferably, the distributed focus loss is calculated as follows: Assume S_dfl is the student's prediction of dfl, and T_dfl is the teacher's prediction of dfl. Here CE stands for cross entropy. gt_dfl still represents the true value.

[0012] The loss_dfl in region 1 is: The loss_dfl in region 2 is: The loss_dfl in region 3 is: The final loss of the dfl part is: .

[0013] Preferably, the S3 specifically includes: For the two sample intervals of teachers and students respectively, the sample intervals are divided into four areas, so that different loss function calculation strategies are adopted respectively; the essence is to introduce the results of positive sample allocation of the teacher model as guidance information, so as to better guide training.

[0014] The present invention also provides a rapid image recognition system based on the yolov8 architecture, which can be used to implement the rapid image recognition method based on the yolov8 architecture, and the system includes: An architecture modification module is configured to modify the serial network architecture of yolov8 into a parallel network architecture, so that the image data passes through the network of the teacher model and the student model during forward propagation; The positive sample allocation module is configured as a positive sample allocation method based on yolov8, which combines the positive sample intervals of the student model and the teacher model into a unified positive sample interval; A loss function design module, configured to design a loss function to obtain an optimized knowledge distillation model; The output module is configured to quickly output image recognition results after training based on the optimized knowledge distillation model.

[0015] Beneficial effects of the present invention: 1. Improved the performance of the yolov8 small model, making it have better application potential in resource-constrained environments; 2. Reduced the computing resource consumption of large yolov8 models, suitable for a wider range of application scenarios; 3. Through the design of specific loss functions, the efficiency of knowledge transfer between teacher and student models in the knowledge distillation process is optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a method for rapid image recognition based on the yolov8 architecture provided in an embodiment of the present disclosure; Figure 2 A positive sample allocation flow chart provided for an embodiment of the present disclosure, illustrating a fusion process diagram of positive samples of a teacher and a student model; Figure 3 Schematic diagram of loss function design for four regions provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0018] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure are not any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" are not limited to quantity, but are at least one. "Including" or "including" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used for relative positional relationships. When the absolute position of the described target changes, the relative positional relationship may also change accordingly.

[0019] In each of the accompanying drawings, the same elements are represented by similar reference numerals. For the sake of clarity, the various parts in the accompanying drawings are not all drawn to scale. In addition, some well-known parts may not be shown in the drawings.

[0020] Many specific details of the present disclosure are described below, such as component structures, materials, dimensions, processing techniques, and technologies, in order to more clearly understand the present disclosure. However, as those skilled in the art will appreciate, the present disclosure may be implemented without following these specific details.

[0021] The initial serial network architecture of yolov8 has the following characteristics: good flexibility, in theory, any network can be built by writing a yaml file; but the disadvantage is that the entire network is based on a serial design, and the input data can only be output through each network layer in turn, which cannot meet the needs of knowledge distillation training. Therefore, the embodiment of the present invention provides a method for rapid image recognition based on the yolov8 architecture, such as Figure 1-Figure 3 As shown, the specific steps include: S1, the serial network architecture of yolov8 is modified into a parallel network architecture, so that the image data passes through the network of the teacher model and the student model during forward propagation.

[0022] The initial network of yolov8 is specified by the yolov8.yaml file (the file is in the ultralytics\cfg\models\v8 folder under the ultralytics-main folder). The specific content is divided into two parts as a whole, the backbone network and the head. The backbone network is constructed by alternating multiple Conv and C2f layers, which is responsible for completing the feature extraction task; the SPPF layer is responsible for integrating the extracted features. The head is constructed by multiple Upsample and Concat layers, which are responsible for fusing features; the Detect layer is responsible for judging the location and category of the target object based on the features, and outputting the final prediction result.

[0023] S2, based on the positive sample allocation method of yolov8, combines the positive sample intervals of the student model and the teacher model into a unified positive sample interval; the teacher model and the student model can be replaced by any other models. Here, yolov8s and yolov8n are used as the teacher model and the student model as examples respectively.

[0024] S3, design the loss function to obtain the optimized knowledge distillation model. That is, based on the characteristics of the sigmoid activation function, an improved distillation method sigmoid_T is proposed, and the temperature control of distillation is achieved by introducing the temperature variable T to obtain the optimized knowledge distillation model; S4, after training based on the optimized knowledge distillation model, quickly outputs the image recognition results.

[0025] It should be noted that the technical solution disclosed in the present invention does not limit the order of the above steps, that is, the execution order of the steps can be arranged arbitrarily. Through the system design and algorithm optimization of the present invention, YOLOv8 not only maintains its fast and accurate target detection capabilities, but also achieves a better balance between computing resource consumption and model performance.

[0026] The knowledge distillation improved code of yolov8 provided by the present invention has the following characteristics: based on the flexibility of the initial code, theoretically, by writing a yaml file, knowledge distillation training for any two teacher and student models can be realized; through data caching and aggregation, parallel computing is realized, which increases the memory usage during training and inference to a certain extent. Two modes of knowledge distillation can be achieved by setting whether the teacher layer is frozen: (a) When the teacher layer is frozen, it is used to guide training, that is, the teacher model parameters are pre-loaded and the corresponding network layer is frozen. By setting a suitable loss function, the training process can achieve the effect of transferring the "knowledge" of the teacher model to the student model; (During training, the corresponding network layer of the teacher can be frozen through the following command model.train(data=dataset,freeze=teacher_peer_list) where teacher_peer_list=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,45]); (b) When the teacher layer is not frozen, it is used for simultaneous training, that is, the teacher model and the student model are trained synchronously. Similarly, by setting a suitable loss function, the "knowledge" of the teacher model can be transferred to the student model in real time.

[0027] In addition to the above, the following settings need to be completed to perform distillation training correctly.

[0028] (1) Add the definition of new modules in `ultralytics\nn\modules\__init__.py` to ensure that they can be referenced by other modules; (2) Add a reference to the new module in `ultralytics\nn\task.py`; (3) Set the stride and scale size of the Detect layer in `ultralytics\nn\task.py`; (4) Specify the number of channels for the newly designed distillation network layer in `ultralytics\nn\task.py`; (5) Set the working mode corresponding to the Detect_Distill layer to detect mode in ultralytics\nn\task.py.

[0029] The preferred solution, S1 specifically includes: Design the network sublayer Empty for image data stream separation and the network sublayer Detect_Distill for image data stream fusion; modify the number of channels and step size of the corresponding network layer so that the network can run normally. On the one hand, the performance of the small YOLOv8 model is improved, making it have better application potential in resource-constrained environments. At the same time, it reduces the computing resource consumption of the large YOLOv8 model, making it suitable for a wider range of application scenarios.

[0030] Based on the initial network of yolov8, two special layers are added: the Empty layer is used to store the common image input of the teacher and the student; the Detect_Distill knowledge distillation detection layer is used to summarize the output results of the teacher and the student and output them uniformly. The specific function definitions of these two layers need to be added to the head.py and block.py files respectively.

[0031] The preferred solution is reference Figure 3 Design a schematic diagram for the loss function to show the calculation method of classification, positioning and feature distillation loss. S2 specifically includes: Expand the original sample interval containing only two areas to a sample interval of four areas; Different distillation strategies are implemented in these four areas.

[0032] In a real-time scenario of autonomous driving, the specific implementation steps are as follows: 1. Add and configure in the yolov8 project file ultralytics-main: (1) Create a main.py file as the main program entry to start the training process.

[0033] (2) Create a `Globals` file to define hyperparameters such as learning rate, batch size, etc., to facilitate modification of hyperparameters.

[0034] (3) Add `yolov8-lite.yaml` and `yolov8-student.yaml` in the `.cfg\models\v8` directory. These are the configuration files for the distillation model and the student model, respectively, which are used to specify the knowledge distillation network architecture during training.

[0035] (4) Download and add the `yolov8n.pt` and `yolov8s.pt` files as the pre-trained versions of YOLOv8, which act as teacher models.

[0036] 2. Dataset configuration update: (1) Download the COCO2017 training set; or use other self-built or public data sets in the field of transportation; (2) Modify `.cfg\datasets\coco.yaml` in the yolov8 project file ultralytics-main and specify the folder address of the COCO training set.

[0037] 3.Module addition and update: (1) Add a new module `Empty` in `ultralytics\nn\modules\block.py`, which does not perform any operations.

[0038] (2) Introduce distillation hyperparameters in `ultralytics\nn\modules\head.py` and add new modules `Detect_Teacher`, `Detect_Student` and `Detect_Distill`. The structure of the `Detect_Teacher` and `Detect_Student` layers is exactly the same as that of `Detect`, in order to distinguish the two; the `Detect_Distill` layer is responsible for aggregating and outputting the data of the first two.

[0039] 4. Module reference and distillation parameter configuration: (1) Add the definition of the new module in `ultralytics\nn\modules\__init__.py` to ensure that they can be referenced by other modules.

[0040] (2) Add a reference to the new module in ultralytics\nn\task.py, set the stride and scale of the Detect layer, specify the number of channels for the newly designed distillation network layer, and set the working mode corresponding to the Detect_Distill layer to detect mode.

[0041] 5. Loss function design and distillation hyperparameter introduction: Introduce distillation hyperparameters in `ultralytics\utils\loss.py` and write a loss function suitable for distillation mode.

[0042] The preferred solution, yolov8's positive sample allocation method and loss function are as follows: (1) The positive sample allocation method of yolov8 is determined by the TaskAlignedAssigner function of tal.py (the file is in the \ultralytics\utils folder). The specific principle is as follows: Assume that s is the score of a sample point with respect to the true label, u is the complete intersection-over-union ratio of the sample point and the true value prediction box, α and β are the set hyperparameters, which are 0.5 and 0.6 respectively, then the positive sample weight t of the sample is: The positive sample weight t is calculated for all samples. After further screening, a limited number of samples are taken as positive samples, and the remaining samples are uniformly taken as negative samples.

[0043] (2) The loss function of yolov8 is determined by the v8DetectionLoss function of loss.py (the file is also in the \ultralytics\utils folder). The loss function of YOLOv8 is designed as follows: The entire loss function is divided into three parts, namely classification loss, prediction box loss and feature point loss. Assume that in the final output of the network, S_cls is the classification result of all samples, gt_cls is the true value of all samples, gt_cls_p is also the true value of only positive samples, BCE() is the binary cross entropy calculation formula, sum() represents the sum of the input on its sample space, then the classification loss loss_cls is as follows: Assuming that in the final output of the network, S_box_p is the prediction result only about the positive sample, is the true value only about the positive sample, and CIOU() is the complete intersection-over-union calculation formula, then the prediction box loss loss_box is as follows: Assuming that in the final output of the network, S_dfl_p is the dfl calculation result only for positive samples, gt_dfl_p is the true value only for positive samples, and CE() is the cross entropy calculation formula, then the feature point loss loss_dfl is as follows: The final loss function is as follows: Among them, C_cls, C_box, and C_dfl are the set hyperparameters, which are 0.5, 7.5, and 1.5 respectively.

[0044] (3) The initial positive sample allocation and loss function of yolov8 have the following characteristics: the positive sample allocation divides the entire sample space into two regions. In the subsequent loss function calculation, only the loss of the positive sample region is calculated, thereby effectively reducing the parameters involved in the operation during back propagation and achieving efficient training; however, in the knowledge distillation process, since the teacher and student models have their own positive sample allocation methods, the initial positive sample allocation and loss function method cannot be directly used.

[0045] A further solution, the positive sample allocation method and loss function based on knowledge distillation are as follows: (1) The positive sample allocation method of knowledge distillation is based on the positive sample division method of the teacher model and the student model. In the common sample space, the sample set is divided into four regions as follows: Figure 2 shown.

[0046] (2) The knowledge loss function is based on the above division method, and different loss function calculation strategies are adopted for the above four sample intervals, such as Figure 3 shown.

[0047] The preferred solution is that the knowledge distillation loss function design also includes the following three parts: classification loss, prediction box loss and distribution focus loss (dfl).

[0048] 1. Classification loss calculation Assume that S_cls is the student's prediction for the classification after sigmoid and T_cls is the teacher's prediction for the classification after sigmoid. BCE stands for binary cross entropy. gt_cls is the ground truth. Also, similarly assume that S_cls_t is the student's prediction after sigmoid_T and T_cls_t is the teacher's prediction after sigmoid_T. In addition, area_i is the mask of each region. Where sigmoid_T is defined as follows: The loss_classification of region 1 is, where a and b range from 0 to 1: The loss_classification of region 2 is: The loss_classification of region 3 is: The loss_classification of region 4 is: The weights for region 1 are: The weights for region 2 are: The weights for region 3 are: The weights for region 4 are: The sum of these weights is: The loss of the final classification part is: 2. Prediction box loss calculation Assume S_box is the student's prediction of the bounding box and T_box is the teacher's prediction of the bounding box. Here CIOU stands for Complete Intersection over Union. gt_box is still the ground truth.

[0049] The bounding box loss is calculated as follows: The loss_bounding_box for region 1 is: The loss_bounding_box for region 2 is: The loss_bounding_box for region 3 is: The loss of the final prediction box is: 3. Distribution focus loss calculation Assume S_dfl is the student's prediction of dfl, and T_dfl is the teacher's prediction of dfl. Here CE stands for cross entropy. gt_dfl still represents the true value.

[0050] The loss_dfl in region 1 is: The loss_dfl in region 2 is: The loss_dfl in region 3 is: The final loss of the dfl part is: 4. The final loss function of distillation is as follows: Where C_cls, C_box and C_dfl are the coefficients of these parts, which are 0.5, 7.5 and 1.5 respectively. However, the requirements of the region partitioning method change these weights, so the coefficients about these coefficients are given. They are 0.7, 1.1 and 1.0 respectively, namely C1, C2, C3, to balance the various parts. Therefore, the final loss should be: .

[0051] The embodiment of the present invention further provides a system for rapid image recognition based on the yolov8 architecture, which can be used to implement the rapid image recognition method based on the yolov8 architecture. The system includes: An architecture modification module is configured to modify the serial network architecture of yolov8 into a parallel network architecture, so that the image data passes through the network of the teacher model and the student model during forward propagation; The positive sample allocation module is configured as a positive sample allocation method based on yolov8, which combines the positive sample intervals of the student model and the teacher model into a unified positive sample interval; A loss function design module, configured to design a loss function to obtain an optimized knowledge distillation model; The output module is configured to quickly output image recognition results after training based on the optimized knowledge distillation model.

[0052] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A fast image recognition method based on yolov8 architecture, characterized in that: The following steps are involved: S1, the serial network architecture of yolov8 is modified into a parallel network architecture, so that the image data passes through the network of the teacher model and the student model during forward propagation; S2, based on the positive sample allocation method of yolov8, combines the positive sample intervals of the student model and the teacher model into a unified positive sample interval; S3, design the loss function to obtain the optimized knowledge distillation model; S4, after training based on the optimized knowledge distillation model, quickly outputs the image recognition results.

2. The image rapid recognition method based on yolov8 architecture according to claim 1 is characterized in that: The S1 specifically includes: Design the network sublayer Empty for image data stream separation and the network sublayer Detect_Distill for image data stream fusion; Modify the number of channels and step size of the corresponding network layer so that the network can run normally.

3. The image rapid recognition method based on yolov8 architecture according to claim 1 is characterized in that: The S2 specifically includes: Expand the original sample interval containing only two areas to a sample interval of four areas; Different distillation strategies are implemented in these four areas.

4. The image rapid recognition method based on yolov8 architecture according to claim 1 is characterized in that: The S3 specifically includes: The loss function design includes the sum of parts, namely classification loss loss_cls, prediction box loss loss_box and distribution focus loss loss_dfl; Where C_cls, C_box and C_dfl are the coefficients of these parts, and the optimal solutions in engineering practice are 0.7, 1.1 and 1.0 respectively.

5. The image rapid recognition method based on yolov8 architecture according to claim 4 is characterized in that: The classification loss is calculated as follows: Assume S_cls is the student's prediction for the classification after sigmoid, T_cls is the teacher's prediction for the classification after sigmoid, BCE stands for binary cross entropy, gt_cls is the ground truth, S_cls_t is the student's prediction after sigmoid_T, and T_cls_t is the teacher's prediction after sigmoid_T, area_i is the mask of each region, where sigmoid_T is defined as follows: The loss_classification of region 1 is, where a and b range from 0 to 1: The loss_classification of region 2 is: The loss_classification of region 3 is: The loss_classification of region 4 is: The weights for region 1 are: The weights for region 2 are: The weights for region 3 are: The weights for region 4 are: The sum of these weights is: The loss of the final classification part is: 。 6. The image rapid recognition method based on yolov8 architecture according to claim 4 is characterized in that: The prediction box loss is calculated as follows: Assume S_box is the student's prediction of the bounding box and T_box is the teacher's prediction of the bounding box, where CIOU stands for Complete Intersection Over Union, and gt_box is still the ground truth; The bounding box loss is calculated as follows: The loss_bounding_box for region 1 is: The loss_bounding_box for region 2 is: The loss_bounding_box for region 3 is: The loss of the final prediction box is: 。 7. The image rapid recognition method based on yolov8 architecture according to claim 4 is characterized in that: The distributed focal loss is calculated as follows: Assume S_dfl is the student's prediction of dfl, and T_dfl is the teacher's prediction of dfl. Here CE stands for cross entropy. gt_dfl still represents the true value. The loss_dfl in region 1 is: The loss_dfl in region 2 is: The loss_dfl in region 3 is: The final loss of the dfl part is: 。 8. The image rapid recognition method based on yolov8 architecture according to claim 1 is characterized in that: The S3 specifically includes: For the two sample intervals of teachers and students respectively, the sample intervals are divided into four areas, so that different loss function calculation strategies are adopted respectively; the essence is to introduce the results of positive sample allocation of the teacher model as guidance information, so as to better guide training.

9. A fast image recognition system based on yolov8 architecture, characterized in that: The system can be used to implement any one of the above-mentioned methods for rapid image recognition based on the yolov8 architecture in claims 1 to 8, and the system comprises: An architecture modification module is configured to modify the serial network architecture of yolov8 into a parallel network architecture, so that the image data passes through the network of the teacher model and the student model during forward propagation; The positive sample allocation module is configured as a positive sample allocation method based on yolov8, which combines the positive sample intervals of the student model and the teacher model into a unified positive sample interval; A loss function design module, configured to design a loss function to obtain an optimized knowledge distillation model; The output module is configured to quickly output image recognition results after training based on the optimized knowledge distillation model.

Images