Image detection method and device, electronic device and storage medium

Abstract

The invention relates to an image detection method and device, an electronic device and a storage medium, and the method comprises the steps: obtaining a to-be-processed image; inputting a to-be-processed image into the trained multi-task detection network model, and performing feature extraction processing on the to-be-processed image through a transfer learning network of the multi-task detection network model to obtain general features; wherein the transfer learning network is obtained by training based on a pre-trained basic neural network; utilizing a plurality of independent task branches in a multi-task target detection network of the multi-task detection network model to respectively perform task branch feature extraction on the general features, and outputting a plurality of task branch detection results; and obtaining a target detection result based on the multiple task branch detection results. Through the method and the device, the problems of high labor cost, difficulty in model iteration and recognition errors in a complex scene in product display detection are solved, and an efficient, flexible and high-precision automatic solution is provided for product display detection.

Description

Technical Field

[0001] The present application relates to the field of target detection, and in particular to an image detection method, device, electronic device and storage medium. Background Art

[0002] With the retail industry's growing demand for standardized brand image, efficient and accurate automated verification of product display compliance within retail stores has become a key issue. In the tobacco industry, in particular, brands use display competitions and other initiatives to encourage retailers to display products in accordance with standards. Traditional manual review methods are inefficient and costly, making them unable to meet dynamic regulatory requirements.

[0003] Currently, the process for display inspection in retail stores is generally as follows: First, a large number of display photos are collected; second, these photos are carefully annotated manually to clearly identify product placement, brand, and other information; third, image enhancement techniques are used to further enrich the sample diversity; and finally, an image classification model is trained based on these processed images. This process alleviates the time-consuming and labor-intensive nature of manual review to a certain extent, but it also has significant limitations. During the data preparation phase, a significant amount of manpower is required to collect and annotate images. Because annotation involves subjective judgment, differences in the cognitive abilities of annotators can lead to inconsistent annotation standards, which in turn affects the accuracy of subsequent model training. Furthermore, traditional image classification models lack flexibility in the way they label categories. Adding or removing categories requires re-annotation and model training, which is not only time-consuming and labor-intensive, but also increases the difficulty of model iteration. Furthermore, traditional models often struggle to accurately identify the complex and ever-changing retail store display scenarios, resulting in frequent misclassifications and omissions. This not only affects the accuracy of display inspection but also creates unnecessary inconvenience for users.

[0004] Currently, no effective solution has been proposed to address the problems of high labor costs, difficulty in model iteration, and recognition errors in complex scenarios in related technologies for product display inspection. Summary of the Invention

[0005] The embodiments of the present application provide an image detection method, device, electronic device, and storage medium to at least address the problems of high labor costs, difficult model iteration, and recognition errors in complex scenarios in related technologies in product display detection.

[0006] In a first aspect, an embodiment of the present application provides an image detection method, comprising:

[0007] Get the image to be processed;

[0008] Inputting the image to be processed into a trained multi-task detection network model, and performing feature extraction processing on the image to be processed via a transfer learning network of the multi-task detection network model to obtain common features; wherein the transfer learning network is trained based on a pre-trained basic neural network;

[0009] Utilizing multiple independent task branches in the multi-task target detection network of the multi-task detection network model, respectively performing task branch feature extraction on the common features, and outputting multiple task branch detection results;

[0010] Based on the detection results of the multiple task branches, a target detection result is obtained.

[0011] In some embodiments, obtaining an image to be processed includes:

[0012] Get the initial image;

[0013] Performing standardization and normalization processing on the initial image to obtain a standard image with a size of preset standard pixels;

[0014] Performing a data enhancement operation on the standard image to obtain the image to be processed.

[0015] In some embodiments, performing a data augmentation operation on the standard image to obtain the image to be processed includes:

[0016] Performing a data augmentation operation on the standard image to obtain an enhanced image;

[0017] A deduplication operation is performed on the enhanced image to obtain the image to be processed.

[0018] In some embodiments, the training process of the multi-task detection network model includes:

[0019] Get training images and object detection tasks;

[0020] Freezing all convolutional layers in the pre-trained basic neural network, and after freezing, replacing the original classification head in the basic neural network with a newly added classification head adapted to the target detection task based on the target detection task, to obtain a training transfer learning network;

[0021] The training image is input into the training transfer learning network, and the training transfer learning network is trained using a preset loss function to obtain the multi-task detection network model.

[0022] In some embodiments, inputting the training image into the training transfer learning network, and training the training transfer learning network using a preset loss function to obtain the multi-task detection network model includes:

[0023] Inputting the training image into the training transfer learning network, and performing preliminary training on the training transfer learning network using a preset loss function to obtain a preliminary training model;

[0024] Determining a number of high-level convolutional layers in the preliminary training model;

[0025] Based on a preset dynamic learning rate adjustment strategy, fine-tune the high-level convolutional layer and the newly added classification head to obtain the high-level weights corresponding to the high-level convolutional layer and the classification head weights corresponding to the newly added classification head;

[0026] Based on the high-level weights and the classification head weights, the multi-task detection network model based on transfer learning is obtained.

[0027] In some embodiments, the training process of the multi-task detection network model further includes:

[0028] Based on the acquired object detection tasks, multiple training task branches are set up in the multi-task object detection training network;

[0029] Assigning a corresponding branch loss function to each of the training task branches, and calculating a loss calculation result corresponding to each of the training task branches based on each of the branch loss functions;

[0030] Based on the loss calculation results, the branch weights corresponding to the training task branches are iteratively updated to obtain the trained multi-task object detection network.

[0031] In some embodiments, the step of iteratively updating the branch weights corresponding to the training task branches based on the loss calculation results to obtain the trained multi-task object detection network includes:

[0032] Inputting the loss calculation results into the weighted total loss function to obtain the total loss result;

[0033] The branch weights are iteratively updated based on the total loss result to obtain the trained multi-task object detection network.

[0034] In a second aspect, an embodiment of the present application provides an image detection device, comprising:

[0035] An image acquisition module, used for acquiring an image to be processed;

[0036] A transfer learning module is configured to input the image to be processed into a trained multi-task detection network model, and perform feature extraction on the image to be processed via the transfer learning network of the multi-task detection network model to obtain common features; wherein the transfer learning network is trained based on a pre-trained basic neural network;

[0037] A multi-task detection module is used to utilize multiple independent task branches in the multi-task target detection network of the multi-task detection network model to perform task branch feature extraction on the common features respectively, and output multiple task branch detection results;

[0038] A result generation module is used to obtain a target detection result based on the detection results of the multiple task branches.

[0039] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the image detection method as described in the first aspect above when executing the computer program.

[0040] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the image detection method as described in the first aspect above is implemented.

[0041] Compared with related technologies, the image detection method, device, electronic device and storage medium provided in the embodiments of the present application solve the problems of high labor costs, difficult model iteration and recognition errors in complex scenarios in product display detection through the technical collaboration of transfer learning and multi-task learning, providing an efficient, flexible and high-precision automated solution for product display detection.

[0042] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 1 is a hardware structure block diagram of a terminal of an image detection method according to an embodiment of the present invention;

[0045] Figure 2 is a flowchart of an image detection method according to an embodiment of the present application;

[0046] Figure 3is a flow chart of an image detection method according to a preferred embodiment of the present application;

[0047] Figure 4 This is a structural diagram of the EfficientNet-B3 neural network according to the preferred embodiment of the present application;

[0048] Figure 5 is a training loss curve diagram of the image detection method according to the preferred embodiment of the present application;

[0049] Figure 6 and Figure 7 is a schematic diagram of the detection results of the image detection method according to the preferred embodiment of the present application;

[0050] Figure 8 It is a structural block diagram of an image detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means and should not be understood as the contents disclosed in the present application being insufficient.

[0052] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0053] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application means greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0054] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 FIG is a block diagram of the hardware structure of the terminal of the image detection method according to an embodiment of the present invention. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0055] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image detection method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0056] The transmission device 106 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by the terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0057] This embodiment provides an image detection method. Figure 2 is a flow chart of an image detection method according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0058] Step S201, obtaining an image to be processed;

[0059] The process involves collecting display image data from retail stores using image acquisition devices (such as cameras and scanners) or storage devices (such as hard drives and cloud storage). The image data can be digital photos, video frames, or other forms of image information. The collected image information covers different display methods (such as brand placement and promotional material layout), store layouts (such as shelf structure and background environment), and lighting conditions. This image information undergoes a series of preprocessing steps to obtain the image to be processed. In this step, the image to be processed serves as the input object for the image detection solution, which is the basis for subsequent feature extraction and target detection.

[0060] Step S202: Input the image to be processed into the trained multi-task detection network model, and perform feature extraction on the image to be processed via the transfer learning network of the multi-task detection network model to obtain common features; wherein the transfer learning network is trained based on the pre-trained basic neural network;

[0061] Among them, the transfer learning network is obtained by further training based on the pre-trained basic neural network (such as convolutional neural network CNN, EfficientNet-B3, etc.). The image to be processed is first input into the transfer learning network. The transfer learning network extracts features from the image through its convolutional layer, pooling layer and other structures to generate universal features that can represent the image content. These universal features are abstract representations of the high-level semantic information of the image and are used for subsequent task branch detection. In this step, the transfer learning network reduces the training time and computing resource requirements for new tasks by utilizing the powerful feature extraction capabilities of the pre-trained model. At the same time, through transfer learning, the model can better adapt to image detection tasks in different fields and improve the generalization ability and detection accuracy of the model.

[0062] Step S203, using multiple independent task branches in the multi-task target detection network of the multi-task detection network model, respectively extracting task branch features from the common features, and outputting multiple task branch detection results;

[0063] The multi-task object detection network consists of multiple independent task branches, each responsible for handling a specific detection task (such as object classification, object localization, attribute recognition, etc.). These task branches further extract and process common features and ultimately output their own task branch detection results. These detection results can be the target's category label, location coordinates, attribute information, etc. For example, the objectives of multiple task branches can be defined as:

[0064] Task 1: Liqun brand detection (multi-classification);

[0065] Task 2: Promotional material detection (two-classification);

[0066] Task 3: Competitive product detection (binary classification / multi-classification);

[0067] Task 4: Compliance judgment (two-classification).

[0068] The architecture of the corresponding multi-task object detection network includes a shared backbone part (the transfer learning network serves as a shared feature extraction layer and outputs common features) and independent branch parts (each task is equipped with an independent fully connected layer and outputs the task results separately).

[0069] This step can process multiple detection tasks simultaneously through multi-task learning, improving the efficiency and practicality of the model. By sharing common features, different task branches can promote each other and improve the detection accuracy and robustness of the model. At the same time, multi-task learning also helps to alleviate the overfitting problem and improve the generalization ability of the model.

[0070] Step S204: Obtain target detection results based on the detection results of multiple task branches.

[0071] More accurate target detection results are obtained by fusing the detection results of different task branches. Fusion methods can include weighted averaging, voting decisions, Bayesian reasoning, etc. The final target detection results can include complete information such as the target category, location, and attributes. This step comprehensively analyzes and processes the detection results of multiple task branches to obtain more accurate and comprehensive target detection results, improving the practicality and accuracy of the image detection solution and providing strong support for subsequent image processing and analysis.

[0072] Through the above steps, compared with the traditional product display detection solution that requires manual labeling of massive images (such as brand location, competitor product identification), the labeling standards are prone to deviations, and data cleaning and balancing require additional investment. The transfer learning of this application solves the problems of data dependence and training efficiency, and adopts a pre-trained basic neural network as the feature extraction backbone network. Its pre-trained weights already have the ability to extract common visual features (edges, textures, shapes), reducing the dependence on labeled data. It only needs to fine-tune the basic neural network for the output classification head and the high-level network, and then the generalization of the pre-trained features can be used to quickly adapt to new tasks. Compared with traditional solutions, the amount of training data is reduced, the training time is shortened, and the destruction of the underlying common features is avoided. Compared with the traditional product display detection scheme, the addition of new detection categories (such as adding new brands) requires re-labeling of data and retraining of models, which is a time-consuming and inflexible process. The multi-task learning of this application improves the adaptability to complex scenarios and the flexibility of the model. The common features extracted by the transfer learning network are reused by all task branches (brand detection, material detection, and competitor detection) to reduce redundant calculations, and each task branch is independently equipped with a dedicated output layer (such as a multi-classification layer for brand detection and a binary classification layer for compliance judgment) to support differentiated classification logic. Multi-task correlation modeling significantly improves the detection accuracy of complex scenarios (such as competitors blocking brands), and new tasks (such as adding new brands) only require expanding branches without reconstructing the model. It can be seen that this application solves the problems of high labor costs, difficult model iterations, and recognition errors in complex scenarios in product display detection through the technical collaboration of transfer learning and multi-task learning, providing an efficient, flexible and high-precision automation solution for product display detection.

[0073] In some embodiments, obtaining an image to be processed includes:

[0074] Get the initial image;

[0075] Performing standardization and normalization on the initial image to obtain a standard image with a size of preset standard pixels;

[0076] Perform data augmentation on the standard image to obtain the image to be processed.

[0077] Among them, the initial image is obtained by taking a picture with a photographic device (such as a camera or a mobile phone) or obtained from an existing image database, and the initial image includes various scenes of product display in a retail store. Since the input of the model usually requires a fixed size, it is necessary to uniformly adjust all collected images to the input size required by the model. In this embodiment, the collected subject scene pictures are converted into preset standard pixels (such as 680*680 pixels). At the same time, in order to maintain the original image ratio, the shorter sides are padded after adjustment to prevent image distortion. According to the requirements of the pre-trained model, the pixel values of the image are normalized. Normalization usually scales the pixel values to a specific range (such as between [0,1] or between [-1,1]). For example, the image is standardized using mean (mean) = [0.485, 0.456, 0.406] and std (variance) = [0.229, 0.224, 0.225]. Data enhancement is used to increase sample diversity and prevent model overfitting, which is particularly useful when the amount of data is small. A variety of data enhancement methods can be used, including random rotation, random scaling and cropping, brightness and contrast adjustment, noise addition, and affine transformation. These operations can simulate different shooting angles, distances, and lighting conditions, thereby improving the model's adaptability to different scenes. The data enhancement operation in this embodiment reduces the risk of overfitting of the model during training by increasing sample diversity. The data enhancement operation reduces the risk of overfitting of the model during training by increasing sample diversity. Through a series of preprocessing operations, it provides high-quality data input for subsequent transfer learning and multi-task learning, which helps to improve the final detection accuracy.

[0078] In some embodiments, performing a data augmentation operation on a standard image to obtain an image to be processed includes:

[0079] Perform data augmentation on the standard image to obtain an enhanced image;

[0080] Perform a deduplication operation on the enhanced image to obtain an image to be processed.

[0081] Among them, data enhancement is an important means to improve the generalization ability and robustness of the model. Various data enhancement techniques can be used to process standard images, including but not limited to:

[0082] Random rotation: Rotate the image by a random angle (e.g., -15 to 15 degrees) to simulate different shooting angles;

[0083] Random scaling and cropping: Randomly scale the image (e.g., 0.5x scaling) and then randomly crop it to a fixed size to simulate the effects of shooting at different distances.

[0084] Brightness and contrast adjustment: Randomly adjust the brightness and contrast of the image to enhance the model's robustness to images under different lighting conditions;

[0085] Noise addition: Add random noise (such as Gaussian noise) to the image to increase image diversity and simulate image quality issues during shooting;

[0086] Affine transformation: includes geometric transformations such as translation, scaling, rotation, and tilt to increase the complexity of the image.

[0087] Through these data enhancement techniques, multiple enhanced images can be obtained, which together constitute the enhanced images for subsequent model training. During the data enhancement process, some duplicate or highly similar images may be generated. This redundant data not only increases the training burden of the model, but may also cause the model to overfit to these duplicate data. Therefore, a hashing algorithm or other deduplication technology is used to deduplicate the enhanced image set. After removing duplicate or highly similar images, the remaining images are the images to be processed. The data enhancement in this embodiment increases sample diversity, enabling the model to learn feature representations in more different situations, thereby improving generalization ability. The deduplication operation avoids the risk of the model overfitting to duplicate data during training. After removing redundant data, the amount of training data for the model is reduced, thereby improving training efficiency, reducing the use of redundant data, optimizing the utilization of computing resources, and reducing storage and computing costs. In addition, the images processed by data enhancement can better simulate changes in actual scenes, making the model more robust in practical applications. Through data enhancement and deduplication operations, the quality and diversity of the images to be processed are effectively improved, providing a higher-quality data foundation for subsequent transfer learning and multi-task learning, thereby helping to improve the overall performance of the model and practical application effects.

[0088] In some embodiments, the training process of the multi-task detection network model includes:

[0089] Get training images and object detection tasks;

[0090] Freeze all convolutional layers in the pre-trained basic neural network. After freezing, replace the original classification head in the basic neural network with a new classification head adapted to the target detection task based on the target detection task to obtain a training transfer learning network.

[0091] The training images are input into the training transfer learning network, and the training transfer learning network is trained using the preset loss function to obtain a multi-task detection network model.

[0092] Collect a large number of retail store display photos as training images. These images should cover various display styles, store layouts, and brand placement scenarios. At the same time, define the target detection task, such as detecting the location of Liqun brand cigarettes, the presence of competing cigarettes, the correct placement of promotional materials, and whether display photos meet standards. Select a pre-trained, high-performance, and efficient basic neural network, such as EfficientNet-B3. Freeze all convolutional layers of the basic neural network, preserving their weights. These layers extract low-level and mid-level features from the image, which are universal across different tasks. Based on the requirements of the target detection task, replace the original classification head with a new one. For example, in the case of themed display scene detection, different cigarette series are classified into display compliance and display non-compliance, using a sigmoid activation function for binary classification. Input the training images into the training transfer learning network, perform forward propagation through the new classification head, and obtain predictions. Based on the predictions and the true labels of the training images, calculate the value of a pre-set loss function (such as the binary cross-entropy loss function for binary classification). Use an optimizer (such as the Adam optimizer) to backpropagate and update the parameters of the training transfer learning network according to the loss function value. During the training process, a phased training strategy is adopted, that is, the newly added classification head is trained first, and after it converges, some convolutional layers are gradually unfrozen for fine-tuning to further improve the model performance. This embodiment uses transfer learning to utilize the weights of the pre-trained basic neural network, which can significantly reduce the training time of new tasks, reduce the dependence on large amounts of labeled data, and reduce labor costs.

[0093] In some embodiments, a training image is input into a training transfer learning network, and the training transfer learning network is trained using a preset loss function to obtain a multi-task detection network model, including:

[0094] Input the training image into the training transfer learning network, use the preset loss function to perform preliminary training on the training transfer learning network, and obtain a preliminary training model;

[0095] Determine several high-level convolutional layers in the preliminary training model;

[0096] Based on the preset dynamic learning rate adjustment strategy, fine-tune the high-level convolutional layer and the newly added classification head to obtain the high-level weights corresponding to the high-level convolutional layer and the classification head weights corresponding to the newly added classification head;

[0097] Based on the high-level weights and classification head weights, a multi-task detection network model based on transfer learning is obtained.

[0098] In the first phase of multi-task detection network model training, training images are fed into a transfer learning network built on a pre-trained base neural network (e.g., EfficientNet-B3). The convolutional layers are frozen, and only the newly added classification head is trained. A preset loss function (e.g., binary cross-entropy loss) is used to calculate the difference between the predicted results and the true labels. The weights of the newly added classification head are then updated via backpropagation. After multiple iterations of training, a preliminary trained model is obtained. Based on this preliminary trained model, the network identifies high-level convolutional layers that require fine-tuning, such as the unfrozen 30% of high-level convolutional layers (responsible for high-level semantic features). These high-level convolutional layers are typically responsible for extracting higher-level features and are crucial for adapting to new tasks. During the fine-tuning training phase, the selected high-level convolutional layers are unfrozen and trained together with the newly added classification head. A dynamic learning rate adjustment strategy (e.g., Learning Rate Scheduler) is used to dynamically adjust the learning rate based on training performance to accelerate convergence and avoid overfitting. The weights of the high-level convolutional layers and the newly added classification head are updated over multiple iterations of training. After fine-tuning training, the high-level weights corresponding to the high-level convolutional layers and the classification head weights corresponding to the newly added classification head are combined to form the final multi-task detection network model. For example, in the first stage (classification head training): all convolutional layers are frozen and only the newly added classification head is trained; the optimizer uses Adam, the initial learning rate is set to 1e-5, and the loss function is the cross-entropy corresponding to the task (such as Binary Cross-Entropy for compliance judgment). In the second stage (fine-tuning the high-level layers): 30% of the convolutional layers (responsible for high-level semantic features) are unfrozen and the entire network is fine-tuned; using dynamic learning rate scheduling (such as cosine annealing), the learning rate is gradually reduced to 1e-6 to balance pre-trained features with task specificity.

[0099] In this embodiment, the model can better adapt to new tasks and improve detection accuracy by combining preliminary training and fine-tuning training; the dynamic learning rate adjustment strategy helps the model converge quickly during training and reduce training time; fine-tuning training not only optimizes the performance on specific tasks, but also helps the model learn more generalized feature representations and improves generalization capabilities in practical applications; by determining and fine-tuning high-level convolutional layers, the model structure can be flexibly adjusted according to actual task requirements to improve the adaptability of the model; through reasonable model training strategies (such as staged training, dynamic learning rate adjustment, etc.), the risk of model overfitting is effectively reduced.

[0100] In some embodiments, the training process of the multi-task detection network model further includes:

[0101] Based on the acquired object detection tasks, multiple training task branches are set up in the multi-task object detection training network;

[0102] Assign a corresponding branch loss function to each training task branch, and calculate the loss calculation result corresponding to each training task branch based on each branch loss function;

[0103] Based on the loss calculation results, the branch weights corresponding to each training task branch are iteratively updated to obtain a trained multi-task object detection network.

[0104] Multiple training task branches are established based on the specific needs of product display inspection, such as inspecting the location of Liqun brand cigarettes, the placement of promotional materials, the presence of competing cigarettes, and the overall compliance of the display. Each training task branch focuses on solving a specific inspection task, and by sharing the underlying feature extraction network (i.e., the training transfer learning network), the synergistic effect of multi-task learning is achieved. An appropriate loss function is selected for each training task branch. For example, for binary classification tasks (such as compliance judgment), the binary cross-entropy loss function is used; for multi-classification tasks (such as brand detection), the categorical cross-entropy loss function is used. Each branch loss function is used to calculate the difference between the branch's prediction result and the true label. The training image is input into the multi-task object detection training network, and the prediction results of each branch are obtained through forward propagation. The corresponding branch loss function is used to calculate the loss value of each branch. These loss values reflect the performance of the model on each task. Based on the loss calculation results of each branch, the weights of each branch are updated through the backpropagation algorithm. In the iterative update process, a dynamic weight allocation strategy is adopted to adjust the weight of each branch according to its performance in the training process to ensure the balance and effectiveness of multi-task learning. After multiple iterative training, a trained multi-task target detection network is obtained. This embodiment sets up multiple training task branches and assigns appropriate loss functions and weights to each branch, which can fully utilize the synergistic effect of multi-task learning and improve the overall detection effect. Multi-task learning enables the model to learn more generalized feature representations by sharing the underlying feature extraction network, thereby improving adaptability and robustness in different scenarios. By assigning different weights and loss functions to each branch, the model structure and training strategy can be flexibly adjusted according to actual task requirements, thereby improving the adaptability and flexibility of the model. By assigning different weights and loss functions to each branch, the model structure and training strategy can be flexibly adjusted according to actual task requirements, thereby improving the adaptability and flexibility of the model and achieving more flexible model adjustment.

[0105] In some embodiments, based on the loss calculation results, the branch weights corresponding to the training task branches are iteratively updated to obtain a trained multi-task object detection network, including:

[0106] Input the calculation results of each loss into the weighted total loss function to obtain the total loss result;

[0107] The branch weights are iteratively updated based on the total loss results to obtain a trained multi-task object detection network.

[0108] During the multi-task object detection training process, each training task branch will calculate a loss value, which reflects the performance of the model on each task. In order to comprehensively evaluate the overall performance of the model, the loss values of each branch need to be input into a weighted total loss function. The weighted total loss function performs a weighted summation of the loss values of each branch according to the preset branch weights to obtain the total loss result. The formula of the weighted total loss function is usually:

[0109] Total loss = Σ(branch weight i × branch loss i), where i represents different training task branches.

[0110] For example, define the weighted total loss function: L = α1*L 品牌 +α2*L 物料 +α3*L 竞品 +α4*L 合规 ;

[0111] Among them, L represents the total loss; L 品牌 represents the loss of the Liqun brand detection task branch; L 物料 Indicates the loss of the propaganda material detection task branch; L 竞品 represents the loss of the competitor detection task branch; L 合规 Indicates the loss of the compliance judgment task branch.

[0112] Using the backpropagation algorithm, the parameters of the multi-task object detection network are updated according to the calculated total loss result, including updating the weights of each branch (such as the weights of the convolutional layer and the weights of the classification head) and adjusting the branch weights themselves. In the iterative update process, the adjustment of branch weights is a key step. If the loss value of a branch remains high during training, it means that the model performs poorly on this task. At this time, the weight of the branch can be appropriately increased to give it more attention in subsequent training. Conversely, if the loss value of a branch is low, its weight can be appropriately reduced to avoid the branch dominating the training process. Through multiple iterative training, the model gradually converges. When the total loss result reaches a smaller value, the model is considered to have been trained. At this time, the multi-task object detection network obtained can perform well on all tasks.

[0113] This embodiment uses a weighted total loss function to perform a weighted summation of the loss values of each branch, which can ensure the balance in the multi-task learning process and avoid the situation where a certain task dominates the training process, resulting in a decline in the performance of other tasks; by dynamically adjusting the branch weights, the model can converge faster during the training process. When the performance of a branch improves slowly, increasing its weight can prompt the model to invest more training resources in the branch, thereby accelerating its convergence speed; by comprehensively considering the loss values of each branch for model updating, the model can perform better on all tasks. This not only improves the accuracy of the model, but also enhances the generalization ability and robustness of the model; the dynamic adjustment strategy of branch weights enables the model to be flexibly adjusted according to actual performance during training. This flexibility helps the model better adapt to different task scenarios and data distributions.

[0114] The embodiments of the present application are described and illustrated below through preferred embodiments.

[0115] Figure 3 is a flow chart of an image detection method according to a preferred embodiment of the present application, such as Figure 3 As shown, the image detection method includes the following steps:

[0116] Step S301: prepare data.

[0117] Collect photos of retail store displays, covering various display styles, store layouts, and brand placement. Also, perform data annotation, focusing on the location of Liqun brand cigarettes, the presence of competing cigarettes, the correct placement of promotional materials, and whether the display photos meet standards.

[0118] Step S302: Data preprocessing:

[0119] In order to ensure data quality and consistency during model training, data preprocessing mainly includes image preprocessing, data enhancement, and data segmentation.

[0120] Image preprocessing: Image size standardization. Since the model input usually requires a fixed size, it is recommended to uniformly adjust all collected images to the input size required by the model. The collected subject scene images are converted to 680*680. At the same time, the original image ratio can be maintained. After adjustment, the shorter sides are padded to prevent image distortion. At the same time, the image is normalized. According to the requirements of the pre-trained model, the pixel values of the image are normalized (usually between [0,1] or [-1,1]). The image is standardized using mean (mean) = [0.485, 0.456, 0.406] and std (variance) = [0.229, 0.224, 0.225].

[0121] Data augmentation: Data augmentation is used to increase sample diversity and prevent overfitting, which is particularly useful when the amount of data is small. The following data augmentation techniques are used to process standard images:

[0122] Random Rotation: Rotate the image by a random angle (-15 to 15 degrees) to simulate different shooting angles.

[0123] Random scaling and cropping: The image is randomly scaled (0.5x scaling) and then randomly cropped to a fixed size to simulate the effects of shooting at different distances.

[0124] Brightness and contrast adjustment: Randomly adjust the brightness and contrast of the image to enhance the robustness of the model to images under different lighting conditions.

[0125] Noise addition: Add random noise, such as Gaussian noise, to the image to increase image diversity and simulate image quality issues during shooting.

[0126] Affine transformation: includes geometric transformations such as translation, scaling, rotation, and tilt to increase the complexity of the image.

[0127] Remove redundant data and low-quality images: Screen the image library and remove blurry, unclear, or unrecognizable photos to ensure that the images input to the model are of sufficient quality.

[0128] Duplicate data processing: If there are duplicate or highly similar images, it is recommended to remove redundant images through methods such as hashing algorithms to prevent overfitting to these duplicate data during model training.

[0129] Data balancing problem: Dealing with data imbalance problem: If the number of certain categories (such as samples with non-compliant display) is too small, it may be necessary to balance the data by oversampling or undersampling, or use upsampling techniques (such as SMOTE) to generate more samples of minority categories.

[0130] Step S303, transfer learning task:

[0131] EfficientNet-B3 was chosen as the original model for migration. EfficientNet-B3 is a high-performance and efficient convolutional neural network with pre-trained weights on ImageNet, enabling excellent image feature extraction. EfficientNet is a highly efficient convolutional neural network architecture that uses a novel compound scaling method to simultaneously optimize the network's depth, width, and resolution, significantly improving model performance while reducing computational effort. Compared to smaller models, EfficientNet-B3 achieves better accuracy while maintaining high computational efficiency, making it an ideal choice for high-quality image classification in resource-constrained environments. Compound Scaling is employed to collaboratively optimize the network's depth, width, and resolution across multiple layers. EfficientNet effectively controls network complexity while maintaining classification accuracy.

[0132] Figure 4 This is a structural diagram of the EfficientNet-B3 neural network according to the preferred embodiment of the present application. Figure 4 As shown, the first step in transfer learning is to freeze some of the pre-trained layers of EfficientNet-B3. Common freezing methods include freezing the initial layers. A common practice in transfer learning is to freeze the first few layers of the model, keeping their weights unchanged. These layers are typically responsible for extracting lower-level features (such as edges and textures) and are universal across different tasks. Unfreezing some of the higher layers: These layers are typically responsible for extracting higher-level features (such as object shape or context). Therefore, the higher layers of EfficientNet can be unfrozen according to the requirements of the new task, allowing them to be adapted to the display detection task.

[0133] In the themed scene display detection scenario, 70% of the initial layers of EfficientNet-B3 are frozen, while the higher layers remain trainable, allowing for adaptation to the display detection task. This also requires a customized classification head. The last layer of the pre-trained EfficientNet-B3 model is typically a fully connected layer. Since the themed display detection task is a scenario-specific image classification problem, this classification head needs to be modified or replaced based on the task's requirements.

[0134] Specific steps: Remove the last fully connected layer of EfficientNet and add a new classification head; Global average pooling layer: The output of the convolution layer is pooled through global average pooling to reduce the number of parameters and prevent overfitting; Fully connected layer: Add a custom fully connected layer. The number of output nodes depends on the number of categories of the classification task. In the theme display scene detection, different cigarette series are divided into display compliance and display non-compliance, so the final activation function is changed to sigmoid for binary classification of different cigarette series. The Sigmoid formula is as follows:

[0135]

[0136] Among them, σ(z) represents the probability that a certain type of cigarette is stored in compliance with regulations. If it is greater than 0.5, it means compliance, and if it is less than 0.5, it means non-compliance. Its value range is (0, 1); z represents the calculated value of all eigenvectors.

[0137] Design a loss function, choosing an appropriate loss function based on the specific task. Since this is a binary classification task for different cigarette brands, a binary cross-entropy loss function is used. Select an appropriate optimizer, Adam, and set the learning rate. For fine-tuning tasks in transfer learning, a smaller learning rate (1e-5) is usually used to avoid large changes in pre-trained weights.

[0138] Set the model training strategy, which is divided into two stages:

[0139] Phase 1: Only train the custom classification head. At this point, freeze all convolutional layers of EfficientNet and train only the newly added classification layer to ensure that the model can learn basic classification capabilities for the new task.

[0140] Phase 2: Unfreeze some EfficientNet layers and perform fine-tuning. After the model converges in the first phase, unfreeze the high-level layers of EfficientNet and fine-tune the entire network using a smaller learning rate to adapt the model to the high-level features of the display detection task.

[0141] By migrating EfficientNet-B3 to the themed scene display detection task and leveraging its pre-training capabilities on large-scale image data, we were able to significantly reduce training time and data requirements. By freezing some layers, adjusting the classification head, and fine-tuning the model, we adapted EfficientNet-B3 to the specific needs of the company's display detection task, improving the accuracy and efficiency of display photo detection.

[0142] Step S304, multi-task learning:

[0143] Introducing multi-task learning (MTL) in the display inspection task allows for simultaneous processing of multiple related tasks. Since compliance and non-compliance testing is performed on different cigarette series, each type of cigarette is trained as a separate task. By sharing parts of the model, multi-task learning can improve the learning results of each task and offer advantages in computational efficiency. Task definition: Display inspection can be divided into multiple tasks based on requirements, with the following objectives for each task:

[0144] Task 1: Liqun brand cigarette detection: Determine whether the image correctly displays Liqun brand cigarettes and confirm their placement;

[0145] Task 2: Detection of promotional materials: Detect whether designated promotional materials, such as display boards and related handicrafts, are placed in the store;

[0146] Task 3: Competitive product detection: Detect whether other brands of cigarettes appear in the display and evaluate the presence of competing products;

[0147] Task 4: Display Compliance Judgment: Based on the specified display standards, determine whether the entire display meets the requirements (yes / no).

[0148] The multi-task learning framework design includes shared feature extraction layers and task-specific output layers. The shared feature extraction layers are the first few layers of the network, responsible for extracting common low-level and mid-level features from the image. All tasks share these layers; the task-specific output layers are independent output layers used by each task to complete specific task goals according to its own needs.

[0149] The model architecture consists of a shared backbone network and task-specific branches. The shared backbone network uses a pre-trained EfficientNet model as a feature extractor. In multi-task learning, all tasks share the first few layers of the model, which can extract common image features (such as texture, shape, and color). The task-specific branches follow the shared backbone network and are designed independently for each task. Each branch consists of specialized fully connected layers, dedicated to performing classification or regression for a specific task.

[0150] Task 1: Liqun brand detection branch: A multi-classification output layer used to determine whether Liqun brand cigarettes are displayed correctly.

[0151] Task 2: Promotional Material Detection Branch: A binary classification output layer used to determine whether the correct promotional material is displayed.

[0152] Task 3: Competitive product detection branch: A multi-classification or binary classification output layer used to detect the presence of other competing cigarettes.

[0153] Task 4: Display compliance judgment branch: A binary classification output layer is used to determine whether the display is compliant.

[0154] Each task has an independent loss function, and the final total loss is the weighted sum of all task losses. Figure 5 is a training loss curve diagram of the image detection method according to the preferred embodiment of the present application, such as Figure 5 The specific plan is as follows:

[0155] Task 1: Brand Detection Loss: We use categorical cross-entropy as the loss function because this is a multi-classification problem that requires determining whether the cigarette brand is placed correctly.

[0156] Task 2: Promotional material detection loss: Use binary cross-entropy because this is a two-classification problem (whether the specified material is placed).

[0157] Task 3: Competitive product detection loss: Depending on the detection target, use binary cross-entropy (two-classification) or categorical cross-entropy (multi-classification).

[0158] Task 4: Compliance judgment loss: Use binary cross-entropy to determine whether the display criteria are met.

[0159] The total loss function can be defined as:

[0160] L=α1*L 品牌检测 +α2*L 物料检测 +α3*L 竞品检测 +α4*L 合规检测 ;

[0161] Among them, α1, α2, α3, and α4 are loss weights used to balance the contributions of different tasks. These weights can be adjusted through experiments to find the optimal combination.

[0162] Set the optimizer and learning rate:

[0163] Optimizer: Choose the Adam optimizer because of its good stability in multi-task learning.

[0164] Learning rate: Set the initial learning rate to a small value, especially when fine-tuning the pre-trained shared feature extraction layer. A learning rate of 1e-5 can be used. The Learning Rate Scheduler can be used to dynamically adjust the learning rate to help the model converge gradually.

[0165] Because multi-task learning requires processing data from multiple tasks in a single batch, the data preprocessing phase ensures that the data annotations and inputs for each task are aligned. Image input: All tasks share the same image input. Multi-task labeling: Generates corresponding multi-task labels for each image. Label formats include brand detection, material detection, competitor detection, and compliance assessment.

[0166] Multi-task model training strategy:

[0167] Gradual unlocking of tasks: You can start training with a few tasks and gradually unlock more tasks to reduce the complexity and task conflict in the early stages of training.

[0168] Weight sharing strategy: The number of layers shared by each task can be controlled. Low-level features (such as texture, color, etc.) can usually be fully shared, while high-level features (such as high-level semantics of brand recognition) may require more task-specific parameters.

[0169] Balanced loss: If the loss of some tasks is too large or too small, the weights in the loss function can be adjusted to prevent one task from dominating the model's learning.

[0170] Model tuning process:

[0171] Adjust the loss function weight: Through experiments, adjust the weight of each task loss to ensure the balance between multiple tasks and prevent the loss of a certain task from dominating the entire training process.

[0172] Hyperparameter tuning: Adjust hyperparameters such as learning rate, optimizer, batch size, and adjust model training strategy based on performance on the validation set.

[0173] Step S305, model usage:

[0174] After the second step of data preprocessing, the identified data is put into the model that combines transfer learning and multi-task learning. The model prediction can determine whether the image is displayed in compliance. Figure 6 、 Figure 7 , Figure 6 In the figure, different rectangular boxes are used to mark the identified items, such as Liqun product 1, background board 1, etc. Figure 7 The identified items are marked with different rectangular frames, such as Liqun product 2, stainless steel display table, crystal ball ornaments, etc.

[0175] This embodiment also provides an image detection device, which is used to implement the above-mentioned embodiments and preferred embodiments. Details that have already been described will not be repeated. As used below, the terms "module," "unit," "subunit," etc. may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0176] Figure 8 is a structural block diagram of an image detection device according to an embodiment of the present application, such as Figure 8 As shown, the device includes:

[0177] An image acquisition module 10 is used to acquire an image to be processed;

[0178] The transfer learning module 20 is used to input the image to be processed into the trained multi-task detection network model, and perform feature extraction processing on the image to be processed through the transfer learning network of the multi-task detection network model to obtain common features; wherein the transfer learning network is trained based on the pre-trained basic neural network;

[0179] A multi-task detection module 30 is used to use multiple independent task branches in the multi-task target detection network of the multi-task detection network model to extract task branch features from the common features respectively, and output multiple task branch detection results;

[0180] The result generation module 40 is used to obtain a target detection result based on the detection results of multiple task branches.

[0181] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0182] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0183] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0184] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0185] S1, obtain the image to be processed;

[0186] S2: Input the image to be processed into the trained multi-task detection network model, and perform feature extraction on the image to be processed through the transfer learning network of the multi-task detection network model to obtain common features; wherein the transfer learning network is trained based on the pre-trained basic neural network;

[0187] S3, using multiple independent task branches in the multi-task target detection network of the multi-task detection network model, respectively extracts task branch features from common features and outputs multiple task branch detection results;

[0188] S4, based on the detection results of multiple task branches, obtain the target detection results.

[0189] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be repeated here.

[0190] In addition, in conjunction with the image detection method in the above embodiments, the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the image detection methods in the above embodiments is implemented.

[0191] Those skilled in the art should understand that the various technical features of the above-described embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the various technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0192] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An image detection method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a trained multi-task detection network model, and performing feature extraction processing on the image to be processed via a transfer learning network of the multi-task detection network model to obtain common features; wherein the transfer learning network is trained based on a pre-trained basic neural network; Utilizing multiple independent task branches in the multi-task target detection network of the multi-task detection network model, respectively performing task branch feature extraction on the common features, and outputting multiple task branch detection results; Based on the detection results of the multiple task branches, a target detection result is obtained.

2. The image detection method according to claim 1, wherein: The step of obtaining an image to be processed includes: Get the initial image; Performing standardization and normalization processing on the initial image to obtain a standard image with a size of preset standard pixels; Performing a data enhancement operation on the standard image to obtain the image to be processed.

3. The image detection method according to claim 2, wherein: The performing of a data enhancement operation on the standard image to obtain the image to be processed includes: Performing a data augmentation operation on the standard image to obtain an enhanced image; A deduplication operation is performed on the enhanced image to obtain the image to be processed.

4. The image detection method according to claim 1, wherein: The training process of the multi-task detection network model includes: Get training images and object detection tasks; Freezing all convolutional layers in the pre-trained basic neural network, and after freezing, replacing the original classification head in the basic neural network with a newly added classification head adapted to the target detection task based on the target detection task, thereby obtaining a training transfer learning network; The training image is input into the training transfer learning network, and the training transfer learning network is trained using a preset loss function to obtain the multi-task detection network model.

5. The image detection method according to claim 4, characterized in that: The step of inputting the training image into the training transfer learning network and training the training transfer learning network using a preset loss function to obtain the multi-task detection network model includes: Inputting the training image into the training transfer learning network, and performing preliminary training on the training transfer learning network using a preset loss function to obtain a preliminary training model; Determining a number of high-level convolutional layers in the preliminary training model; Based on a preset dynamic learning rate adjustment strategy, fine-tune the high-level convolutional layer and the newly added classification head to obtain the high-level weights corresponding to the high-level convolutional layer and the classification head weights corresponding to the newly added classification head; Based on the high-level weights and the classification head weights, the multi-task detection network model based on transfer learning is obtained.

6. The image detection method according to claim 1, wherein: The training process of the multi-task detection network model also includes: Based on the acquired object detection tasks, multiple training task branches are set up in the multi-task object detection training network; Assigning a corresponding branch loss function to each of the training task branches, and calculating a loss calculation result corresponding to each of the training task branches based on each of the branch loss functions; Based on the loss calculation results, the branch weights corresponding to the training task branches are iteratively updated to obtain the trained multi-task object detection network.

7. The image detection method according to claim 6, characterized in that: The step of iteratively updating the branch weights corresponding to the training task branches based on the loss calculation results to obtain the trained multi-task object detection network includes: Inputting the loss calculation results into the weighted total loss function to obtain the total loss result; The branch weights are iteratively updated based on the total loss result to obtain the trained multi-task object detection network.

8. An image detection device, characterized in that: include: An image acquisition module, used for acquiring an image to be processed; A transfer learning module is configured to input the image to be processed into a trained multi-task detection network model, and perform feature extraction on the image to be processed via the transfer learning network of the multi-task detection network model to obtain common features; wherein the transfer learning network is trained based on a pre-trained basic neural network; A multi-task detection module is used to utilize multiple independent task branches in the multi-task target detection network of the multi-task detection network model to perform task branch feature extraction on the common features respectively, and output multiple task branch detection results; A result generation module is used to obtain a target detection result based on the detection results of the multiple task branches.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the image detection method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the image detection method according to any one of claims 1 to 7 when running.

Images