Neural network training method, image detection method, device, equipment and product

Abstract

The present disclosure provides a neural network training method, an image detection method, an apparatus, a device, a medium and a product. A plurality of training sample images with sample object annotation information under a target business scenario and a plurality of auxiliary open source images with auxiliary object annotation information under a plurality of business scenarios are used to generate target sample images and target object annotation information for target objects of various object types in the target sample images. Then, the target sample images are used to train a neural network to obtain an image detection model for detecting objects in images. The type and effect of the target sample images used for training can be enriched, the richness of the target sample images is increased, the data imbalance between different categories of objects is reduced, and through multi-scale fusion and merging of deep and shallow layer features, a plurality of object detection channels are set to detect objects with different area ratios, thereby greatly improving the image detection accuracy of the image detection model.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and more specifically, to a neural network training method, an image detection method, an apparatus, a device, a medium, and a product. Background Technology

[0002] With the advancement of technology and the progress of the times, the types of business in the production process are constantly expanding. As an important operational area in the production process, the intelligent management and control of warehouses is becoming increasingly important. Warehouse scenarios involve various objects that require attention, such as personnel, equipment, and goods. To avoid high-risk hazards, it is necessary to detect the behavior of each object to identify behaviors with safety risks, thereby promptly resolving and reducing safety issues in the warehouse scenario and effectively promoting safe production operations.

[0003] In the intelligent management and control of warehouses, most methods involve configuring camera equipment in the warehouse to capture images of objects of interest as training samples. This training process generates a model that can locate and detect objects in the images to identify behaviors that pose a security risk. However, the uneven distribution of the number of objects in each category affects the accuracy and precision of the trained model, making it difficult to obtain high-quality detection results. Summary of the Invention

[0004] This disclosure provides at least one neural network training method, image detection method, apparatus, device, medium, and product.

[0005] This disclosure provides a neural network training method, the method comprising:

[0006] Acquire multiple training sample images with sample object annotation information in the target business scenario, and multiple auxiliary open source images with auxiliary object annotation information in multiple business scenarios. The training sample images include at least one sample object in the target business scenario, and the auxiliary open source images include at least one auxiliary object in at least one business scenario.

[0007] Based on the auxiliary open-source image, the auxiliary object annotation information, the training sample image, and the sample object annotation information, a target sample image and target object annotation information for target objects of various object types in the target sample image are generated. The target object includes the sample object and the auxiliary object.

[0008] The neural network is trained using the generated target sample image and the target object annotation information to obtain a trained image detection model for detecting objects in the image.

[0009] In one optional implementation, acquiring multiple training sample images with sample object annotation information in the target business scenario includes:

[0010] Acquire images captured in the target business scenario;

[0011] The acquired images are filtered according to the image quality requirements to obtain acquired images that meet the image quality requirements;

[0012] Object annotation is performed on the acquired images that meet the image quality requirements to obtain training sample images and sample object annotation information of the sample objects in the training sample images. The sample object annotation information is used to indicate the object type of the sample object and the annotation position of the sample object in the training sample image.

[0013] In one optional implementation, generating a target sample image and target object annotation information for target objects of various object types in the target sample image based on the auxiliary open-source image, the auxiliary object annotation information, the training sample image, and the sample object annotation information includes:

[0014] Based on the auxiliary object annotation information and the sample object annotation information, the auxiliary object image of the auxiliary object is extracted from the auxiliary open source image;

[0015] Based on the sample object annotation information, the sample object image of the sample object is extracted from the training sample image;

[0016] Based on the auxiliary object image, the auxiliary object annotation information, the sample object image, the sample object annotation information, and the training sample image, a target sample image and target object annotation information for target objects of various object types in the target sample image are generated.

[0017] In one optional implementation, the step of extracting the auxiliary object image from the auxiliary open-source image based on the auxiliary object annotation information and the sample object annotation information includes:

[0018] The object type of the auxiliary object and the annotation position of the auxiliary object in the auxiliary open source image are determined from the auxiliary object annotation information, and the object type of the sample object is determined from the sample object annotation information;

[0019] Detect whether the object type of the auxiliary object is consistent with the object type of the sample object;

[0020] If they match, extract the auxiliary object image from the auxiliary open source image according to the labeled position corresponding to the auxiliary object.

[0021] In one optional implementation, extracting the sample object image content of the sample object from the training sample image based on the sample object annotation information includes:

[0022] According to the annotation position of the sample object in the training sample image indicated by the annotation information of the sample object, the sample object image of the sample object is extracted from the training sample image.

[0023] In an optional implementation, before generating the target sample image and target object annotation information for target objects of various object types in the target sample image based on the auxiliary object image, the auxiliary object annotation information, the sample object image, the sample object annotation information, and the training sample image, the method further includes:

[0024] Obtain the preset object size corresponding to various object types in the target business scenario;

[0025] Based on the object type corresponding to the auxiliary object image, the object type corresponding to the sample object image, and the preset object size for each object type, the auxiliary object image and the sample object image are resized according to the corresponding preset object size to obtain the resized content of the auxiliary object image and the resized content of the sample object image.

[0026] In one optional implementation, generating a target sample image and target object annotation information for target objects of various object types in the target sample image based on the auxiliary object image, the auxiliary object annotation information, the sample object image, the sample object annotation information, and the training sample image includes:

[0027] Based on the sample object annotation information, at least one additional object image is added to the blank image region other than the sample object image in the training sample image to obtain the target sample image. The additional object image includes the auxiliary object image and the sample object image.

[0028] Based on the object type of the added object image and the addition position in the training sample image, as well as the sample object annotation information of the training sample image, the target object annotation information of the target sample image is determined.

[0029] In one optional implementation, the step of adding at least one object image to a blank image region other than the object image in the training sample image based on the sample object annotation information to obtain a target sample image includes:

[0030] Based on the object type indicated by the sample object annotation information, determine the type of object to be added to the training sample image corresponding to the object to be added;

[0031] For the added object image under the added object type, determine the number and position of the added object images added to the training sample images;

[0032] Based on the type of the added object, the number of added objects, and the location of the added objects, the corresponding added object images are added to the training sample images to obtain the target sample images.

[0033] In an optional implementation, before adding the corresponding object image to the training sample image based on the object type, the number of objects added, and the location of the objects added, to obtain the target sample image, the method further includes:

[0034] Based on the addition location, determine the proportion of the added object corresponding to the added object image in the training sample image;

[0035] The added object image is linearly scaled according to the object ratio to obtain the processed added object image.

[0036] In one optional implementation, training the neural network using the generated target sample image and the target object annotation information to obtain a trained image detection model for detecting objects in the image includes:

[0037] The target sample image is input into a neural network. The feature extraction and fusion layer of the neural network extracts and fuses multi-level image features of the target sample image to obtain multi-level fusion features generated during the fusion of the extracted multi-level image features. The image size of the feature image indicated by the fusion features at different levels is different.

[0038] The fusion features at each level are input to the corresponding object detection channels in the object detection layer of the neural network to obtain the detection annotation information of each target object in the target sample image. The object detection layer includes multiple object detection channels, and the area ratio of the objects in the image detected by different object detection channels is different.

[0039] Based on the detection annotation information of each target object in the target sample image, the image detection result of the target sample image output by the neural network is determined;

[0040] Based on the image detection results and the target object annotation information, the network parameters of the neural network are adjusted until the neural network meets the training cutoff condition. The trained neural network is then used as an image detection model for detecting objects in the image.

[0041] This disclosure provides an image detection method, the method comprising:

[0042] Obtain the image to be detected and the image detection model, wherein the image detection model is trained according to the neural network training method described above;

[0043] The image to be detected is detected by the image detection model to obtain the image detection result of the image to be detected.

[0044] This disclosure also provides a neural network training apparatus, the apparatus comprising:

[0045] The data acquisition module is used to acquire multiple training sample images with sample object annotation information in the target business scenario, and multiple auxiliary open source images with auxiliary object annotation information in multiple business scenarios. The training sample images include at least one sample object in the target business scenario, and the auxiliary open source images include at least one auxiliary object in at least one business scenario.

[0046] The data generation module is used to generate a target sample image and target object annotation information for target objects of various object types in the target sample image based on the auxiliary open-source image, the auxiliary object annotation information, the training sample image and the sample object annotation information. The target object includes the sample object and the auxiliary object.

[0047] The model training module is used to train the neural network using the generated target sample image and the corresponding target object annotation information to obtain a trained image detection model for detecting objects in the image.

[0048] In one optional implementation, the data acquisition module is specifically used for:

[0049] Acquire images captured in the target business scenario;

[0050] The acquired images are filtered according to the image quality requirements to obtain acquired images that meet the image quality requirements;

[0051] Object annotation is performed on the acquired images that meet the image quality requirements to obtain training sample images and sample object annotation information of the sample objects in the training sample images. The sample object annotation information is used to indicate the object type of the sample object and the annotation position of the sample object in the training sample image.

[0052] In one optional implementation, the data generation module is specifically used for:

[0053] Based on the auxiliary object annotation information and the sample object annotation information, the auxiliary object image of the auxiliary object is extracted from the auxiliary open source image;

[0054] Based on the sample object annotation information, the sample object image of the sample object is extracted from the training sample image;

[0055] Based on the auxiliary object image, the auxiliary object annotation information, the sample object image, the sample object annotation information, and the training sample image, a target sample image and target object annotation information for target objects of various object types in the target sample image are generated.

[0056] In one optional implementation, when the data generation module extracts the auxiliary object image of the auxiliary object from the auxiliary open-source image based on the auxiliary object annotation information and the sample object annotation information, it is specifically used for:

[0057] The object type of the auxiliary object and the annotation position of the auxiliary object in the auxiliary open source image are determined from the auxiliary object annotation information, and the object type of the sample object is determined from the sample object annotation information;

[0058] Detect whether the object type of the auxiliary object is consistent with the object type of the sample object;

[0059] If they match, extract the auxiliary object image from the auxiliary open source image according to the labeled position corresponding to the auxiliary object.

[0060] In one optional implementation, when the data generation module extracts the sample object image content of the sample object from the training sample image based on the sample object annotation information, it is specifically used for:

[0061] According to the annotation position of the sample object in the training sample image indicated by the annotation information of the sample object, the sample object image of the sample object is extracted from the training sample image.

[0062] In an optional implementation, the data generation module is further configured to:

[0063] Obtain the preset object size corresponding to various object types in the target business scenario;

[0064] Based on the object type corresponding to the auxiliary object image, the object type corresponding to the sample object image, and the preset object size for each object type, the auxiliary object image and the sample object image are resized according to the corresponding preset object size to obtain the resized content of the auxiliary object image and the resized content of the sample object image.

[0065] In an optional implementation, when the data generation module generates a target sample image and target object annotation information for target objects of various object types in the target sample image based on the auxiliary object image, the auxiliary object annotation information, the sample object image, the sample object annotation information, and the training sample image, it is specifically used for:

[0066] Based on the sample object annotation information, at least one additional object image is added to the blank image region other than the sample object image in the training sample image to obtain the target sample image. The additional object image includes the auxiliary object image and the sample object image.

[0067] Based on the object type of the added object image and the addition position in the training sample image, as well as the sample object annotation information of the training sample image, the target object annotation information of the target sample image is determined.

[0068] In an optional implementation, when the data generation module adds at least one object image to a blank image region other than the object image in the training sample image based on the object annotation information to obtain the target sample image, it specifically performs the following:

[0069] Based on the object type indicated by the sample object annotation information, determine the type of object to be added to the training sample image corresponding to the object to be added;

[0070] For the added object image under the added object type, determine the number and position of the added object images added to the training sample images;

[0071] Based on the type of the added object, the number of added objects, and the location of the added objects, the corresponding added object images are added to the training sample images to obtain the target sample images.

[0072] In an optional implementation, the data generation module is further configured to:

[0073] Based on the addition location, determine the proportion of the added object corresponding to the added object image in the training sample image;

[0074] The added object image is linearly scaled according to the object ratio to obtain the processed added object image.

[0075] In one optional implementation, the model training module is specifically used for:

[0076] The target sample image is input into a neural network. The feature extraction and fusion layer of the neural network extracts and fuses multi-level image features of the target sample image to obtain multi-level fusion features generated during the fusion of the extracted multi-level image features. The image size of the feature image indicated by the fusion features at different levels is different.

[0077] The fusion features at each level are input to the corresponding object detection channels in the object detection layer of the neural network to obtain the detection annotation information of each target object in the target sample image. The object detection layer includes multiple object detection channels, and the area ratio of the objects in the image detected by different object detection channels is different.

[0078] Based on the detection annotation information of each target object in the target sample image, the image detection result of the target sample image output by the neural network is determined;

[0079] Based on the image detection results and the target object annotation information, the network parameters of the neural network are adjusted until the neural network meets the training cutoff condition. The trained neural network is then used as an image detection model for detecting objects in the image.

[0080] This disclosure also provides an image detection apparatus, the apparatus comprising:

[0081] The image acquisition module is used to acquire the image to be detected and the image detection model, wherein the image detection model is trained according to the aforementioned neural network training device;

[0082] The image detection module is used to detect the image to be detected using the image detection model, and obtain the image detection result of the image to be detected.

[0083] This disclosure also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the neural network training method or image detection method described above are performed.

[0084] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the neural network training method or image detection method described above.

[0085] This disclosure also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the neural network training method or image detection method described above.

[0086] The neural network training method, image detection method, apparatus, device, medium, and product provided in this disclosure can enrich the types and effects of target sample images used for training when training a neural network, increase the richness of target sample images, enhance the diversity of training data, reduce data imbalance between different categories of objects, and improve the learning effect of the neural network for different image features. At the same time, by fusing and merging deep and shallow features at multiple scales and setting multiple object detection channels to detect objects with different area proportions in a targeted manner, the image detection accuracy of the image detection model is greatly improved.

[0087] Furthermore, when performing image detection using an image detection model, multiple object detection channels in the object detection layer of the image detection model, through fine-grained channel division, allow for targeted detection of objects with different area proportions. This helps improve the detection accuracy for objects with various area proportions and object types, thereby effectively improving the accuracy of the detection results, reducing the impact of different object area proportions on the detection results, and exhibiting high anti-interference capabilities.

[0088] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings.

[0089] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0090] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0091] Figure 1 A flowchart of a neural network training method provided in an embodiment of this disclosure is shown;

[0092] Figure 2 The flowchart illustrates a specific method for generating target sample images in the neural network training method provided in this embodiment of the present disclosure.

[0093] Figure 3a This diagram illustrates the structure of a feature extraction and fusion layer in a neural network provided by an embodiment of the present disclosure.

[0094] Figure 3b This diagram illustrates the structure of an object detection layer in a neural network provided in an embodiment of the present disclosure.

[0095] Figure 4 A flowchart of an image detection method provided by an embodiment of this disclosure is shown;

[0096] Figure 5 A schematic diagram of a neural network training apparatus provided in an embodiment of this disclosure is shown;

[0097] Figure 6 A schematic diagram of an image detection apparatus provided in an embodiment of this disclosure is shown;

[0098] Figure 7 A schematic diagram of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0099] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0100] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0101] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0102] Research has found that in intelligent warehouse management, most methods involve deploying cameras in the warehouse to capture images of objects of interest as training samples. This training aims to develop a model capable of locating and detecting objects in the images to identify potentially dangerous behaviors. However, the captured images are highly susceptible to interference from environmental factors, location, time, and the movement of the objects themselves. This can lead to problems such as difficulty in detection, inaccurate detection, and low precision in image detection results. Furthermore, if the distribution of objects in different categories is uneven, it can affect the accuracy and precision of the trained model, making it difficult to obtain a high-quality training model and resulting in poor training outcomes.

[0103] Based on the above research, this disclosure provides a neural network training method and an image detection method. When training the neural network, the types and effects of target sample images used for training can be enriched, the richness of target sample images can be increased, the diversity of training data can be improved, the data imbalance between different categories of objects can be reduced, and the learning effect of the neural network for different image features can be improved. At the same time, by multi-scale fusion and merging of deep and shallow features, multiple object detection channels can be set to detect objects with different area proportions in a targeted manner, which greatly improves the image detection accuracy of the image detection model.

[0104] Furthermore, when performing image detection using an image detection model, multiple object detection channels in the object detection layer of the image detection model, through fine-grained channel division, allow for targeted detection of objects with different area proportions. This helps improve the detection accuracy for objects with various area proportions and object types, thereby effectively improving the accuracy of the detection results, reducing the impact of different object area proportions on the detection results, and exhibiting high anti-interference capabilities.

[0105] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.

[0106] To facilitate understanding of this embodiment, a neural network training method disclosed in this disclosure will first be described in detail. The execution entity of the neural network training method provided in this disclosure can be a neural network training device, such as a terminal device, a server, or other processing device. In some possible implementations, the neural network training method can be implemented by a processor calling computer-readable instructions stored in memory.

[0107] Please see Figure 1 , Figure 1 This is a flowchart illustrating a neural network training method provided in an embodiment of this disclosure. Figure 1 As shown in the embodiments of this disclosure, the neural network training method includes:

[0108] S101: Obtain multiple training sample images with sample object annotation information in the target business scenario, and multiple auxiliary open source images with auxiliary object annotation information in multiple business scenarios. The training sample images include at least one sample object in the target business scenario, and the auxiliary open source images include at least one auxiliary object in at least one business scenario.

[0109] In this step, for the target business scenario requiring intelligent management and control, images collected in the target business scenario can be used as training sample images for training the neural network. For example, for intelligent management and control in the production process, the target business scenario can be a warehouse, workshop, factory, etc.; for intelligent management and control in the teaching process, the target business scenario can be a classroom, library, laboratory, etc.

[0110] Optionally, training sample images of the target business scenario can be obtained by setting up shooting devices such as cameras in the target business scenario and periodically capturing images according to a preset time setting. Alternatively, videos including business processes can be captured by shooting devices set up in the target business scenario, and key frames can be extracted from the videos as training sample images. No limitation is made here.

[0111] Here, the training sample images include at least one sample object from the target business scenario. For example, for the target business scenario of a warehouse, the sample objects may include staff, goods, vehicles, machines, etc.; for the target business scenario of a laboratory, the sample objects may include students, teachers, teaching equipment, experimental equipment, etc.

[0112] To facilitate subsequent verification of the neural network's training status, the sample objects in the training sample images can be labeled to generate sample object labeling information for each sample object in the training sample images.

[0113] Optionally, the sample object annotation information may include the object type of the sample object and the annotation position of the sample object in the training sample image.

[0114] To enhance the diversity of training samples, in addition to obtaining training sample images from the target business scenario, data from other business scenarios can also be incorporated. Specifically, multiple auxiliary open-source images from various business scenarios can be acquired, each including at least one auxiliary object from at least one business scenario. Here, the multiple business scenarios can include the target business scenario.

[0115] To improve training speed and reduce data processing workload, we can use the existing open-source (CommonObjects in Context, COCO) dataset as auxiliary open-source images. The COCO dataset is a large-scale dataset for object detection, segmentation, keypoint detection, and captioning, containing tens of thousands of images from various business scenarios.

[0116] To facilitate the enrichment of training sample images in the future, auxiliary objects in the auxiliary open source images can be labeled to generate auxiliary object labeling information for each auxiliary object in the auxiliary open source images.

[0117] Optionally, the auxiliary object annotation information may include the object type of the auxiliary object and the annotation position of the auxiliary object in the auxiliary open source image.

[0118] It is understandable that some images captured by shooting equipment set up in the target business scenario may have problems such as unclear objects due to dim lighting, blurred images due to objects in motion, blurred images due to incorrect camera parameter settings, or images of objects that do not exist at a specific time point. These images, when used as training samples, will cause the neural network to learn incorrect image features. Therefore, it is necessary to filter out images with the above problems.

[0119] Accordingly, in some possible implementations, acquiring multiple training sample images with sample object annotation information in the target business scenario includes:

[0120] Acquire images captured in the target business scenario;

[0121] The acquired images are filtered according to the image quality requirements to obtain acquired images that meet the image quality requirements;

[0122] Object annotation is performed on the acquired images that meet the image quality requirements to obtain training sample images and sample object annotation information of the sample objects in the training sample images. The sample object annotation information is used to indicate the object type of the sample object and the annotation position of the sample object in the training sample image.

[0123] In this step, after acquiring the captured images by the shooting device set up in the target business scenario, blurry captured images, captured images with blurred objects in the images, and captured images without objects in the images can be filtered out and deleted according to the image quality requirements, so as to obtain captured images that meet the image quality requirements. It can be understood that the image quality requirements can be that the image is clear, that there are objects in the image and the objects are clear, etc.

[0124] Here, the image screening process can be based on manual judgment, or it can be achieved by using a pre-trained image screening model to identify and delete images that do not meet the image quality requirements. No limitations are imposed here.

[0125] In other embodiments, the acquired image can be processed based on its display status, such as color temperature and contrast, including flipping, scaling, grayscale correction, color separation, sharpening, white balance, rotation, color transformation, contrast transformation, automatic contrast adjustment, and tone equalization.

[0126] After deleting the acquired images that do not meet the image quality requirements, object annotation can be performed on the acquired images that meet the image quality requirements to obtain training sample images and sample object annotation information of the sample objects in the training sample images.

[0127] Here, the sample object annotation information is used to indicate the object type of the sample object and the annotation position of the sample object in the training sample image. For example, for the target business scenario of a warehouse, the object type of the sample object may include people, goods, vehicles, machines, etc.

[0128] The labeled position of the sample object in the training sample image can be labeled in the form of a bounding box. Specifically, the labeled position of the sample object can be represented by the coordinates of the upper left and lower right corners of the bounding box, or by the coordinates of the lower left and upper right corners of the bounding box, or by the center point, length, and width of the bounding box, or by the coordinates of the four corners of the bounding box. No limitation is made here.

[0129] Optionally, for a pre-built neural network to be trained, the data format of the sample object annotation information can be converted into a data format adapted to the neural network according to the data format corresponding to the neural network.

[0130] For example, for the sample object annotation information of a certain sample object, the sample type is truck, and the annotation position is the coordinates of the upper left and lower right corners of the annotation box. However, the neural network to be trained can only recognize the sample object annotation information of sample type as truck and annotation position as the center point, length and width of the annotation box. Therefore, the current sample object annotation information needs to be converted into a data format so that the converted sample object annotation information can be recognized by the neural network.

[0131] S102: Based on the auxiliary open-source image, the auxiliary object annotation information, the training sample image, and the sample object annotation information, generate a target sample image and target object annotation information for target objects of various object types in the target sample image, wherein the target object includes the sample object and the auxiliary object.

[0132] In this step, to improve the diversity of training data and reduce data imbalance between different categories of objects, data augmentation processing can be performed on the training sample images. Specifically, based on the auxiliary object annotation information and the sample object annotation information, auxiliary objects from the auxiliary open-source images and sample objects from the training sample images can be adaptively added to the training sample images to obtain target sample objects for neural network training. For details, please also refer to... Figure 2 , Figure 2 A flowchart illustrating a specific method for generating a target sample image in the neural network training method provided in this disclosure embodiment. For example... Figure 2 As shown, the method includes:

[0133] S1021: Based on the auxiliary object annotation information and the sample object annotation information, extract the auxiliary object image of the auxiliary object from the auxiliary open source image.

[0134] In this step, in order to enrich the types and effects of the target sample images used for training, the auxiliary object image of the auxiliary object in the auxiliary open source image can be extracted so that the training sample image can be data augmented based on the extracted auxiliary object image in the subsequent process.

[0135] Here, in order to improve training speed and reduce data processing workload, the auxiliary object image can be extracted by an image extraction model that matches the auxiliary open source image.

[0136] Furthermore, in order to reduce interference from background factors, the auxiliary object can be cut along its outer edge line to extract the auxiliary object image.

[0137] Therefore, in some possible implementations, the step of extracting the auxiliary object image from the auxiliary open-source image based on the auxiliary object annotation information and the sample object annotation information includes:

[0138] The object type of the auxiliary object and the annotation position of the auxiliary object in the auxiliary open source image are determined from the auxiliary object annotation information, and the object type of the sample object is determined from the sample object annotation information;

[0139] Detect whether the object type of the auxiliary object is consistent with the object type of the sample object;

[0140] If they match, extract the auxiliary object image from the auxiliary open source image according to the labeled position corresponding to the auxiliary object.

[0141] It is understood that the auxiliary open-source images come from multiple business scenarios. Therefore, the object types of the auxiliary objects in the auxiliary open-source images may not belong to the target business scenario. For example, for the target business scenario of a warehouse, if the auxiliary open-source image comes from the business scenario of a road, the auxiliary objects of the type of people and vehicles in the road image belong to the target business scenario of a warehouse, but the auxiliary objects of the type of traffic lights and road signs in the road image do not belong to the target business scenario of a warehouse.

[0142] To ensure consistency of object types in the target business scenario, the object type of the auxiliary object can first be determined from the auxiliary object annotation information, and the object type of the sample object can be determined from the sample object annotation information. Then, it is checked whether the object type of the auxiliary object is consistent with the object type of the sample object. It can be understood that if the object type of the auxiliary object is consistent with the object type of the sample object, it means that the auxiliary object belongs to the target business scenario. Therefore, the auxiliary object image of the auxiliary object can be extracted from the auxiliary open source image according to the annotation position corresponding to the auxiliary object.

[0143] S1022: Based on the sample object annotation information, extract the sample object image of the sample object from the training sample image.

[0144] In this step, in order to enrich the types and effects of the target sample images used for training, the sample object images of the sample objects in the training sample images can be extracted so that the training sample images can be data augmented based on the extracted sample object images in the subsequent process.

[0145] Here, in order to improve training speed and reduce data processing workload, the image extraction model that matches the auxiliary open source image can be adjusted using the training sample image and the sample object annotation information, so that the adjusted image extraction model matches the training sample image, and thus the sample object image can be extracted using the adjusted image extraction model.

[0146] Alternatively, other image extraction models or artificial intelligence algorithms can be used to extract the sample object image.

[0147] Furthermore, in order to reduce interference from background factors, the sample object can be cut along its outer edge line to extract the sample object image.

[0148] Specifically, in order to extract the sample object image content of the sample object from the training sample image based on the sample object annotation information, the sample object image of the sample object can be extracted from the training sample image according to the annotation position of the sample object in the training sample image indicated by the sample object annotation information.

[0149] S1023: Based on the auxiliary object image, the auxiliary object annotation information, the sample object image, the sample object annotation information, and the training sample image, generate a target sample image and target object annotation information for target objects of various object types in the target sample image.

[0150] In this step, in order to enrich the types and effects of target sample images used for training, when the auxiliary object image is extracted from the auxiliary open source image and the sample object image is extracted from the training sample image, the training sample image can be data augmented based on the auxiliary object annotation information and the sample object annotation information to obtain the target sample image and the target object annotation information for the target object under various object types in the target sample image.

[0151] In practical applications, the image sizes of the auxiliary open-source image and the training sample image may be different. Therefore, the image sizes of the auxiliary object image extracted from the auxiliary open-source image and the sample object image extracted from the training sample image may also be different. In order to facilitate unified management, after the auxiliary object image and the sample object image are extracted, adaptive adjustments can be made to the auxiliary object image and the sample object image.

[0152] Accordingly, in some possible implementations, before generating the target sample image and target object annotation information for target objects of various object types in the target sample image based on the auxiliary object image, the auxiliary object annotation information, the sample object image, the sample object annotation information, and the training sample image, the method further includes:

[0153] Obtain the preset object size corresponding to various object types in the target business scenario;

[0154] Based on the object type corresponding to the auxiliary object image, the object type corresponding to the sample object image, and the preset object size for each object type, the auxiliary object image and the sample object image are resized according to the corresponding preset object size to obtain the resized content of the auxiliary object image and the resized content of the sample object image.

[0155] In this step, preset object sizes corresponding to various object types in the target business scenario can be obtained. For each auxiliary object image and each sample object image, the corresponding preset object size is determined from multiple preset object sizes according to its corresponding object type. Then, the image size is adjusted accordingly according to the corresponding preset object size to obtain the adjusted auxiliary object image content and the adjusted sample object image content.

[0156] After obtaining the adjusted auxiliary object image content and the adjusted sample object image content, the adjusted auxiliary object image content and the adjusted sample object image content can be used to perform data augmentation processing on the training sample image.

[0157] Specifically, in some possible implementations, generating a target sample image and target object annotation information for target objects of various object types in the target sample image based on the auxiliary object image, the auxiliary object annotation information, the sample object image, the sample object annotation information, and the training sample image includes:

[0158] Based on the sample object annotation information, at least one additional object image is added to the blank image region other than the sample object image in the training sample image to obtain the target sample image. The additional object image includes the auxiliary object image and the sample object image.

[0159] Based on the object type of the added object image and the addition position in the training sample image, as well as the sample object annotation information of the training sample image, the target object annotation information of the target sample image is determined.

[0160] In this step, based on the sample object annotation information, the type, quantity, and position of the object image to be added in the training sample image can be determined, thereby selecting at least one object image to be added to the training sample image from the auxiliary object image content and the sample object image content.

[0161] To ensure the integrity of the training data, the added object image can be added to the blank image area other than the sample object image in the training sample image. Then, the training sample image with the added object image is used as the target sample image, and the object type and the addition position of the added object image in the training sample image are used as the added object annotation information.

[0162] It is understood that the target sample image at this time includes the added object and the sample object. Therefore, the added object annotation information of the added object and the sample object annotation information of the sample object can be used as the target object annotation information of the target sample image.

[0163] Furthermore, in some possible implementations, the step of adding at least one object image to the blank image region other than the object image in the training sample image based on the object annotation information to obtain the target sample image includes:

[0164] Based on the object type indicated by the sample object annotation information, determine the type of object to be added to the training sample image corresponding to the object to be added;

[0165] For the added object image under the added object type, determine the number and position of the added object images added to the training sample images;

[0166] Based on the type of the added object, the number of added objects, and the location of the added objects, the corresponding added object images are added to the training sample images to obtain the target sample images.

[0167] In this step, in order to reduce the difference in the number of objects of different object types in the target sample image, when the object type of the sample object indicated by the sample object annotation information is determined, the object type that is missing or has a small number in the training sample image can be determined, and this type of object type is taken as the object type to be added to the training sample image.

[0168] Given the labeled position of the sample object in the training sample image as indicated by the labeled information of the sample object, the number of sample objects in the current training sample image and the area of ​​each sample object can be determined. Combined with the area of ​​the training sample image, the area ratio of the sample object in the training sample image can be determined, that is, the object density of the training sample image can be determined. Based on the object density, the number and position of adding the object image to the training sample image can be determined. Specifically, the first number of object images to be added to the training sample and the second number of object images to be added for each type of object can be determined.

[0169] After determining the addition type and the addition quantity, the addition object images of the specified type and quantity can be selected from the auxiliary object image content and the sample object image content. This allows for the determination of the addition object image corresponding to each addition position. Here, to prevent the addition object images from being cropped and to ensure their integrity, it is necessary to ensure that after the addition object images are added to the training sample image at their corresponding addition positions, the content of the addition object images is entirely within the range of the training sample image, i.e., the content of the addition object images does not exceed the range of the training sample image. Therefore, the corresponding addition object images can be added at the addition positions in the training sample image according to the addition positions to obtain the target sample image.

[0170] Optionally, in order to ensure the effectiveness and richness of data augmentation processing, the content of the auxiliary object image and the content of the sample object image can be mixed according to a preset ratio to generate a set of object image content, so as to select the added object image from the set of object image content.

[0171] Here, in order to reduce data imbalance between different categories of objects and improve the learning effect of neural networks on different image features, the target sample image can be made to include objects of multiple object types as much as possible, and the number of objects under different object types can be made as consistent as possible.

[0172] In practical applications, the target sample images are mostly taken from a top-down angle. Therefore, objects near the bottom of the target sample image have smaller areas, while objects near the top have larger areas. Furthermore, the size of objects in the target sample image should conform to real-world scenarios; for example, a person in the same location should be smaller than a car. Therefore, the added object images can be adaptively adjusted before being added to the training sample images.

[0173] Accordingly, in some possible implementations, before adding the corresponding object image to the training sample image based on the object type, the number of objects added, and the location of the objects added, to obtain the target sample image, the method further includes:

[0174] Based on the addition location, determine the proportion of the added object corresponding to the added object image in the training sample image;

[0175] The added object image is linearly scaled according to the object ratio to obtain the processed added object image.

[0176] In this step, the object ratio corresponding to each addition position in the training sample image can be obtained, thereby determining the object ratio of the added object corresponding to the added object image at the addition position in the training sample image. Then, the added object image can be linearly scaled so that the added object corresponding to the processed added object image meets the object ratio, so that the processed added object image can be added to the training sample image in the future.

[0177] Following S102 and S103 above: the generated target sample image and the target object annotation information are used to train the neural network to obtain a trained image detection model for detecting objects in the image.

[0178] In this step, after obtaining the target sample image and the target object annotation information, the target sample image can be used to train the neural network to be trained. It can be understood that model training is a continuous fitting process. Therefore, the trained model can be verified through the target object annotation information to finally obtain the trained image detection model for detecting objects in the image.

[0179] Here, in order to improve the accuracy of the detection results and reduce the impact of different area proportions of objects on the detection results, fine-grained detection channels can be used to target objects with different area proportions for separate detection.

[0180] Specifically, in some possible implementations, training the neural network using the generated target sample image and the target object annotation information to obtain a trained image detection model for detecting objects in the image includes:

[0181] The target sample image is input into a neural network. The feature extraction and fusion layer of the neural network extracts and fuses multi-level image features of the target sample image to obtain multi-level fusion features generated during the fusion of the extracted multi-level image features. The image size of the feature image indicated by the fusion features at different levels is different.

[0182] The fusion features at each level are input to the corresponding object detection channels in the object detection layer of the neural network to obtain the detection annotation information of each target object in the target sample image. The object detection layer includes multiple object detection channels, and the area ratio of the objects in the image detected by different object detection channels is different.

[0183] Based on the detection annotation information of each target object in the target sample image, the image detection result of the target sample image output by the neural network is determined;

[0184] Based on the image detection results and the target object annotation information, the network parameters of the neural network are adjusted until the neural network meets the training cutoff condition. The trained neural network is then used as an image detection model for detecting objects in the image.

[0185] In this step, after obtaining the target sample image, the target sample image is input into a pre-constructed neural network. The neural network includes a feature extraction and fusion layer and an object detection layer. The feature fusion and extraction layer is used to extract multi-level image features from the target sample image to obtain the extracted multi-level image features. Then, the extracted multi-level image features are fused to obtain multi-level fused features.

[0186] Here, the number of levels of feature extraction and feature fusion depends on the actual detection requirements, and no limit is imposed here.

[0187] The object detection layer is used to detect objects using multi-level fusion features. Specifically, the object detection layer includes multiple object detection channels. Different object detection channels are used to detect objects with different area proportions in the image. It can be understood that after the extraction, fusion, and transmission of image features, the original size of each target object in the feature image corresponding to each fusion feature will be enlarged or reduced accordingly, relative to the multi-level fusion features with different corresponding image sizes. In the feature images corresponding to different fusion features, the target objects with different sizes will be basically the same. Therefore, after inputting the multi-level fusion features into the object detection layer, the object detection channel corresponding to each area of ​​the target object can be determined according to the area proportion range corresponding to each object detection channel. Thus, the fusion features at each level are input into the corresponding object detection channel. The object detection channel can detect objects with the same area proportion as the corresponding target object, thereby obtaining the detection annotation information of each target object in the target sample image.

[0188] The number of object detection channels can be set according to actual detection needs, and no limit is imposed here.

[0189] The method for obtaining the area ratio of the target object in the target sample image after inputting the multi-level fusion features into the object detection layer can be as follows: First, determine the area of ​​the target sample image and the area of ​​each target object in the target sample image to obtain the area ratio of each target object in the target sample image. Second, first determine the area of ​​each target sample image, then perform image size adjustment processing according to a preset target sample image size, scaling the target sample image proportionally to the same image size, and then determine the area of ​​each target object in the target sample image to obtain the area ratio of each target object in the target sample image.

[0190] In this process, after inputting the fusion features at each level into the corresponding object detection channel, in order to improve the detection accuracy for multiple object types, each object detection channel can be equipped with multiple detection heads. Each detection head corresponds one-to-one with the object type of the target object included in the target sample image. Through the detection head, the detection object type corresponding to the target object in the target sample image can be detected, and then the detection position of each target object in the target sample image can be detected.

[0191] For example, please refer to Figure 3a and Figure 3b , Figure 3a This disclosure provides a schematic diagram of the structure of a feature extraction and fusion layer in a neural network, according to an embodiment. Figure 3bThis is a schematic diagram of the object detection layer in a neural network provided in an embodiment of the present disclosure. The neural network includes an extraction and fusion layer and an object detection layer. This example uses a three-level feature extraction, a three-level feature fusion, and three object detection channels. Specifically, the extraction and fusion layer includes three feature extraction layers, three first feature fusion layers, and three second feature fusion layers, and the object detection layer includes three object detection channels. Accordingly, as... Figure 3a As shown, after the target sample image is input into the first feature extraction layer, the first-level image features can be obtained through feature extraction by the first feature extraction layer. The obtained first-level image features are then input into the second feature extraction layer. The first-level image features can be further processed by the second feature extraction layer to extract the second-level image features. Similarly, the second-level image features extracted by the second feature extraction layer can be input into the third feature extraction layer to extract the third-level image features. Furthermore, after inputting the first-level image features into the first-level first-feature fusion layer, the first-level image features can be initially fused through the first-level first-feature fusion layer, thereby obtaining the first-level intermediate fused features. After inputting the second-level image features and the first-level intermediate fused features into the second-level first-feature fusion layer, the second-level image features and the first-level intermediate fused features can be initially fused through the second-level first-feature fusion layer, thereby obtaining the second-level intermediate fused features. After inputting the third-level image features and the second-level intermediate fused features into the third-level first-feature fusion layer, the third-level image features and the second-level intermediate fused features can be initially fused through the third-level first-feature fusion layer, thereby obtaining the third-level intermediate fused features. Next, the third-level intermediate image features are input into the third-level second feature fusion layer. After passing through the third-level second feature fusion layer, the third-level intermediate fusion features can be further fused to obtain the third-level fusion features. After inputting the second-level intermediate fusion features and the third-level fusion features into the second-level second feature fusion layer, the second-level intermediate fusion features and the third-level fusion features can be further fused to obtain the second-level fusion features. After inputting the first-level intermediate fusion features and the second-level fusion features into the first-level second feature fusion layer, the first-level intermediate fusion features and the second-level fusion features can be further fused to obtain the first-level fusion features.

[0192] Here, depending on the actual detection requirements, the first layer first feature fusion layer, the second layer first feature fusion layer, the third layer first feature fusion layer, the first layer second feature fusion layer, the second layer second feature fusion layer, and the third layer second feature fusion layer can be convolutional layers, pooling layers, fully connected layers, etc., and no restrictions are imposed here.

[0193] In this way, by setting up the first feature fusion layer, the second feature fusion layer, and the third feature fusion layer, the localization features of the lower layer are passed to the higher layer. By setting up the first feature fusion layer, the second feature fusion layer, and the third feature fusion layer, the strong semantic features of the higher layer are passed to the lower layer, thereby improving the learning effect of the neural network for image features at different levels.

[0194] Then, as Figure 3b As shown, the first-level fusion features, second-level fusion features, and third-level fusion features can be input into the object detection layer respectively. In this example, the object detection layer includes three object detection channels, and each fusion feature is input into its corresponding object detection channel. For the first-level, second-level, and third-level fusion features, the image sizes corresponding to the three fusion features are different. Similarly, for the target sample image, the sizes of each target object in the target sample image are also different, meaning their area proportions in the target sample image are different. After the extraction, fusion, and transmission of image features, relative to the three fusion features with different image sizes, the original sizes of each target object in the feature image corresponding to each fusion feature will be enlarged or reduced accordingly. Furthermore, in the feature images corresponding to different fusion features, two target objects that were originally different sizes will become essentially the same. In other words, after the fusion processing of different levels of features, the required detection... Target objects with different area proportions can be considered to have roughly the same object image size. This can improve the balance of object detection. For example, if the target sample image contains three target objects, A, B, and C, with A being larger than B, B being larger than C, and their corresponding area proportions being different, after image feature extraction, fusion, and transfer, the object size of A in the feature image corresponding to the third-level feature fusion feature, the object size of B in the feature image corresponding to the second-level feature fusion feature, and the object size of C in the feature image corresponding to the first-level feature fusion feature can be considered to be roughly the same. Thus, through three object detection channels, objects with the same area proportion as A, B, and C can be detected respectively.

[0195] Accordingly, multiple detection heads can be set in each object detection channel. Each detection head can detect objects belonging to the same object type. In this example, each object detection channel includes N detection heads. Through the detection heads, the detection object type corresponding to the target object in the target sample image can be detected, and then the detection position of each target object in the target sample image can be detected, thereby obtaining the detection annotation information of each target object in the target sample image.

[0196] It can be understood that the image detection result output by the neural network for the target sample image may include the detection object type of the target object in the target sample image and the detection position of the target object in the target sample image.

[0197] Next, based on the image detection results and the target object annotation information, the neural network is adjusted. Specifically, the weights of each object detection channel can be adaptively adjusted. Optionally, the parameters can be adjusted manually or through artificial intelligence algorithms. No limitations are imposed here, thereby obtaining the adjusted neural network. The process then returns to the step of inputting the target sample image into the neural network for training until the training cutoff condition is met. Here, if the accuracy difference between the image detection results and the target object annotation information is less than a preset accuracy threshold, it is determined that the neural network meets the training cutoff condition, and the trained neural network is used as an image detection model for detecting objects in the image.

[0198] The neural network training method provided in this disclosure can acquire multiple training sample images with sample object annotation information in a target business scenario, and multiple auxiliary open-source images with auxiliary object annotation information in multiple business scenarios. The training sample images include at least one sample object in the target business scenario, and the auxiliary open-source images include at least one auxiliary object in at least one business scenario. Based on the auxiliary open-source images, the auxiliary object annotation information, the training sample images, and the sample object annotation information, a target sample image and target object annotation information for target objects of various object types in the target sample image are generated. The target objects include the sample objects and the auxiliary objects. The generated target sample images and target object annotation information are used to train the neural network to obtain a trained image detection model for detecting objects in images.

[0199] In this way, when training the neural network, the types and effects of target sample images used for training can be enriched, the richness of target sample images can be increased, the diversity of training data can be improved, the data imbalance between different categories of objects can be reduced, and the learning effect of the neural network on different image features can be improved. At the same time, by fusing and merging deep and shallow features at multiple scales, multiple object detection channels can be set to detect objects with different area proportions in a targeted manner, which greatly improves the image detection accuracy of the image detection model.

[0200] To facilitate understanding of this embodiment, a detailed description of the image detection method disclosed in this disclosure will be provided first. The execution entity of the image detection method provided in this disclosure can be an image detection device, for example, the image detection method can be executed by a terminal device, a server, or other processing devices. In some possible implementations, the image detection method can be implemented by a processor calling computer-readable instructions stored in memory.

[0201] Please see Figure 4 , Figure 4 This is a flowchart illustrating an image detection method provided in an embodiment of this disclosure. Figure 4 As shown in the embodiments of this disclosure, the image detection method includes:

[0202] S401: Obtain the image to be detected and the image detection model, wherein the image detection model is trained according to the neural network training method described above.

[0203] In this step, when image detection is required, the image to be detected and the image detection model trained according to the neural network training method described above can be obtained.

[0204] The image to be detected can be obtained by parsing the video to be identified by video acquisition devices added to the production environment, or it can be obtained by image acquisition devices added to the production environment; no limitation is made here.

[0205] S402: The image to be detected is detected by the image detection model to obtain the image detection result of the image to be detected.

[0206] In this step, after inputting the image to be detected into the image detection model, the image detection result for the image to be detected can be obtained.

[0207] Specifically, after obtaining the image to be detected, the image to be detected is input into a trained image detection model. The image detection model includes a feature extraction and fusion layer and an object detection layer. The feature fusion and extraction layer is used to extract multi-level image features from the image to be detected, and then the extracted multi-level image features are fused to obtain multi-level fused features. Here, the image size of the feature images indicated by different levels of fused features is different.

[0208] The object detection layer is used to perform object detection on multi-level fusion features. Specifically, the object detection layer includes multiple object detection channels. Different object detection channels are used to detect objects with different area proportions in the image. It can be understood that after the extraction, fusion, and transmission of image features, the original size of each target object in the feature image corresponding to each fusion feature will be enlarged or reduced accordingly, relative to the multi-level fusion features with different corresponding image sizes. In the feature images corresponding to different fusion features, the target objects with different sizes will be basically the same. Therefore, after the multi-level fusion features are input into the object detection layer, the object detection channel corresponding to each target object can be determined according to the area proportion range corresponding to each object detection channel. Thus, the fusion features at each level are input into the corresponding object detection channel. The object detection channel can detect objects with the same area proportion. The detection head set in the object detection channel corresponds one-to-one with the object type of the target object included in the image to be detected. Through the detection head, the detection object type corresponding to the target object in the image to be detected can be detected, and then the detection position of each target object in the target sample image can be detected, thereby obtaining the image detection result for the image to be detected.

[0209] Optionally, the image detection result may include the object type of the object to be detected and the marked position of the object in the image to be detected.

[0210] The image detection method provided in this disclosure can acquire an image to be detected and an image detection model, wherein the image detection model is trained according to the above-described neural network training method; and the image to be detected is detected by the image detection model to obtain the image detection result of the image to be detected.

[0211] In this way, when performing image detection using the image detection model, multiple object detection channels in the object detection layer of the image detection model, through fine-grained detection channel division, can specifically detect objects with different area proportions, which helps to improve the detection accuracy for objects with various area proportions and object types. This can effectively improve the accuracy of the detection results, reduce the impact of different object area proportions on the detection results, and have high anti-interference ability.

[0212] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0213] Based on the same inventive concept, this disclosure also provides a neural network training device corresponding to the neural network training method. Since the principle of the device in this disclosure for solving the problem is similar to that of the neural network training method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0214] Please see Figure 5 , Figure 5 This is a schematic diagram of a neural network training device provided in an embodiment of this disclosure. Figure 5 As shown in the illustration, the neural network training apparatus 500 provided in this embodiment includes:

[0215] The data acquisition module 510 is used to acquire multiple training sample images with sample object annotation information in the target business scenario, and multiple auxiliary open source images with auxiliary object annotation information in multiple business scenarios. The training sample images include at least one sample object in the target business scenario, and the auxiliary open source images include at least one auxiliary object in at least one business scenario.

[0216] Data generation module 520 is used to generate target sample images and target object annotation information for target objects of various object types in the target sample images based on the auxiliary open source image, the auxiliary object annotation information, the training sample image and the sample object annotation information, wherein the target objects include the sample objects and the auxiliary objects;

[0217] The model training module 530 is used to train the neural network using the generated target sample image and the corresponding target object annotation information to obtain a trained image detection model for detecting objects in the image.

[0218] In one optional implementation, the data acquisition module 510 is specifically used for:

[0219] Acquire images captured in the target business scenario;

[0220] The acquired images are filtered according to the image quality requirements to obtain acquired images that meet the image quality requirements;

[0221] Object annotation is performed on the acquired images that meet the image quality requirements to obtain training sample images and sample object annotation information of the sample objects in the training sample images. The sample object annotation information is used to indicate the object type of the sample object and the annotation position of the sample object in the training sample image.

[0222] In one optional implementation, the data generation module 520 is specifically used for:

[0223] Based on the auxiliary object annotation information and the sample object annotation information, the auxiliary object image of the auxiliary object is extracted from the auxiliary open source image;

[0224] Based on the sample object annotation information, the sample object image of the sample object is extracted from the training sample image;

[0225] Based on the auxiliary object image, the auxiliary object annotation information, the sample object image, the sample object annotation information, and the training sample image, a target sample image and target object annotation information for target objects of various object types in the target sample image are generated.

[0226] In an optional implementation, when the data generation module 520 extracts the auxiliary object image of the auxiliary object from the auxiliary open-source image based on the auxiliary object annotation information and the sample object annotation information, it is specifically used for:

[0227] The object type of the auxiliary object and the annotation position of the auxiliary object in the auxiliary open source image are determined from the auxiliary object annotation information, and the object type of the sample object is determined from the sample object annotation information;

[0228] Detect whether the object type of the auxiliary object is consistent with the object type of the sample object;

[0229] If they match, extract the auxiliary object image from the auxiliary open source image according to the labeled position corresponding to the auxiliary object.

[0230] In an optional implementation, when the data generation module 520 extracts the sample object image content of the sample object from the training sample image based on the sample object annotation information, it is specifically used for:

[0231] According to the annotation position of the sample object in the training sample image indicated by the annotation information of the sample object, the sample object image of the sample object is extracted from the training sample image.

[0232] In an optional implementation, the data generation module 520 is further configured to:

[0233] Obtain the preset object size corresponding to various object types in the target business scenario;

[0234] Based on the object type corresponding to the auxiliary object image, the object type corresponding to the sample object image, and the preset object size for each object type, the auxiliary object image and the sample object image are resized according to the corresponding preset object size to obtain the resized content of the auxiliary object image and the resized content of the sample object image.

[0235] In an optional implementation, when the data generation module 520 generates a target sample image and target object annotation information for target objects of various object types in the target sample image based on the auxiliary object image, the auxiliary object annotation information, the sample object image, the sample object annotation information, and the training sample image, it is specifically used for:

[0236] Based on the sample object annotation information, at least one additional object image is added to the blank image region other than the sample object image in the training sample image to obtain the target sample image. The additional object image includes the auxiliary object image and the sample object image.

[0237] Based on the object type of the added object image and the addition position in the training sample image, as well as the sample object annotation information of the training sample image, the target object annotation information of the target sample image is determined.

[0238] In an optional implementation, when the data generation module 520 adds at least one object image to a blank image region other than the object image in the training sample image based on the object annotation information to obtain a target sample image, it specifically performs the following:

[0239] Based on the object type indicated by the sample object annotation information, determine the type of object to be added to the training sample image corresponding to the object to be added;

[0240] For the added object image under the added object type, determine the number and position of the added object images added to the training sample images;

[0241] Based on the type of the added object, the number of added objects, and the location of the added objects, the corresponding added object images are added to the training sample images to obtain the target sample images.

[0242] In an optional implementation, the data generation module 520 is further configured to:

[0243] Based on the addition location, determine the proportion of the added object corresponding to the added object image in the training sample image;

[0244] The added object image is linearly scaled according to the object ratio to obtain the processed added object image.

[0245] In one optional implementation, the model training module 530 is specifically used for:

[0246] The target sample image is input into a neural network. The feature extraction and fusion layer of the neural network extracts and fuses multi-level image features of the target sample image to obtain multi-level fusion features generated during the fusion of the extracted multi-level image features. The image size of the feature image indicated by the fusion features at different levels is different.

[0247] The fusion features at each level are input to the corresponding object detection channels in the object detection layer of the neural network to obtain the detection annotation information of each target object in the target sample image. The object detection layer includes multiple object detection channels, and the area ratio of the objects in the image detected by different object detection channels is different.

[0248] Based on the detection annotation information of each target object in the target sample image, the image detection result of the target sample image output by the neural network is determined;

[0249] Based on the image detection results and the target object annotation information, the network parameters of the neural network are adjusted until the neural network meets the training cutoff condition. The trained neural network is then used as an image detection model for detecting objects in the image.

[0250] The neural network training apparatus provided in this disclosure can acquire multiple training sample images with sample object annotation information in a target business scenario, and multiple auxiliary open-source images with auxiliary object annotation information in multiple business scenarios. The training sample images include at least one sample object in the target business scenario, and the auxiliary open-source images include at least one auxiliary object in at least one business scenario. Based on the auxiliary open-source images, the auxiliary object annotation information, the training sample images, and the sample object annotation information, a target sample image and target object annotation information for target objects of various object types in the target sample image are generated. The target object includes the sample object and the auxiliary object. The generated target sample image and the target object annotation information are used to train the neural network to obtain a trained image detection model for detecting objects in the image.

[0251] In this way, when training the neural network, the types and effects of target sample images used for training can be enriched, the richness of target sample images can be increased, the diversity of training data can be improved, the data imbalance between different categories of objects can be reduced, and the learning effect of the neural network on different image features can be improved. At the same time, by fusing and merging deep and shallow features at multiple scales, multiple object detection channels can be set to detect objects with different area proportions in a targeted manner, which greatly improves the image detection accuracy of the image detection model.

[0252] Based on the same inventive concept, this disclosure also provides an image detection device corresponding to the image detection method. Since the principle of the device in this disclosure for solving the problem is similar to that of the image detection method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0253] Please see Figure 6 , Figure 6 This is a schematic diagram of an image detection device provided in an embodiment of this disclosure. Figure 6 As shown in the figure, the image detection apparatus 600 provided in this embodiment includes:

[0254] The image acquisition module 610 is used to acquire the image to be detected and the image detection model, wherein the image detection model is trained according to the neural network training device described above.

[0255] The image detection module 620 is used to detect the image to be detected using the image detection model, and obtain the image detection result of the image to be detected.

[0256] The image detection apparatus provided in this embodiment can acquire an image to be detected and an image detection model, wherein the image detection model is trained according to the above-described neural network training method; and the image to be detected is detected by the image detection model to obtain the image detection result of the image to be detected.

[0257] In this way, when performing image detection using the image detection model, multiple object detection channels in the object detection layer of the image detection model, through the division of fine-grained detection channels, can specifically detect objects with different area proportions, which helps to improve the detection accuracy for objects with various area proportions and object types. This can effectively improve the accuracy of the detection results, reduce the impact of different object area proportions on the detection results, and have high anti-interference ability.

[0258] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0259] Corresponding to the aforementioned neural network training method or image detection method, this disclosure also provides an electronic device 700, such as... Figure 7 The diagram shown is a structural schematic of an electronic device 700 provided in an embodiment of this disclosure, including:

[0260] The device includes a processor 710, a memory 720, and a bus 730. The memory 720 stores execution instructions and includes a main memory 721 and an external memory 722. The main memory 721, also known as internal memory, is used to temporarily store computational data in the processor 710 and data exchanged with external memory such as a hard disk. The processor 710 exchanges data with the external memory 722 through the main memory 721. When the electronic device 700 is running, the processor 710 and the memory 720 communicate through the bus 730, enabling the processor 710 to execute the steps of the aforementioned neural network training method or image detection method.

[0261] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 700. In other embodiments of this application, the electronic device 700 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0262] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the neural network training method or image detection method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0263] This disclosure also provides a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, they can perform the steps of the neural network training method or image detection method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0264] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0265] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices and apparatuses described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0266] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0267] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0268] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0269] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A neural network training method, characterized in that, The method includes: Acquire multiple training sample images with sample object annotation information in the target business scenario, and multiple auxiliary open source images with auxiliary object annotation information in multiple business scenarios. The training sample images include at least one sample object in the target business scenario, and the auxiliary open source images include at least one auxiliary object in at least one business scenario. Based on the auxiliary object annotation information and the sample object annotation information, an auxiliary object image of the auxiliary object is extracted from the auxiliary open source image, wherein the auxiliary object image is extracted by cutting along the outer edge line of the auxiliary object; Based on the sample object annotation information, a sample object image of the sample object is extracted from the training sample image, wherein the sample object image is extracted by cutting along the outer edge line of the sample object; Based on the auxiliary object image, the auxiliary object annotation information, the sample object image, and the sample object annotation information, data augmentation processing is performed on the training sample image to generate a target sample image and target object annotation information for target objects of various object types in the target sample image. The target object includes the sample object and the auxiliary object. The neural network is trained using the generated target sample image and the target object annotation information to obtain a trained image detection model for detecting objects in the image.

2. The method according to claim 1, characterized in that, The acquisition of multiple training sample images with sample object annotation information in the target business scenario includes: Acquire images captured in the target business scenario; The acquired images are filtered according to the image quality requirements to obtain acquired images that meet the image quality requirements; Object annotation is performed on the acquired images that meet the image quality requirements to obtain training sample images and sample object annotation information of the sample objects in the training sample images. The sample object annotation information is used to indicate the object type of the sample object and the annotation position of the sample object in the training sample image.

3. The method according to claim 1, characterized in that, The step of extracting the auxiliary object image from the auxiliary open-source image based on the auxiliary object annotation information and the sample object annotation information includes: The object type of the auxiliary object and the annotation position of the auxiliary object in the auxiliary open source image are determined from the auxiliary object annotation information, and the object type of the sample object is determined from the sample object annotation information; Detect whether the object type of the auxiliary object is consistent with the object type of the sample object; If they match, extract the auxiliary object image from the auxiliary open source image according to the labeled position corresponding to the auxiliary object.

4. The method according to claim 1, characterized in that, The step of extracting the sample object image content of the sample object from the training sample image based on the sample object annotation information includes: According to the annotation position of the sample object in the training sample image indicated by the annotation information of the sample object, the sample object image of the sample object is extracted from the training sample image.

5. The method according to claim 1, characterized in that, Before generating the target sample image and the target object annotation information for target objects of various object types in the target sample image, the method further includes: Obtain the preset object size corresponding to various object types in the target business scenario; Based on the object type corresponding to the auxiliary object image, the object type corresponding to the sample object image, and the preset object size for each object type, the auxiliary object image and the sample object image are resized according to the corresponding preset object size to obtain the resized content of the auxiliary object image and the resized content of the sample object image.

6. The method according to claim 1, characterized in that, The step of performing data augmentation processing on the training sample image based on the auxiliary object image, the auxiliary object annotation information, the sample object image, and the sample object annotation information to generate a target sample image and target object annotation information for target objects of various object types in the target sample image includes: Based on the sample object annotation information, at least one additional object image is added to the blank image region other than the sample object image in the training sample image to obtain the target sample image. The additional object image includes the auxiliary object image and the sample object image. Based on the object type of the added object image and the addition position in the training sample image, as well as the sample object annotation information of the training sample image, the target object annotation information of the target sample image is determined.

7. The method according to claim 6, characterized in that, The step of adding at least one object image to the blank image region other than the object image in the training sample image based on the sample object annotation information to obtain the target sample image includes: Based on the object type indicated by the sample object annotation information, determine the type of object to be added to the training sample image corresponding to the object to be added; For the added object image under the added object type, determine the number and position of the added object images added to the training sample images; Based on the type of the added object, the number of added objects, and the location of the added objects, the corresponding added object images are added to the training sample images to obtain the target sample images.

8. The method according to claim 7, characterized in that, Before adding the corresponding object image to the training sample image based on the added object type, the added quantity, and the added position to obtain the target sample image, the method further includes: Based on the addition location, determine the proportion of the added object corresponding to the added object image in the training sample image; The added object image is linearly scaled according to the object ratio to obtain the processed added object image.

9. The method according to claim 1, characterized in that, The step of training the neural network using the generated target sample image and the target object annotation information to obtain a trained image detection model for detecting objects in the image includes: The target sample image is input into a neural network. The feature extraction and fusion layer of the neural network extracts and fuses multi-level image features of the target sample image to obtain multi-level fusion features generated during the fusion of the extracted multi-level image features. The feature images indicated by different levels of fusion features have different image sizes. The fusion features at each level are input to the corresponding object detection channels in the object detection layer of the neural network to obtain the detection annotation information of each target object in the target sample image. The object detection layer includes multiple object detection channels, and the area ratio of the objects in the image detected by different object detection channels is different. Based on the detection annotation information of each target object in the target sample image, the image detection result of the target sample image output by the neural network is determined; Based on the image detection results and the target object annotation information, the network parameters of the neural network are adjusted until the neural network meets the training cutoff condition. The trained neural network is then used as an image detection model for detecting objects in the image.

10. An image detection method, characterized in that, The method includes: Obtain the image to be detected and the image detection model, wherein the image detection model is trained according to the method of any one of claims 1 to 9; The image to be detected is detected by the image detection model to obtain the image detection result of the image to be detected.

11. A neural network training device, characterized in that, The device includes: The data acquisition module is used to acquire multiple training sample images with sample object annotation information in the target business scenario, and multiple auxiliary open source images with auxiliary object annotation information in multiple business scenarios. The training sample images include at least one sample object in the target business scenario, and the auxiliary open source images include at least one auxiliary object in at least one business scenario. The data generation module is used to: extract an auxiliary object image of the auxiliary object from the auxiliary open-source image based on the auxiliary object annotation information and the sample object annotation information, wherein the auxiliary object image is extracted by cutting along the outer edge line of the auxiliary object; extract a sample object image of the sample object from the training sample image based on the sample object annotation information, wherein the sample object image is extracted by cutting along the outer edge line of the sample object; and perform data augmentation processing on the training sample image based on the auxiliary object image, the auxiliary object annotation information, the sample object image, and the sample object annotation information to generate a target sample image and target object annotation information for target objects of various object types in the target sample image, wherein the target object includes the sample object and the auxiliary object. The model training module is used to train the neural network using the generated target sample image and the corresponding target object annotation information to obtain a trained image detection model for detecting objects in the image.

12. An image detection device, characterized in that, The device includes: An image acquisition module is used to acquire an image to be detected and an image detection model, wherein the image detection model is trained by the neural network training device according to claim 11. The image detection module is used to detect the image to be detected using the image detection model, and obtain the image detection result of the image to be detected.

13. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the neural network training method as described in any one of claims 1 to 9 or the image detection method as described in claim 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the neural network training method as described in any one of claims 1 to 9 or the image detection method as described in claim 10.

15. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the neural network training method according to any one of claims 1 to 9 or the image detection method according to claim 10.

Images