Deep learning model training method, object detection method, apparatus, and electronic device

Abstract

The embodiment of the application provides a deep learning model training method, an object detection method, an apparatus and an electronic device, relates to the technical field of computers, and adjusts the resolution of the obtained to-be-processed image data of multiple data sources according to a preset resolution, thereby obtaining first image data, so that the resolution of the final sample image data is adjusted according to the preset resolution, and based on attribute information of each data source, the fusion proportion of each data source is determined, for each data source, a corresponding number of first image data is selected from the first image data of the data source according to the fusion proportion of the data source, sample image data of the data source is obtained, so that the data of multiple data sources is integrated, and finally, the deep learning model is trained by using the sample image data of each data source obtained through integration, so that the deep learning model training effect can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a deep learning model training method, an object detection method, an apparatus, and an electronic device. Background Technology

[0002] Deep learning requires training based on image data. However, image data sampling, image data annotation, and the collection of diverse image data all require a significant amount of human and material resources.

[0003] For data from different sources, such as images, if high-resolution and low-resolution image data are crudely mixed for training, the detected objects will hardly be trained because their proportion is too small. Even if they are involved in training, the effective feature information of the detected objects will be blurred due to multiple downsampling, and too much effective information will be lost, resulting in poor training performance and low accuracy of deep learning model recognition.

[0004] Therefore, how to integrate data from different sources to improve the training effect of deep learning models has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a deep learning model training method, an object detection method, a device, and an electronic device to solve the problem of integrating data from different sources. The specific technical solution is as follows:

[0006] This application provides a method for training a deep learning model, the method comprising:

[0007] Acquire image data to be processed from multiple data sources;

[0008] The resolution of the image data to be processed is adjusted according to a preset resolution to obtain the first image data;

[0009] Based on the attribute information of each data source, the fusion ratio of each data source is determined;

[0010] For each data source, according to the fusion ratio of that data source, a corresponding number of first image data are selected from the first image data of that data source to obtain the sample image data of that data source;

[0011] The deep learning model is trained using sample image data from each of the aforementioned data sources to obtain a trained deep learning model.

[0012] In one possible embodiment, when the plurality of data sources include video data sources; the step of acquiring the image data to be processed from the plurality of data sources includes:

[0013] Obtain video data from the aforementioned video data source;

[0014] At least one video frame is extracted from the video data and added to the first image set;

[0015] From the remaining video frames of the video data, select the unselected video frames to obtain the current video frame to be processed, wherein the remaining video frames are the video frames that have not been extracted from the video data;

[0016] Calculate the similarity between the current video frame to be processed and each video frame in the first image set; if the similarity between the current video frame to be processed and each video frame in the first image set is less than a preset similarity threshold, then add the current video frame to be processed to the first image set.

[0017] Return step: Select unselected video frames from the remaining video frames of the video data to obtain the current video frame to be processed and continue execution until all video frames in the remaining video frames have been selected or the number of video frames in the first image set reaches a first quantity threshold, thereby obtaining the image data to be processed from the video data source, wherein the image data to be processed from the video data source is each video frame in the first image set.

[0018] In one possible embodiment, adjusting the resolution of the image data to be processed according to a preset resolution to obtain the first image data includes:

[0019] For each image data to be processed, if the resolution of the image data to be processed is lower than the preset resolution, the image data to be processed is filled or enlarged according to the preset resolution to obtain the first image data.

[0020] If the resolution of the image data to be processed is greater than the preset resolution, the image data to be processed is cropped according to the preset resolution to obtain the first image data.

[0021] In one possible embodiment, after cropping the image data to be processed according to the preset resolution to obtain the first image data, the method further includes:

[0022] Calculate the intersection-union ratio (IUU) of the labeled object regions in the first image data and the labeled object regions in the image data to be processed. If the IUU is greater than a preset ratio threshold, retain the labels in the first image data.

[0023] If the cross-union ratio is not greater than a preset ratio threshold, the labeled object regions in the first image data are filled with random pixels, and the labels of the first image data are modified to negative samples.

[0024] In one possible embodiment, the attribute information of the data source represents the diversity of the labeled objects in the data source;

[0025] Determining the fusion ratio of each type of data source based on the attribute information of each data source includes:

[0026] For each data source, the fusion ratio of that data source is determined based on the diversity of the labeled objects in that data source. The fusion ratio of that data source is positively correlated with the diversity of the labeled objects in that data source.

[0027] In one possible embodiment, the step of selecting a corresponding number of first image data from the first image data of each data source according to the fusion ratio of that data source to obtain sample image data of that data source includes:

[0028] For each data source, determine the number of images to be selected from that data source based on its fusion ratio;

[0029] For each type of first image data in the data source, determine the category proportion of the first image data based on the object information of the labeled objects in that type of first image data;

[0030] Based on the number of images selected from the data source to which the first image data belongs and the category ratio of the first image data, a corresponding number of sample image data are selected from the first image data.

[0031] In one possible embodiment, the object information of the labeled object includes the shape richness, scale, and aspect ratio of the labeled object;

[0032] For each category of first image data in the data source, the percentage of that category of first image data is determined based on the object information of the labeled objects in that category of first image data, including:

[0033] For each type of first image data in the data source, the first proportion of the first image data of that type is determined based on the morphological richness of the labeled objects in the first image data of that type, wherein the morphological richness of the first image data of that type is positively correlated with the first proportion;

[0034] Based on the scale of the labeled objects in the first image data of this type, a second proportion of the first image data of this type is determined, wherein the scale of the first image data of this type is negatively correlated with the second proportion;

[0035] Based on the aspect ratio of the labeled objects in the first image data of this type, the third proportion of the first image data of this type is determined, wherein the aspect ratio of the first image data of this type is negatively correlated with the third proportion;

[0036] The category proportion of the first image data is determined based on the first proportion, second proportion, and third proportion of the first image data.

[0037] This application embodiment also provides an object detection method, the method comprising:

[0038] Acquire the image data to be detected;

[0039] Using a pre-trained deep learning model, the object detection result of the image data to be detected is predicted, wherein the deep learning model is trained by any of the deep learning model training methods described above.

[0040] In one possible embodiment, the image data to be detected is a video frame in the video data to be detected; the step of using a pre-trained deep learning model to predict the object detection result of the image data to be detected includes:

[0041] The image data to be detected is segmented into a background to obtain static and moving regions;

[0042] Based on the type of the object to be detected, a first region to which the object to be detected belongs is determined in the static region and the moving region;

[0043] The object detection result of the image data to be detected is obtained by using a pre-trained deep learning model to predict the first region.

[0044] In one possible embodiment, the image data to be detected is a video frame in the video data to be detected; the step of using a pre-trained deep learning model to predict the object detection result of the image data to be detected includes:

[0045] The image data to be detected is segmented into a background to obtain static and moving regions;

[0046] Based on the type of the object to be detected, a first region to which the object to be detected belongs is determined in the static region and the moving region;

[0047] When the image data to be detected is a first type of video frame, a pre-trained deep learning model is used to predict the first region to obtain a first detection result of the image data to be detected; the first detection result of the image data to be detected is merged with the most recently obtained second detection result to obtain an object detection result of the image data to be detected; wherein, the first type of video frame is a video frame obtained by extracting frames from the video data to be detected according to a first frame rate.

[0048] When the image data to be detected is a second type of video frame, a pre-trained deep learning model is used to predict the second region to obtain a second detection result of the image data to be detected; the second detection result of the image data to be detected is merged with the most recently obtained first detection result to obtain the object detection result of the image data to be detected; wherein, the second type of video frame is a video frame obtained by extracting frames from the video data to be detected according to a second frame rate; the second region is the static region and another region in the moving region other than the first region; the first frame rate is greater than the second frame rate.

[0049] In one possible embodiment, the image data to be detected is a video frame in the video data to be detected; the step of using a pre-trained deep learning model to predict the object detection result of the image data to be detected includes:

[0050] Based on the existing detection results of the object to be detected in the image data to be detected, a third region in which the object to be detected may appear in the image data to be detected is determined;

[0051] The object detection result of the image data to be detected is obtained by using a pre-trained deep learning model to predict the third region.

[0052] In one possible embodiment, acquiring the image data to be detected includes:

[0053] According to the preset frame extraction rules, extract the image data to be detected from the video data to be detected;

[0054] The method further includes:

[0055] Based on the location information of the object to be detected in the image data to be detected, predict the location information of the object to be detected in other video frames of other video data to be detected;

[0056] A sliding window detection method is used to filter the position information of the object to be detected.

[0057] In one possible embodiment, after the step of acquiring the image data to be detected, the method further includes:

[0058] The slice size of the image data to be detected is determined based on the resolution of the image data to be detected and the maximum downsampling ratio of the pre-trained deep learning model.

[0059] According to the slice size, the image data to be detected is cropped into multiple image slices;

[0060] The method of using a pre-trained deep learning model to predict the object detection result of the image data to be detected includes:

[0061] The inference results for each image slice are obtained by using a pre-trained deep learning model to perform inference on multiple image slices.

[0062] The inference results of each image slice are merged, fused, and deduplicated to obtain the object detection result of the image data to be detected.

[0063] This application embodiment also provides a deep learning model training apparatus, the apparatus comprising:

[0064] Acquire image data to be processed from multiple data sources;

[0065] The resolution of the image data to be processed is adjusted according to a preset resolution to obtain the first image data;

[0066] Based on the attribute information of each data source, the fusion ratio of each data source is determined;

[0067] For each data source, according to the fusion ratio of that data source, a corresponding number of first image data are selected from the first image data of that data source to obtain the sample image data of that data source;

[0068] The deep learning model is trained using sample image data from each of the aforementioned data sources to obtain a trained deep learning model.

[0069] In one possible embodiment, when the plurality of data sources include video data sources; the step of acquiring the image data to be processed from the plurality of data sources includes:

[0070] Obtain video data from the aforementioned video data source;

[0071] At least one video frame is extracted from the video data and added to the first image set;

[0072] From the remaining video frames of the video data, select the unselected video frames to obtain the current video frame to be processed, wherein the remaining video frames are the video frames that have not been extracted from the video data;

[0073] Calculate the similarity between the current video frame to be processed and each video frame in the first image set; if the similarity between the current video frame to be processed and each video frame in the first image set is less than a preset similarity threshold, then add the current video frame to be processed to the first image set.

[0074] Return step: Select unselected video frames from the remaining video frames of the video data to obtain the current video frame to be processed and continue execution until all video frames in the remaining video frames have been selected or the number of video frames in the first image set reaches a first quantity threshold, thereby obtaining the image data to be processed from the video data source, wherein the image data to be processed from the video data source is each video frame in the first image set.

[0075] In one possible embodiment, adjusting the resolution of the image data to be processed according to a preset resolution to obtain the first image data includes:

[0076] For each image data to be processed, if the resolution of the image data to be processed is lower than the preset resolution, the image data to be processed is filled or enlarged according to the preset resolution to obtain the first image data.

[0077] If the resolution of the image data to be processed is greater than the preset resolution, the image data to be processed is cropped according to the preset resolution to obtain the first image data.

[0078] In one possible embodiment, after cropping the image data to be processed according to the preset resolution to obtain the first image data, the device further includes:

[0079] Calculate the intersection-union ratio (IUU) of the labeled object regions in the first image data and the labeled object regions in the image data to be processed. If the IUU is greater than a preset ratio threshold, retain the labels in the first image data.

[0080] If the cross-union ratio is not greater than a preset ratio threshold, the labeled object regions in the first image data are filled with random pixels, and the labels of the first image data are modified to negative samples.

[0081] In one possible embodiment, the attribute information of the data source represents the diversity of the labeled objects in the data source;

[0082] Determining the fusion ratio of each type of data source based on the attribute information of each data source includes:

[0083] For each data source, the fusion ratio of that data source is determined based on the diversity of the labeled objects in that data source. The fusion ratio of that data source is positively correlated with the diversity of the labeled objects in that data source.

[0084] In one possible embodiment, the step of selecting a corresponding number of first image data from the first image data of each data source according to the fusion ratio of that data source to obtain sample image data of that data source includes:

[0085] For each data source, determine the number of images to be selected from that data source based on its fusion ratio;

[0086] For each type of first image data in the data source, determine the category proportion of the first image data based on the object information of the labeled objects in that type of first image data;

[0087] Based on the number of images selected from the data source to which the first image data belongs and the category ratio of the first image data, a corresponding number of sample image data are selected from the first image data.

[0088] In one possible embodiment, the object information of the labeled object includes the shape richness, scale, and aspect ratio of the labeled object;

[0089] For each category of first image data in the data source, the percentage of that category of first image data is determined based on the object information of the labeled objects in that category of first image data, including:

[0090] For each type of first image data in the data source, the first proportion of the first image data of that type is determined based on the morphological richness of the labeled objects in the first image data of that type, wherein the morphological richness of the first image data of that type is positively correlated with the first proportion;

[0091] Based on the scale of the labeled objects in the first image data of this type, a second proportion of the first image data of this type is determined, wherein the scale of the first image data of this type is negatively correlated with the second proportion;

[0092] Based on the aspect ratio of the labeled objects in the first image data of this type, the third proportion of the first image data of this type is determined, wherein the aspect ratio of the first image data of this type is negatively correlated with the third proportion;

[0093] The category proportion of the first image data is determined based on the first proportion, second proportion, and third proportion of the first image data.

[0094] This application embodiment also provides a deep learning model training apparatus, the apparatus comprising:

[0095] The acquisition module is used to acquire image data to be processed from multiple data sources;

[0096] An adjustment module is used to adjust the resolution of the image data to be processed according to a preset resolution to obtain the first image data;

[0097] The determination module is used to determine the fusion ratio of each of the data sources based on the attribute information of each data source;

[0098] The selection module is used to select a corresponding number of first image data from the first image data of each data source according to the fusion ratio of that data source, so as to obtain the sample image data of that data source.

[0099] The training module is used to train the deep learning model using sample image data from each of the aforementioned data sources, thereby obtaining a trained deep learning model.

[0100] In one possible embodiment, the acquisition module is specifically used for:

[0101] In the case where the multiple data sources include video data sources; obtain the video data from the video data source;

[0102] At least one video frame is extracted from the video data and added to the first image set;

[0103] From the remaining video frames of the video data, select the unselected video frames to obtain the current video frame to be processed, wherein the remaining video frames are the video frames that have not been extracted from the video data;

[0104] Calculate the similarity between the current video frame to be processed and each video frame in the first image set; if the similarity between the current video frame to be processed and each video frame in the first image set is less than a preset similarity threshold, then add the current video frame to be processed to the first image set.

[0105] Return step: Select unselected video frames from the remaining video frames of the video data to obtain the current video frame to be processed and continue execution until all video frames in the remaining video frames have been selected or the number of video frames in the first image set reaches a first quantity threshold, thereby obtaining the image data to be processed from the video data source, wherein the image data to be processed from the video data source is each video frame in the first image set.

[0106] In one possible embodiment, the adjustment module is specifically used for:

[0107] For each image data to be processed, if the resolution of the image data to be processed is lower than the preset resolution, the image data to be processed is filled or enlarged according to the preset resolution to obtain the first image data.

[0108] If the resolution of the image data to be processed is greater than the preset resolution, the image data to be processed is cropped according to the preset resolution to obtain the first image data.

[0109] In one possible embodiment, the device further includes:

[0110] The calculation module is used to calculate the intersection-union ratio (IUU) of the labeled object regions in the first image data and the labeled object regions in the image data to be processed after cropping the image data to be processed according to the preset resolution to obtain the first image data. If the IUU is greater than a preset ratio threshold, the labels of the first image data are retained.

[0111] The filling module is used to fill the labeled object region in the first image data with random pixels when the cross-union ratio is not greater than a preset ratio threshold, and to modify the label of the first image data to a negative sample.

[0112] In one possible embodiment, the attribute information of the data source represents the diversity of labeled objects in the data source; the determining module is specifically used for:

[0113] For each data source, the fusion ratio of that data source is determined based on the diversity of the labeled objects in that data source. The fusion ratio of that data source is positively correlated with the diversity of the labeled objects in that data source.

[0114] In one possible embodiment, the selection module is specifically used for:

[0115] For each data source, determine the number of images to be selected from that data source based on its fusion ratio;

[0116] For each type of first image data in the data source, determine the category proportion of the first image data based on the object information of the labeled objects in that type of first image data;

[0117] Based on the number of images selected from the data source to which the first image data belongs and the category ratio of the first image data, a corresponding number of sample image data are selected from the first image data.

[0118] In one possible embodiment, the object information of the labeled object includes the shape richness, scale, and aspect ratio of the labeled object;

[0119] The selection module is specifically used for:

[0120] For each type of first image data in the data source, the first proportion of the first image data of that type is determined based on the morphological richness of the labeled objects in the first image data of that type, wherein the morphological richness of the first image data of that type is positively correlated with the first proportion;

[0121] Based on the scale of the labeled objects in the first image data of this type, a second proportion of the first image data of this type is determined, wherein the scale of the first image data of this type is negatively correlated with the second proportion;

[0122] Based on the aspect ratio of the labeled objects in the first image data of this type, the third proportion of the first image data of this type is determined, wherein the aspect ratio of the first image data of this type is negatively correlated with the third proportion;

[0123] The category proportion of the first image data is determined based on the first proportion, second proportion, and third proportion of the first image data.

[0124] In one possible embodiment, the training module is specifically used for:

[0125] The slice size of the sample image is determined based on the resolution of the sample image data and the maximum downsampling ratio of the deep learning model.

[0126] Select one sample image from each of the sample image data;

[0127] According to the slice size, the currently selected sample image data is cropped into multiple sample image slices;

[0128] The deep learning model is used to infer the results of each sample image slice.

[0129] The inference results of each sample image slice are merged, fused, and deduplicated to obtain the inference result of the currently selected sample image data;

[0130] Based on the inference results and annotations of the currently selected sample image data, adjust the parameters of the deep learning model; return step: select one sample image data from each of the sample image data and continue execution until the preset termination condition is met to obtain the trained deep learning model.

[0131] This application embodiment also provides an object detection device, the device comprising:

[0132] The acquisition module is used to acquire the image data to be detected;

[0133] The prediction module is used to predict the object detection result of the image data to be detected using a pre-trained deep learning model, wherein the deep learning model is trained by any of the deep learning model training devices described above.

[0134] In one possible embodiment, the image data to be detected is a video frame in the video data to be detected; the prediction module is specifically used for:

[0135] The image data to be detected is segmented into a background to obtain static and moving regions;

[0136] Based on the type of the object to be detected, a first region to which the object to be detected belongs is determined in the static region and the moving region;

[0137] The object detection result of the image data to be detected is obtained by using a pre-trained deep learning model to predict the first region.

[0138] In one possible embodiment, the image data to be detected is a video frame in the video data to be detected; the prediction module is specifically used for:

[0139] The image data to be detected is segmented into a background to obtain static and moving regions;

[0140] Based on the type of the object to be detected, a first region to which the object to be detected belongs is determined in the static region and the moving region;

[0141] When the image data to be detected is a first type of video frame, a pre-trained deep learning model is used to predict the first region to obtain a first detection result of the image data to be detected; the first detection result of the image data to be detected is merged with the most recently obtained second detection result to obtain an object detection result of the image data to be detected; wherein, the first type of video frame is a video frame obtained by extracting frames from the video data to be detected according to a first frame rate.

[0142] When the image data to be detected is a second type of video frame, a pre-trained deep learning model is used to predict the second region to obtain a second detection result of the image data to be detected; the second detection result of the image data to be detected is merged with the most recently obtained first detection result to obtain the object detection result of the image data to be detected; wherein, the second type of video frame is a video frame obtained by extracting frames from the video data to be detected according to a second frame rate; the second region is the static region and another region in the moving region other than the first region; the first frame rate is greater than the second frame rate.

[0143] In one possible embodiment, the image data to be detected is a video frame in the video data to be detected; the prediction module is specifically used for:

[0144] Based on the existing detection results of the object to be detected in the image data to be detected, a third region in which the object to be detected may appear in the image data to be detected is determined;

[0145] The object detection result of the image data to be detected is obtained by using a pre-trained deep learning model to predict the third region.

[0146] In one possible embodiment, the acquisition module is specifically used for;

[0147] According to the preset frame extraction rules, extract the image data to be detected from the video data to be detected;

[0148] The device further includes:

[0149] The generation module is used to predict the position information of the object to be detected in other video frames of other video data to be detected based on the position information of the object detection result in the image data to be detected;

[0150] The filtering module is used to filter the position information of the object to be detected using a sliding window detection method.

[0151] In one possible embodiment, the device further includes:

[0152] The slice size determination module is used to determine the slice size of the image data to be detected based on the resolution of the image data to be detected and the maximum downsampling ratio of the pre-trained deep learning model after the step of acquiring the image data to be detected.

[0153] The cropping module is used to crop the image data to be detected into multiple image slices according to the slice size;

[0154] The prediction module is specifically used for:

[0155] The inference results for each image slice are obtained by using a pre-trained deep learning model to perform inference on multiple image slices.

[0156] The inference results of each image slice are merged, fused, and deduplicated to obtain the object detection result of the image data to be detected.

[0157] This application also provides an electronic device, including: a processor and a memory;

[0158] Memory, used to store computer programs;

[0159] The processor, when executing a program stored in memory, implements any of the deep learning model training methods or object detection methods described above.

[0160] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the deep learning model training methods or object detection methods described above.

[0161] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the deep learning model training methods or object detection methods described above.

[0162] Beneficial effects of the embodiments in this application:

[0163] The deep learning model training method, object detection method, apparatus, and electronic device provided in this application adjust the resolution of image data to be processed from multiple data sources according to a preset resolution to obtain first image data. This ensures that the resolution of the final sample image data is adjusted according to the preset resolution. Furthermore, based on the attribute information of each data source, the fusion ratio of each data source is determined. For each data source, a corresponding number of first image data are selected from the first image data of that data source according to the fusion ratio of that data source to obtain sample image data of that data source. This integrates data from multiple data sources. Finally, the integrated sample image data from each data source is used to train the deep learning model, thereby improving the training effect of the deep learning model.

[0164] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0165] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0166] Figure 1a A schematic diagram of a first method for training a deep learning model provided in an embodiment of this application;

[0167] Figure 1b This is a schematic diagram of a second process for a deep learning model training method provided in an embodiment of this application;

[0168] Figure 2This is a schematic flowchart of an object detection method provided in an embodiment of this application;

[0169] Figure 3 A schematic diagram of a deep learning model training device provided in an embodiment of this application;

[0170] Figure 4 This is a schematic diagram of the structure of an object detection device provided in an embodiment of this application;

[0171] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0172] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0173] To integrate data from different sources, embodiments of this application provide a deep learning model training method, an object detection method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product containing instructions.

[0174] The deep learning model training method provided in this application embodiment will be described below. This method can be applied to any electronic device that can provide deep learning model training services, such as a personal computer or a server. The deep learning model training method provided in this application embodiment can be implemented in at least one of software, hardware circuits, and logic circuits within an electronic device.

[0175] like Figure 1a As shown, Figure 1a This is a schematic diagram of a first flowchart of a deep learning model training method provided in an embodiment of this application. The method includes:

[0176] S110, acquire image data to be processed from multiple data sources;

[0177] S120, the resolution of the above-mentioned image data to be processed is adjusted according to the preset resolution to obtain the first image data;

[0178] S130, Based on the attribute information of each of the above data sources, determine the fusion ratio of each of the above data sources;

[0179] S140, For each data source, according to the fusion ratio of that data source, select a corresponding number of first image data from the first image data of that data source to obtain the sample image data of that data source;

[0180] S150, the deep learning model is trained using sample image data from the aforementioned data sources to obtain a trained deep learning model.

[0181] The image data to be processed comes from multiple data sources. These sources can be any open-source data or user-collected and labeled data, depending on actual needs and not limited here. It's understood that open-source image data includes annotations. These annotations target objects in the image, such as the region where the object is located and its type. Objects can be anything to be detected, such as people, animals, plants, buildings, traffic signs, cars, bicycles, motorcycles, etc., and can be configured according to specific circumstances. For example, in a traffic sign detection scenario, the object could be a traffic sign, and the annotation data for the image data to be processed could be the bounding box of the traffic sign in the image, and could also include the sign type. Similarly, in a car detection scenario, the object could be a car, and the annotation data for the image data to be processed could be the bounding box of the car in the image.

[0182] The resolution of image data to be processed may vary significantly depending on the data source. For example, publicly available datasets typically have lower resolutions, such as 500*300 or 1280*720. Self-collected image data, on the other hand, often has higher resolutions, such as 4096*2160, 3840*2160, or 2666*1520. To address this resolution difference issue, the resolution of the image data to be processed needs to be adjusted according to a preset resolution, ensuring that the resolution of the first image data is adjusted based on this preset resolution. This preset resolution can be a single resolution or a resolution range. For instance, if the preset resolution is a fixed value, and the resolution of the image data to be processed is lower or higher than this preset resolution, then the resolution of the image data to be processed needs to be adjusted. Similarly, if the preset resolution is a resolution range, and the resolution of the image data to be processed is not within this range, then the resolution of the image data to be processed needs to be adjusted to fall within the preset resolution range. The specific adjustment method can be set based on practical experience and is not limited here. To ensure clarity and layout, the following description, in conjunction with another embodiment, details how to adjust the resolution of the image data to be processed according to a preset resolution.

[0183] The attribute information of a data source characterizes data diversity. Specifically, data attribute information includes the region to which the data belongs and data collection status information, including the number of data collectors and the data collection period. For example, given data source A and data source B, data source A is highly diverse because its data was collected from multiple provinces by multiple people over a long period. Data source B, on the other hand, was collected from a few intersections by a small number of people over a short period, resulting in less diversity compared to data source A. Based on this attribute information, the fusion ratio of data sources can be determined, with a higher fusion ratio for data sources with better diversity than for those with poor diversity. For instance, if 1000 sample images are needed, and data source A has better diversity than data source B, different fusion ratios can be assigned to data sources A and B. For example, if the fusion ratio for data source A is 0.7 and for data source B is 0.3, then 700 images need to be selected from data source A and 300 images from data source B. This not only enriches the diversity of the sample data but also ensures a higher proportion of diverse data, thereby improving the model training effect.

[0184] In addition, other diversity measures can be used to strategy data source diversity, such as the phash (aware hashing) method. For details on the phash method, please refer to the relevant technical documentation; it will not be elaborated upon here.

[0185] For each data source, according to the fusion ratio of that data source, a corresponding number of first image data are selected from the first image data of that data source to obtain the sample image data of that data source. Then, the deep learning model is trained using the sample image data of each data source to obtain the trained deep learning model.

[0186] By adjusting the resolution of image data from multiple data sources according to a preset resolution, the first image data is obtained. This ensures that the resolution of the final sample image data is adjusted according to the preset resolution. Furthermore, based on the attribute information of each data source, the fusion ratio of each data source is determined. For each data source, a corresponding number of first image data are selected from the first image data of that data source according to its fusion ratio, thus obtaining the sample image data of that data source. This integrates the data from multiple data sources. Finally, the integrated sample image data from each data source is used to train the deep learning model, which can improve the training effect of the deep learning model.

[0187] In one possible embodiment, when the multiple data sources include video data sources, the above-mentioned acquisition of image data to be processed from multiple data sources includes: acquiring video data from the video data sources; and performing equal-interval frame extraction on the video data to obtain image data to be processed.

[0188] If the data source includes video data sources, then the video data sources need to be processed to obtain video frame images from the video data sources, and these video frame images are used as image data to be processed.

[0189] Specifically, the frame number of the video data can be obtained, and the video data can be processed by extracting frames at equal intervals, for example, every 5 frames, to obtain the image data to be processed. In this way, it is not necessary to process all frames of the video data, reducing the amount of computation.

[0190] In one possible implementation, such as Figure 1b As shown, Figure 1b This is a second flowchart illustrating the deep learning model training method provided in this application embodiment. When the multiple data sources include video data sources, step S110 can be implemented through the following steps:

[0191] S111, Obtain video data from the aforementioned video data source;

[0192] S112, extract at least one video frame from the above video data and add it to the first image set;

[0193] S113, Select unselected video frames from the remaining video frames of the above video data to obtain the current video frame to be processed, wherein the remaining video frames are the video frames that have not been extracted from the above video data.

[0194] S114, calculate the similarity between the current video frame to be processed and each video frame in the first image set; if the similarity between the current video frame to be processed and each video frame in the first image set is less than a preset similarity threshold, then add the current video frame to be processed to the first image set.

[0195] S115, return to step S113 and continue execution until all video frames in the remaining video frames have been selected or the number of video frames in the first image set reaches the first quantity threshold, and obtain the image data to be processed from the video data source, wherein the image data to be processed from the video data source is each video frame in the first image set.

[0196] If the data source includes video data, a dhash (perceptual hashing) method can be used to measure video frame similarity to extract frames from the video data source. Specifically, to obtain video data from the video data source, at least one video frame is extracted from the video data and added to the first image set. Then, from the remaining video frames in the video data, unselected video frames are selected to obtain the current video frame to be processed. The remaining video frames are the video frames that have not been extracted from the video data. By calculating the similarity between the current video frame to be processed and each video frame in the first image set, if the similarity between the current video frame to be processed and each video frame in the first image set is less than a preset similarity threshold, the current video frame to be processed is added to the first image set. The process returns to the step of selecting unselected video frames from the remaining video frames in the video data to obtain the current video frame to be processed. This process continues until all video frames in the remaining video frames have been selected or the number of video frames in the first image set reaches a first quantity threshold, thus obtaining the image data to be processed from the video data source. The image data to be processed from the video data source consists of the video frames in the first image set. This eliminates the need to process all frames of the video data, reducing computational load.

[0197] The similarity between the current video frame to be processed and each video frame in the first image set can be calculated using a preset similarity calculation method. This preset similarity calculation method can employ methods such as cosine similarity calculation, Euclidean evaluation, or Pearson correlation coefficient. For example, features are extracted from both the current video frame to be processed and each video frame in the first image set to obtain feature vectors for both. Each feature vector includes at least one of the following information: image resolution, image size, image color, signal-to-noise ratio, etc. The cosine value of the two feature vectors is calculated, and the similarity is determined based on the cosine value.

[0198] Optionally, if the similarity between the current video frame to be processed and each video frame in the first image set is not less than a preset similarity threshold, then a first identifier is added to the current video frame to be processed. The first identifier indicates that the current video frame to be processed has been extracted but not selected. If the similarity between the current video frame to be processed and each video frame in the first image set is less than the preset similarity threshold, then a second identifier is added to the current video frame to be processed. The second identifier indicates that the current video frame to be processed has been extracted and selected. By adding corresponding identifiers to each video frame, the efficiency of frame extraction can be improved.

[0199] In one possible embodiment, adjusting the resolution of the image data to be processed according to a preset resolution to obtain first image data includes:

[0200] For each image data to be processed, if the resolution of the image data to be processed is lower than the preset resolution, the image data to be processed is filled or enlarged according to the preset resolution to obtain the first image data.

[0201] If the resolution of the image data to be processed is greater than the preset resolution, the image data to be processed is cropped according to the preset resolution to obtain the first image data.

[0202] The resolution of the first image data can be a preset resolution, or a resolution within a range represented by the preset resolution.

[0203] For each image data to be processed, if the resolution of the image data to be processed is lower than the preset resolution, the image data to be processed can be filled or enlarged according to the preset resolution. For specific filling or enlarging of the image data to be processed, please refer to the relevant operations of filling and enlarging the image in related technologies.

[0204] Processing image data from multiple data sources according to a preset resolution can ensure that the sample image data obtained from multiple data sources are of uniform specifications, avoid excessive differences in resolution between sample image data, and thus improve the training effect of deep learning models.

[0205] In one possible embodiment, when cropping the image, if the object to be detected in the first image data is too small, the region of the object to be detected (the labeled object region) in the first image data needs to be filled. Specifically, random pixels are used to fill the labeled object region in the first image data, and the labels of the first image data are modified to negative samples to avoid introducing dirty samples. After cropping the image data to be processed according to the preset resolution to obtain the first image data, the method further includes:

[0206] Calculate the intersection-union ratio (IUU) between the labeled object regions in the first image data and the labeled object regions in the image data to be processed. If the IUU is greater than a preset ratio threshold, retain the labels in the first image data.

[0207] If the cross-union ratio is not greater than a preset threshold, the labeled object region in the first image data is filled with random pixels, and the labeling of the first image data is modified to a negative sample.

[0208] After cropping the image data to be processed according to a preset resolution to obtain the first image data, the Intersection over Union (IOU) between the labeled object regions in the first image data and the labeled object regions in the image data to be processed can be calculated. The IOU determines whether to retain the labels in the first image data based on the magnitude of the IOU. The preset ratio threshold can be customized according to actual conditions, for example, set to 0.5, 0.6, or 0.8. In one example, taking a preset ratio threshold of 0.5, if IOU > 0.5, the labels in the first image data are retained; if IOU < 0.5, random pixels are used to fill the labeled object regions in the first image data, and the labels in the first image data are modified to negative samples.

[0209] For example, if the object to be detected is a traffic sign, and the traffic sign in the first image data is too small when the image is cropped, the model may not be able to recognize it even if the image has been labeled with the traffic sign because the traffic sign is too small. In this case, it is necessary to calculate the IOU between the labeled area of ​​the traffic sign in the first image data and the labeled object area in the image data to be processed. If the IOU is greater than a preset ratio threshold, the label of the first image data is retained. If it is not greater than the preset ratio threshold, the labeled object area in the first image data is filled with random pixels, and the label of the first image data is modified to a negative sample.

[0210] The cross-union ratio (CUNR) between the labeled object regions in the first image data and the labeled object regions in the image data to be processed is used to determine whether to retain the labels in the first image data. This helps determine whether the first image data is a positive or negative sample, thus avoiding the introduction of dirty samples when the labeled object regions in the first image data are too small, which would affect the training effect of the deep learning model.

[0211] In one possible embodiment, the attribute information of the aforementioned data source represents the diversity of the labeled objects in the aforementioned data source;

[0212] The above determination of the fusion ratio of each of the above data sources based on the attribute information of each of the above data sources includes: for each data source, determining the fusion ratio of the data source based on the diversity of the labeled objects in the data source, wherein the fusion ratio of the data source is positively correlated with the diversity of the labeled objects in the data source.

[0213] The diversity of labeled objects in a data source includes the diversity of the data's geographical location and the diversity of data collection status. The diversity of data collection status includes the diversity of the number of data collectors and the diversity of data collection periods. For example, given data source A and data source B, data source A has good diversity because its data was collected from multiple provinces by multiple people over a long period. Data source B, on the other hand, was collected from a few intersections by a small number of people over a short period, so its diversity is relatively lower than that of data source A. Thus, based on the attribute information of the data sources, the fusion ratio of data sources can be determined, with data sources with good diversity having a higher fusion ratio than data sources with poor diversity.

[0214] In addition to calculating the fusion ratio of each data source, the category ratio can also be calculated for each type of first image data in each data source. In one possible embodiment, for each data source, according to the fusion ratio of that data source, a corresponding number of first image data are selected from the first image data of that data source to obtain sample image data for that data source, including:

[0215] Step 1: For each data source, determine the number of images to be selected based on the fusion ratio of that data source;

[0216] Step 2: For each type of first image data in the data source, determine the category proportion of the first image data based on the object information of the labeled objects in that type of first image data;

[0217] Step 3: Based on the number of images selected from the data source to which the first image data belongs and the category ratio of the first image data, select a corresponding number of sample image data from the first image data.

[0218] After determining the fusion ratio of the data source, the number of images to be selected from that data source can be determined based on that fusion ratio. For example, the fusion ratio represents the proportion of sample image data to be extracted from the first image data in the data source; therefore, the number of images to be selected from that data source is the product of the fusion ratio and the total number of first image data in the data source. Alternatively, the fusion ratio can represent the percentage of sample image data extracted from the data source relative to the total number of sample image data; therefore, multiplying the total number of sample image data by the fusion ratio yields the number of images to be selected from that data source.

[0219] A data source can include multiple categories of first image data; for example, in a vehicle recognition scenario, a data source can include first image data of various types such as cars, vans, and buses. For each category of first image data in the data source, the proportion of that category is determined based on the object information of the labeled objects in that category of first image data. Based on the number of images selected from the data source to which that category of first image data belongs and the proportion of that category of first image data, a corresponding number of sample images are selected from that category of first image data.

[0220] The object information of labeled objects can include the morphological richness of the labeled objects. For example, if the labeled object is a person, the category proportion can be determined based on the person's physical form. For example, if the object has diverse poses, the category proportion will be relatively large. For example, since people have diverse poses and may wear makeup / accessories, the required sample image data proportion will be relatively large. The object information of labeled objects can also include the scale of the labeled objects, and the category proportion can be determined based on the scale of the labeled objects. For example, if the scale is small, the category proportion will be large. The object information of labeled objects can also include the aspect ratio of the labeled objects, and the category proportion can be determined based on the aspect ratio of the labeled objects. For example, if the aspect ratio is small, the category proportion will be large.

[0221] First, the number of images to be selected from each data source is determined based on the fusion ratio of the data sources. Then, based on the category of the image data source within each data source, the category ratio of each type of first image data is determined. Finally, based on the number of images selected from the data source to which the first image data of that category belongs and the category ratio of that first image data, a corresponding number of sample images are selected from that category of first image data. This two-stage selection not only enriches the multi-source nature of the sample data but also ensures a high proportion of diverse data, thereby improving the model training effect.

[0222] In one possible embodiment, the object information of the labeled object includes the shape richness, scale, and aspect ratio of the labeled object;

[0223] For each category of first image data in the data source, the proportion of that category of first image data is determined based on the object information of the labeled objects in that category of first image data, including:

[0224] For each type of first image data in the data source, the first proportion of the first image data of that type is determined based on the morphological richness of the labeled objects in the first image data of that type, wherein the morphological richness of the first image data of that type is positively correlated with the first proportion;

[0225] Based on the scale of the labeled objects in the first image data of this type, a second proportion of the first image data of this type is determined, wherein the scale of the first image data of this type is negatively correlated with the second proportion;

[0226] Based on the aspect ratio of the labeled objects in the first image data of this type, the third proportion of the first image data of this type is determined, wherein the aspect ratio of the first image data of this type is negatively correlated with the third proportion;

[0227] The category proportion of the first image data is determined based on the first proportion, second proportion, and third proportion of the first image data.

[0228] The object information of an labeled object includes its morphological richness, scale, and aspect ratio. For example, if the labeled object is a person, and people have diverse poses and adornments such as makeup and accessories, then a larger proportion of sample data is needed to learn the characteristics of a person and improve the accuracy of model recognition. Alternatively, if the scale or aspect ratio of the first image data of this type is small, then after multiple downsampling, the effective feature information of the detected object becomes blurred, and too much effective information is lost, seriously affecting the model training effect. In this case, a larger proportion of sample data is also needed.

[0229] After obtaining the first proportion, second proportion, and third proportion of the first type of image data, a weighted calculation can be performed based on the first proportion, second proportion, and third proportion to determine the category proportion of the first type of image data.

[0230] Furthermore, sample image data can be selected based on the size of the labeled regions, and the distribution information of each data source can be statistically analyzed. This can be done by considering factors such as the quantiles of the width and height of the labeled regions, the proportion of each type of labeled region, and the proportion of images in each type of labeled region. This allows for the rational selection of sample image data based on the distribution information.

[0231] like Figure 2 As shown, Figure 2 This is a schematic flowchart of an object detection method provided in an embodiment of this application. The method includes:

[0232] S210, acquire the image data to be detected;

[0233] S220, using a pre-trained deep learning model, predict the object detection result of the above-mentioned image data to be detected, wherein the deep learning model is trained by any of the deep learning model training methods described above.

[0234] After training a pre-trained deep learning model based on any of the deep learning model training methods described above, the pre-trained deep learning model can be used to predict the image data to be detected, thereby obtaining the object detection result of the image data to be detected.

[0235] In one possible embodiment, the image data to be detected is a video frame in the video data to be detected; the prediction of the object detection result of the image data to be detected using a pre-trained deep learning model includes:

[0236] Background segmentation is performed on the above image data to be detected to obtain static and moving regions;

[0237] Based on the type of the object to be detected, the first region to which the object to be detected belongs is determined in the above-mentioned static region and the above-mentioned moving region;

[0238] The object detection results of the above-mentioned first region are obtained by using a pre-trained deep learning model to predict the first region.

[0239] After acquiring the image data to be detected, background segmentation can be performed on the image data to obtain static and moving regions. Methods such as inter-frame interval method and Gaussian mixture model method can be used for background segmentation of the image data to be detected.

[0240] After obtaining the static and moving regions, based on the type of the object to be detected, a first region can be determined within each region. A pre-trained deep learning model can then be used to predict the first region, yielding the object detection result for the image data. For example, if the object is a moving person, the region should be a moving region; therefore, a pre-trained deep learning model can be used to predict the moving region, resulting in the object detection result for the image data. Conversely, if the object is a traffic sign, the region should be a static region; therefore, a pre-trained deep learning model can be used to predict the static region, resulting in the object detection result for the image data.

[0241] In one possible embodiment, the image data to be detected is a video frame in the video data to be detected; the prediction of the object detection result of the image data to be detected using a pre-trained deep learning model includes:

[0242] Background segmentation is performed on the above image data to be detected to obtain static and moving regions;

[0243] Based on the type of the object to be detected, the first region to which the object to be detected belongs is determined in the above-mentioned static region and the above-mentioned moving region;

[0244] When the image data to be detected is a first type of video frame, a pre-trained deep learning model is used to predict the first region to obtain a first detection result of the image data to be detected; the first detection result of the image data to be detected is merged with the most recently obtained second detection result to obtain an object detection result of the image data to be detected; wherein, the first type of video frame is a video frame obtained by extracting frames from the video data to be detected according to a first frame rate.

[0245] When the image data to be detected is a second type of video frame, a pre-trained deep learning model is used to predict the second region to obtain a second detection result of the image data to be detected; the second detection result of the image data to be detected is merged with the most recently obtained first detection result to obtain an object detection result of the image data to be detected; wherein, the second type of video frame is a video frame obtained by extracting frames from the video data to be detected according to a second frame rate; the second region is the static region and another region in the moving region other than the first region; the first frame rate is greater than the second frame rate.

[0246] Background segmentation of the image data to be detected is performed to obtain static and moving regions. The object detection results based on these static and moving regions are easily affected by external factors such as lighting and shadows from leaves. For example, if only static / moving regions are detected, some objects may be missed. For instance, traffic signs are normally static areas, so detecting them only requires predicting the separated static regions. However, because some traffic signs are obscured by passing vehicles, this part of the sign becomes a moving area and will not be predicted. Therefore, a fast / slow pointer method is used for detection: regions where objects are more likely to appear (static regions) are detected more frequently (e.g., 20 times per second), while regions where objects are less likely to appear (moving regions) are detected less frequently (e.g., 10 times per second).

[0247] When the image data to be detected is a video frame obtained by sampling the video data to be detected at a first detection frequency, a pre-trained deep learning model is used to predict the first region to obtain the first detection result of the image data to be detected. The first detection result of the image data to be detected is then merged with the most recently obtained second detection result to obtain the object detection result of the image data to be detected. In this way, obtaining the object detection result of the image data to be detected based on the first detection result and the most recently obtained second detection result can improve the prediction accuracy of the deep learning model.

[0248] In one possible embodiment, the image data to be detected is a video frame in the video data to be detected; the prediction of the object detection result of the image data to be detected using a pre-trained deep learning model includes:

[0249] Based on the existing detection results of the object to be detected in the above image data to be detected, determine the third region in which the object to be detected may appear in the above image data to be detected.

[0250] The object detection results of the above-mentioned third region are obtained by using a pre-trained deep learning model to predict the object in the above-mentioned image data to be detected.

[0251] For certain fixed scenes, the areas where target objects exist generally follow certain patterns. By analyzing the spatial information of the target objects in the scene from a temporal perspective, we can roughly draw the third region where the target object may appear in the above-mentioned image data. Subsequently, using this optimal target region for object detection can improve the detection efficiency of deep learning models.

[0252] In one possible embodiment, the acquisition of the image data to be detected includes:

[0253] According to the preset frame extraction rules, extract the image data to be detected from the video data to be detected;

[0254] The above methods also include:

[0255] Based on the location information of the detection result of the object to be detected, predict the location information of the object to be detected in other video frames of other video data to be detected;

[0256] A sliding window detection method is used to filter the position information of the objects to be detected.

[0257] If the acquired data is video data, the image data to be detected can be extracted from the video data according to a preset frame extraction rule. Specifically, it can be done by extracting frames at equal intervals.

[0258] Then, based on the positional information of the detection results of the image to be detected, the positional information of the object to be detected in other video frames of other video data to be detected can be predicted. Specifically, based on the positional information of the object detection results in the image data to be detected, the target object image can be obtained, where the target object is the detection object. Then, based on the similarity of the target object image, the positional information of the target object in the next frame image can be obtained. For example, a filter-based tracing algorithm, such as the KCF (Kernel Correlation Filter) series of tracing generators, can be used to generate the positional information of the object to be detected in the video data to be detected. Filter-based tracing algorithms are fast and consume few resources, and can perform accurate tracking in a short time. Therefore, only a portion of the video frames can be detected to obtain continuous detection results of the object, thus improving detection efficiency.

[0259] Furthermore, a sliding window detection method is employed to filter the position of the object to be detected. This filters out a small number of discontinuous false alarms, while simultaneously identifying and completing the discontinuous results to create a complete trajectory, facilitating object trajectory analysis. Moreover, after obtaining the position information of the object in each frame, motion tracks of the object can be generated and displayed on a preset display page, allowing users to view the motion tracks of the object.

[0260] In one possible embodiment, after the step of acquiring the image data to be detected described above, the method further includes:

[0261] Based on the resolution of the image data to be detected and the maximum downsampling ratio of the pre-trained deep learning model, the slice size of the image data to be detected is determined.

[0262] According to the above slice size, the image data to be detected is cropped into multiple image slices;

[0263] The object detection results predicted by the pre-trained deep learning model for the above-mentioned image data include:

[0264] The inference results for each of the above image slices are obtained by using a pre-trained deep learning model to perform inference on multiple of the above image slices.

[0265] The inference results of each of the above image slices are merged, fused, and deduplicated to obtain the object detection results of the above image data to be detected.

[0266] The slice size of the image data to be detected is determined based on the resolution of the image data and the maximum downsampling ratio of the deep learning model. For example, if the resolution of the image data to be detected is greater than the maximum downsampling ratio of the deep learning model, the image data to be detected needs to be cropped to obtain a suitable slice size. According to the slice size, the currently selected image data to be detected is cropped into multiple image slices, and the deep learning model is used to perform inference on each of these image slices to obtain the inference result for each image slice.

[0267] Assuming the width and height of the image data to be detected are w and h respectively, and the maximum downsampling factor of the deep learning model is s; the recommended inference dimensions are W and H.

[0268] The optimal number of blocks in the image slice to be detected: Nw = np.floor(w / W); Nh = np.floor(h / H)

[0269] The pixels that overlap horizontally are the same, and the pixels that overlap vertically are the same, as shown below:

[0270] pad_w=np.ceil(w%W / (2*Nw-2));

[0271] pad_h=np.ceil(h%H / (2*Nh-2));

[0272] The optimal patch size is:

[0273] P_w=np.ceil((W+2*pad_w) / s)+W+2*pad_w;

[0274] P_h=np.ceil((H+2*pad_h) / s)+H+2*pad_h.

[0275] Here, `np.floor` is a function that returns the largest integer not greater than the input parameter; `np.ceil` is a function that calculates the smallest integer greater than or equal to that value.

[0276] The above method not only ensures a reasonable inference size for deep learning models, but also preserves minimal padding (the space between element borders and element content) and avoids unnecessary computation.

[0277] By merging the inference results of multiple sample image slices, the information of the target region in the original image can be restored. However, if multiple sample image slices all have inference results for the same object, for example, if each sample image slice detects a small part of the target object, then the NMM (Non-Max Merging) method is needed for fusion. If each sample image slice repeatedly detects the target object, then the NMS (Non-Maximum Suppression) method is needed to select the optimal target object and remove duplicate and less desirable detection results.

[0278] In one possible implementation, the slice sizes of multiple images are determined, so that the deep learning model infers over the slice sizes of the multiple images to be detected. This allows the model to test the maximum reasonable inference size of the deep learning model under full-load inference, retain the minimum padding irrelevant elements, and avoid unnecessary computation.

[0279] like Figure 3 As shown, Figure 3 A schematic diagram of a deep learning model training apparatus provided in this application embodiment, the apparatus comprising:

[0280] The acquisition module 310 is used to acquire image data to be processed from multiple data sources;

[0281] The adjustment module 320 is used to adjust the resolution of the above-mentioned image data to be processed according to a preset resolution to obtain the first image data;

[0282] The module 330 is used to determine the fusion ratio of each of the above data sources based on the attribute information of each of the above data sources;

[0283] The selection module 340 is used to select a corresponding number of first image data from the first image data of each data source according to the fusion ratio of that data source, so as to obtain the sample image data of that data source.

[0284] The training module 350 is used to train the deep learning model using sample image data from the aforementioned data sources, thereby obtaining a trained deep learning model.

[0285] In one possible embodiment, the acquisition module 310 is specifically used for:

[0286] In the case where the above-mentioned multiple data sources include video data sources; obtain the video data from the above-mentioned video data sources;

[0287] Extract at least one video frame from the above video data and add it to the first image set;

[0288] From the remaining video frames of the above video data, select the video frames that have not been selected to obtain the current video frame to be processed, wherein the above remaining video frames are the video frames that have not been extracted from the above video data.

[0289] Calculate the similarity between the current video frame to be processed and each video frame in the first image set; if the similarity between the current video frame to be processed and each video frame in the first image set is less than a preset similarity threshold, then add the current video frame to be processed to the first image set.

[0290] Return step: From the remaining video frames of the above video data, select the unselected video frames to obtain the current video frames to be processed and continue to execute until all video frames in the remaining video frames are selected or the number of video frames in the first image set reaches the first quantity threshold, and obtain the image data to be processed from the above video data source, wherein the image data to be processed from the above video data source is each video frame in the above first image set.

[0291] In one possible embodiment, the adjustment module 320 is specifically used for:

[0292] For each image data to be processed, if the resolution of the image data to be processed is lower than the preset resolution, the image data to be processed is filled or enlarged according to the preset resolution to obtain the first image data.

[0293] If the resolution of the image data to be processed is greater than the preset resolution, the image data to be processed is cropped according to the preset resolution to obtain the first image data.

[0294] In one possible embodiment, the above-described apparatus further includes:

[0295] The calculation module is used to calculate the intersection-union ratio of the labeled object region in the first image data and the labeled object region in the image data to be processed after cropping the image data to be processed according to the preset resolution to obtain the first image data. If the intersection-union ratio is greater than the preset ratio threshold, the label of the first image data is retained.

[0296] The filling module is used to fill the labeled object region in the first image data with random pixels when the cross-union ratio is not greater than a preset ratio threshold, and to modify the label of the first image data to a negative sample.

[0297] In one possible embodiment, the attribute information of the aforementioned data source represents the diversity of the labeled objects in the aforementioned data source; the determining module 330 is specifically used for:

[0298] For each data source, the fusion ratio of that data source is determined based on the diversity of the labeled objects in that data source. The fusion ratio of that data source is positively correlated with the diversity of the labeled objects in that data source.

[0299] In one possible embodiment, the selection module 340 is specifically used for:

[0300] For each data source, determine the number of images to be selected from that data source based on its fusion ratio;

[0301] For each type of first image data in the data source, determine the category proportion of the first image data based on the object information of the labeled objects in that type of first image data;

[0302] Based on the number of images selected from the data source to which the first image data belongs and the category ratio of the first image data, a corresponding number of sample image data are selected from the first image data.

[0303] In one possible embodiment, the object information of the labeled object includes the shape richness, scale, and aspect ratio of the labeled object;

[0304] The selection module 340 mentioned above is specifically used for:

[0305] For each type of first image data in the data source, the first proportion of the first image data of that type is determined based on the morphological richness of the labeled objects in the first image data of that type, wherein the morphological richness of the first image data of that type is positively correlated with the first proportion;

[0306] Based on the scale of the labeled objects in the first image data of this type, a second proportion of the first image data of this type is determined, wherein the scale of the first image data of this type is negatively correlated with the second proportion;

[0307] Based on the aspect ratio of the labeled objects in the first image data of this type, the third proportion of the first image data of this type is determined, wherein the aspect ratio of the first image data of this type is negatively correlated with the third proportion;

[0308] The category proportion of the first image data is determined based on the first proportion, second proportion, and third proportion of the first image data.

[0309] like Figure 4 As shown, Figure 4 This is a schematic diagram of an object detection device provided in an embodiment of this application. The device includes:

[0310] The acquisition module 410 is used to acquire the image data to be detected;

[0311] The prediction module 420 is used to predict the object detection result of the above-mentioned image data to be detected using a pre-trained deep learning model, wherein the deep learning model is trained by any of the deep learning model training devices described above.

[0312] In one possible embodiment, the image data to be detected is a video frame in the video data to be detected; the prediction module is specifically used for:

[0313] Background segmentation is performed on the above image data to be detected to obtain static and moving regions;

[0314] Based on the type of the object to be detected, the first region to which the object to be detected belongs is determined in the above-mentioned static region and the above-mentioned moving region;

[0315] The object detection results of the above-mentioned first region are obtained by using a pre-trained deep learning model to predict the first region.

[0316] In one possible embodiment, the image data to be detected is a video frame in the video data to be detected; the prediction module is specifically used for:

[0317] Background segmentation is performed on the above image data to be detected to obtain static and moving regions;

[0318] Based on the type of the object to be detected, the first region to which the object to be detected belongs is determined in the above-mentioned static region and the above-mentioned moving region;

[0319] When the image data to be detected is a first type of video frame, a pre-trained deep learning model is used to predict the first region to obtain a first detection result of the image data to be detected; the first detection result of the image data to be detected is merged with the most recently obtained second detection result to obtain an object detection result of the image data to be detected; wherein, the first type of video frame is a video frame obtained by extracting frames from the video data to be detected according to a first frame rate.

[0320] When the image data to be detected is a second type of video frame, a pre-trained deep learning model is used to predict the second region to obtain a second detection result of the image data to be detected; the second detection result of the image data to be detected is merged with the most recently obtained first detection result to obtain an object detection result of the image data to be detected; wherein, the second type of video frame is a video frame obtained by extracting frames from the video data to be detected according to a second frame rate; the second region is the static region and another region in the moving region other than the first region; the first frame rate is greater than the second frame rate.

[0321] In one possible embodiment, the image data to be detected is a video frame in the video data to be detected; the prediction module is specifically used for:

[0322] Based on the existing detection results of the object to be detected in the above image data to be detected, determine the third region in which the object to be detected may appear in the above image data to be detected.

[0323] The object detection results of the above-mentioned third region are obtained by using a pre-trained deep learning model to predict the object in the above-mentioned image data to be detected.

[0324] In one possible embodiment, the acquisition module described above is specifically used for;

[0325] According to the preset frame extraction rules, extract the image data to be detected from the video data to be detected;

[0326] The above-mentioned device also includes:

[0327] The generation module is used to predict the position information of the object to be detected in other video frames of other video data to be detected based on the position information of the object detection result in the above-mentioned image data to be detected.

[0328] The filtering module is used to filter the position information of the objects to be detected using a sliding window detection method.

[0329] In one possible embodiment, the above-described apparatus further includes:

[0330] The slice size determination module is used to determine the slice size of the image data to be detected based on the resolution of the image data to be detected and the maximum downsampling ratio of the pre-trained deep learning model after the above steps of obtaining the image data to be detected.

[0331] The cropping module is used to crop the image data to be detected into multiple image slices according to the above-mentioned slice size;

[0332] The aforementioned prediction module is specifically used for:

[0333] The inference results for each of the above image slices are obtained by using a pre-trained deep learning model to perform inference on multiple of the above image slices.

[0334] The inference results of each of the above image slices are merged, fused, and deduplicated to obtain the object detection results of the above image data to be detected.

[0335] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0336] This application also provides an electronic device, such as... Figure 5 As shown, it includes:

[0337] Memory 501 is used to store computer programs;

[0338] When processor 502 executes the program stored in memory 501, it performs the following steps:

[0339] Acquire image data to be processed from multiple data sources;

[0340] The resolution of the above image data to be processed is adjusted according to the preset resolution to obtain the first image data;

[0341] Based on the attribute information of each of the above data sources, the fusion ratio of each of the above data sources is determined;

[0342] For each data source, according to the fusion ratio of that data source, a corresponding number of first image data are selected from the first image data of that data source to obtain the sample image data of that data source;

[0343] The deep learning model is trained using sample image data from the aforementioned data sources to obtain a trained deep learning model.

[0344] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 502, communication interface, and memory 501 communicating with each other via the communication bus.

[0345] This application also provides an electronic device, including:

[0346] Memory, used to store computer programs;

[0347] When a processor executes a program stored in memory, it performs the following steps:

[0348] Acquire the image data to be detected;

[0349] Using a pre-trained deep learning model, the object detection result of the above-mentioned image data to be detected is predicted, wherein the deep learning model is trained by any of the deep learning model training methods described above.

[0350] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0351] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0352] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0353] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0354] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described deep learning model training methods or object detection methods.

[0355] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the deep learning model training methods or object detection methods described in the above embodiments.

[0356] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0357] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0358] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0359] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A deep learning model training method, characterized in that, The method includes: Acquire image data to be processed from multiple data sources; The resolution of the image data to be processed is adjusted according to a preset resolution to obtain the first image data; Based on the attribute information of each data source, the fusion ratio of each data source is determined; For each data source, determine the number of images to be selected from that data source based on its fusion ratio; For each category of first image data in the data source, a first proportion of the first image data category is determined based on the morphological richness of the labeled objects in that category, wherein the morphological richness of the first image data category is positively correlated with the first proportion; a second proportion of the first image data category is determined based on the scale of the labeled objects in that category, wherein the scale of the first image data category is negatively correlated with the second proportion; and a third proportion of the first image data category is determined based on the aspect ratio of the labeled objects in that category, wherein the aspect ratio of the first image data category is negatively correlated with the third proportion. The category proportion of the first image data is determined based on the first proportion, second proportion, and third proportion of the first image data of this type; Based on the number of images selected from the data source to which the first image data belongs and the category ratio of the first image data, a corresponding number of sample image data are selected from the first image data of that category; The deep learning model is trained using sample image data from each of the aforementioned data sources to obtain a trained deep learning model.

2. The method according to claim 1, characterized in that, In the case where the multiple data sources include video data sources; the step of acquiring the image data to be processed from the multiple data sources includes: Obtain video data from the aforementioned video data source; At least one video frame is extracted from the video data and added to the first image set; From the remaining video frames of the video data, select the unselected video frames to obtain the current video frame to be processed, wherein the remaining video frames are the video frames that have not been extracted from the video data; Calculate the similarity between the current video frame to be processed and each video frame in the first image set; if the similarity between the current video frame to be processed and each video frame in the first image set is less than a preset similarity threshold, then add the current video frame to be processed to the first image set. Return step: Select unselected video frames from the remaining video frames of the video data to obtain the current video frame to be processed and continue execution until all video frames in the remaining video frames have been selected or the number of video frames in the first image set reaches a first quantity threshold, thereby obtaining the image data to be processed from the video data source, wherein the image data to be processed from the video data source is each video frame in the first image set.

3. The method according to claim 1, characterized in that, The step of adjusting the resolution of the image data to be processed according to a preset resolution to obtain the first image data includes: For each image data to be processed, if the resolution of the image data to be processed is lower than the preset resolution, the image data to be processed is filled or enlarged according to the preset resolution to obtain the first image data. If the resolution of the image data to be processed is greater than the preset resolution, the image data to be processed is cropped according to the preset resolution to obtain the first image data.

4. The method according to claim 3, characterized in that, After cropping the image data to be processed according to the preset resolution to obtain the first image data, the method further includes: Calculate the intersection-union ratio (IUU) of the labeled object regions in the first image data and the labeled object regions in the image data to be processed. If the IUU is greater than a preset ratio threshold, retain the labels in the first image data. If the cross-union ratio is not greater than a preset ratio threshold, the labeled object regions in the first image data are filled with random pixels, and the labels of the first image data are modified to negative samples.

5. The method according to claim 1, characterized in that, The attribute information of the data source represents the diversity of the labeled objects in the data source; Determining the fusion ratio of each type of data source based on the attribute information of each data source includes: For each data source, the fusion ratio of that data source is determined based on the diversity of the labeled objects in that data source. The fusion ratio of that data source is positively correlated with the diversity of the labeled objects in that data source.

6. An object detection method, characterized in that, The method includes: Acquire the image data to be detected; Using a pre-trained deep learning model, the object detection result of the image data to be detected is predicted, wherein the deep learning model is trained by the method described in any one of claims 1-5.

7. The method according to claim 6, characterized in that, The image data to be detected is a video frame in the video data to be detected; the step of using a pre-trained deep learning model to predict the object detection result of the image data to be detected includes: The image data to be detected is segmented into a background to obtain static and moving regions; Based on the type of the object to be detected, a first region to which the object to be detected belongs is determined in the static region and the moving region; The object detection result of the image data to be detected is obtained by using a pre-trained deep learning model to predict the first region.

8. The method according to claim 6, characterized in that, The image data to be detected is a video frame in the video data to be detected; the step of using a pre-trained deep learning model to predict the object detection result of the image data to be detected includes: The image data to be detected is segmented into a background to obtain static and moving regions; Based on the type of the object to be detected, a first region to which the object to be detected belongs is determined in the static region and the moving region; When the image data to be detected is a first type of video frame, a pre-trained deep learning model is used to predict the first region to obtain a first detection result of the image data to be detected; the first detection result of the image data to be detected is merged with the most recently obtained second detection result to obtain an object detection result of the image data to be detected; wherein, the first type of video frame is a video frame obtained by extracting frames from the video data to be detected according to a first frame rate. When the image data to be detected is a second type of video frame, a pre-trained deep learning model is used to predict the second region to obtain a second detection result of the image data to be detected; the second detection result of the image data to be detected is merged with the most recently obtained first detection result to obtain the object detection result of the image data to be detected; wherein, the second type of video frame is a video frame obtained by extracting frames from the video data to be detected according to a second frame rate; the second region is the static region and another region in the moving region other than the first region; the first frame rate is greater than the second frame rate.

9. The method according to claim 7, characterized in that, The image data to be detected is a video frame in the video data to be detected; the step of using a pre-trained deep learning model to predict the object detection result of the image data to be detected includes: Based on the existing detection results of the object to be detected in the image data to be detected, a third region in which the object to be detected may appear in the image data to be detected is determined; The object detection result of the image data to be detected is obtained by using a pre-trained deep learning model to predict the third region.

10. The method according to claim 6, characterized in that, The acquisition of the image data to be detected includes: According to the preset frame extraction rules, extract the image data to be detected from the video data to be detected; The method further includes: Based on the location information of the object to be detected in the image data to be detected, predict the location information of the object to be detected in other video frames of other video data to be detected; A sliding window detection method is used to filter the position information of the object to be detected.

11. The method according to claim 6, characterized in that, After the step of acquiring the image data to be detected, the method further includes: The slice size of the image data to be detected is determined based on the resolution of the image data to be detected and the maximum downsampling ratio of the pre-trained deep learning model. According to the slice size, the image data to be detected is cropped into multiple image slices; The method of using a pre-trained deep learning model to predict the object detection result of the image data to be detected includes: The inference results for each image slice are obtained by using a pre-trained deep learning model to perform inference on multiple image slices. The inference results of each image slice are merged, fused, and deduplicated to obtain the object detection result of the image data to be detected.

12. A deep learning model training device, characterized in that, The device includes: The acquisition module is used to acquire image data to be processed from multiple data sources; An adjustment module is used to adjust the resolution of the image data to be processed according to a preset resolution to obtain the first image data; The determination module is used to determine the fusion ratio of each of the data sources based on the attribute information of each data source; The selection module is used to determine the number of images selected from each data source based on its fusion ratio; for each category of first image data in the data source, it determines a first proportion of the first image data category based on the morphological richness of the labeled objects in that category, where the morphological richness of the first image data category is positively correlated with the first proportion; it determines a second proportion of the first image data category based on the scale of the labeled objects in that category, where the scale of the first image data category is negatively correlated with the second proportion; it determines a third proportion of the first image data category based on the aspect ratio of the labeled objects in that category, where the aspect ratio of the first image data category is negatively correlated with the third proportion; it determines the category proportion of the first image data category based on the first, second, and third proportions; and it selects a corresponding number of sample images from the first image data category based on the number of images selected from the data source to which the first image data category belongs and the category proportion of the first image data category. The training module is used to train the deep learning model using sample image data from each of the aforementioned data sources, thereby obtaining a trained deep learning model.

13. The apparatus according to claim 12, characterized in that, The acquisition module is specifically used for: In the case where the multiple data sources include video data sources; obtain the video data from the video data source; At least one video frame is extracted from the video data and added to the first image set; From the remaining video frames of the video data, select the unselected video frames to obtain the current video frame to be processed, wherein the remaining video frames are the video frames that have not been extracted from the video data; Calculate the similarity between the current video frame to be processed and each video frame in the first image set; if the similarity between the current video frame to be processed and each video frame in the first image set is less than a preset similarity threshold, then add the current video frame to be processed to the first image set. Return step: Select unselected video frames from the remaining video frames of the video data to obtain the current video frame to be processed and continue to execute until all video frames in the remaining video frames are selected or the number of video frames in the first image set reaches a first number threshold, to obtain the image data to be processed from the video data source, wherein the image data to be processed from the video data source is each video frame in the first image set. The adjustment module is specifically used for: For each image data to be processed, if the resolution of the image data to be processed is lower than the preset resolution, the image data to be processed is filled or enlarged according to the preset resolution to obtain the first image data. If the resolution of the image data to be processed is greater than the preset resolution, the image data to be processed is cropped according to the preset resolution to obtain the first image data. The device further includes: The calculation module is used to calculate the intersection-union ratio (IUU) of the labeled object regions in the first image data and the labeled object regions in the image data to be processed after cropping the image data to be processed according to the preset resolution to obtain the first image data. If the IUU is greater than a preset ratio threshold, the labels of the first image data are retained. The filling module is used to fill the labeled object region in the first image data with random pixels when the cross-union ratio is not greater than a preset ratio threshold, and to modify the label of the first image data to a negative sample. The attribute information of the data source represents the diversity of the labeled objects in the data source; the determining module is specifically used for: For each data source, the fusion ratio of that data source is determined based on the diversity of the labeled objects in that data source. The fusion ratio of that data source is positively correlated with the diversity of the labeled objects in that data source.

14. An object detection device, characterized in that, The device includes: The acquisition module is used to acquire the image data to be detected; A prediction module is used to predict the object detection result of the image data to be detected using a pre-trained deep learning model, wherein the deep learning model is trained by the apparatus according to any one of claims 12-13.

15. The apparatus according to claim 14, characterized in that, The image data to be detected is a video frame in the video data to be detected; the prediction module is specifically used for: The image data to be detected is segmented into a background to obtain static and moving regions; Based on the type of the object to be detected, a first region to which the object to be detected belongs is determined in the static region and the moving region; The first region is predicted using a pre-trained deep learning model to obtain the object detection result of the image data to be detected; The image data to be detected is a video frame in the video data to be detected; the prediction module is specifically used for: The image data to be detected is segmented into a background to obtain static and moving regions; Based on the type of the object to be detected, a first region to which the object to be detected belongs is determined in the static region and the moving region; When the image data to be detected is a first type of video frame, a pre-trained deep learning model is used to predict the first region to obtain the first detection result of the image data to be detected. The first detection result of the image data to be detected is merged with the most recently obtained second detection result to obtain the object detection result of the image data to be detected; wherein, the first type of video frame is a video frame obtained by extracting frames from the video data to be detected according to a first frame rate; When the image data to be detected is a second type of video frame, a pre-trained deep learning model is used to predict the second region to obtain a second detection result for the image data to be detected; the second detection result for the image data to be detected is then merged with the most recently obtained first detection result to obtain an object detection result for the image data to be detected; wherein, the second type of video frame is a video frame obtained by extracting frames from the video data to be detected according to a second frame rate; the second region is the static region and another region in the moving region other than the first region; the first frame rate is greater than the second frame rate; The image data to be detected is a video frame in the video data to be detected; the prediction module is specifically used for: Based on the existing detection results of the object to be detected in the image data to be detected, a third region in which the object to be detected may appear in the image data to be detected is determined; The object detection result of the image data to be detected is obtained by using a pre-trained deep learning model to predict the third region. The acquisition module is specifically used for; According to the preset frame extraction rules, extract the image data to be detected from the video data to be detected; The device further includes: The generation module is used to predict the position information of the object to be detected in other video frames of other video data to be detected based on the position information of the object detection result in the image data to be detected; The filtering module is used to filter the position information of the object to be detected using a sliding window detection method; The device further includes: The slice size determination module is used to determine the slice size of the image data to be detected based on the resolution of the image data to be detected and the maximum downsampling ratio of the pre-trained deep learning model after the step of acquiring the image data to be detected. The cropping module is used to crop the image data to be detected into multiple image slices according to the slice size; The prediction module is specifically used for: The inference results for each image slice are obtained by using a pre-trained deep learning model to perform inference on multiple image slices. The inference results of each image slice are merged, fused, and deduplicated to obtain the object detection result of the image data to be detected.

16. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-5 or 6-11.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-5 or 6-11.

Images