Training method of image element detection model, image element detection method and device

Abstract

The embodiment of the application discloses a kind of training method of image element detection model, image element detection method and device, and can be applicable to artificial intelligence, block chain and image processing etc. field.Image element detection method includes: determining the initial image feature of to-be-processed image, the initial image feature of to-be-processed image is input into image element detection model, at least one target detection area and its target prediction result are obtained;Wherein, each target prediction result is used to indicate whether the image element detection model predicts that the corresponding target detection area includes the image element of pre-set type, and the image element detection model is obtained based on the training method of image element detection model Training;For each target detection area, based on the target prediction result of the target detection area, determine whether the target detection area actually includes the image element of pre-set type.Using the embodiment of the application, the image element belonging to pre-set type in image can be accurately detected.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to training methods for image element detection models, image element detection methods and apparatus. Background Technology

[0002] With the continuous development of image processing technology, how to effectively detect image elements of a specific type has become an increasingly important technical issue. Furthermore, due to the combined influence of factors such as imaging angle, element size, color, and outline, different types of image elements often appear highly similar, making the detection of specific types of image elements quite challenging.

[0003] While existing technologies can train detection models based on artificial intelligence to identify specific types of image elements, their accuracy is limited. Summary of the Invention

[0004] This application provides a training method for an image element detection model, an image element detection method, and an apparatus, which can improve the detection accuracy of image elements of a preset type and has high applicability.

[0005] On one hand, embodiments of this application provide a training method for an image element detection model, the method comprising: The initial image features of each sample image are input into the initial model to obtain the target prediction result of at least one target detection region of the sample image. Each of the above prediction results is used to indicate whether the target detection region corresponding to the model prediction includes image elements of a preset type. The total training loss is determined based on the target prediction results and the true classification results of each of the above target detection regions. The initial model is trained based on each of the above sample images until the total training loss meets the preset conditions. Training is stopped when the total training loss meets the preset conditions. The model at which training stops is determined as the image element detection model. Each of the above true classification results is used to indicate whether the corresponding target detection region actually includes image elements of the preset type. The initial model above determines the prediction results for each of the above objectives based on the following method: Based on the initial image features of each sample image, at least one initial detection region of the sample image is iteratively adjusted a predetermined number of times to obtain the corresponding target detection region, wherein each adjustment is performed in the following manner: Based on the initial image features of the sample image, each first detection region is adjusted to obtain a second detection region and its corresponding initial prediction result. Each initial prediction result is used to indicate whether the second detection region corresponding to the model prediction includes image elements of a preset type. Each adjustment of the corresponding first detection region is the second detection region obtained in the previous adjustment. The first adjustment of the corresponding first detection region is the aforementioned initial detection region. The second detection region obtained in the last adjustment is the target detection region of the sample image. Based on the initial prediction results of the second detection region obtained from each adjustment, the target prediction results of each target detection region of the sample image are determined.

[0006] On the other hand, embodiments of this application provide an image element detection method, the method comprising: Determine the initial image features of the image to be processed, input the initial image features of the image to be processed into the image element detection model, and obtain at least one target detection region and its target prediction result; Each of the above prediction results is used to indicate whether the target detection region predicted by the above image element detection model includes image elements of a preset type. The above image element detection model is trained based on the training method of the image element detection model provided in the embodiments of this application. For each of the above target detection regions, based on the target prediction results of that target detection region, it is determined whether the target detection region actually includes image elements of a preset type.

[0007] On the other hand, embodiments of this application provide a training apparatus for an image element detection model, the apparatus comprising: The detection module is used to input the initial image features of each sample image into the initial model to obtain the target prediction result of at least one target detection region of the sample image. Each of the above target prediction results is used to indicate whether the target detection region corresponding to the model prediction includes image elements of a preset type. The training module is used to determine the total training loss based on the target prediction results and the true classification results of each of the above target detection regions, and to train the above initial model based on each of the above sample images until the total training loss meets the preset conditions and then stop training. The model at which training stops is determined as the image element detection model. Each of the above true classification results is used to indicate whether the corresponding target detection region actually includes image elements of a preset type. The detection module determines the prediction results for each of the aforementioned targets based on the following method: Based on the initial image features of each sample image, at least one initial detection region of the sample image is iteratively adjusted a predetermined number of times to obtain the corresponding target detection region, wherein each adjustment is performed in the following manner: Based on the initial image features of the sample image, each first detection region is adjusted to obtain a second detection region and its corresponding initial prediction result. Each initial prediction result is used to indicate whether the second detection region corresponding to the model prediction includes image elements of a preset type. Each adjustment of the corresponding first detection region is the second detection region obtained in the previous adjustment. The first adjustment of the corresponding first detection region is the aforementioned initial detection region. The second detection region obtained in the last adjustment is the target detection region of the sample image. Based on the initial prediction results of the second detection region obtained from each adjustment, the target prediction results of each target detection region of the sample image are determined.

[0008] On the other hand, embodiments of this application provide an image element detection device, which includes: The prediction result determination module is used to determine the initial image features of the image to be processed, input the initial image features of the image to be processed into the image element detection model, and obtain at least one target detection region and its target prediction result; Each of the above prediction results is used to indicate whether the target detection region predicted by the above image element detection model includes image elements of a preset type. The above image element detection model is trained based on the training method of the image element detection model provided in the embodiments of this application. The image element determination module is used to determine whether the target detection region actually includes image elements of a preset type based on the target prediction results of each of the above target detection regions.

[0009] On the other hand, embodiments of this application provide an electronic device, including a processor and a memory, which are interconnected; The memory is used to store computer programs; The processor is configured to execute, when the computer program is invoked, the training method of the image element detection model or the image element detection method provided in the embodiments of this application.

[0010] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the training method or image element detection method of the image element detection model provided in embodiments of this application.

[0011] On the other hand, embodiments of this application provide a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the training method or image element detection method of the image element detection model provided in embodiments of this application.

[0012] The training method for the image element detection model provided in this application can iteratively adjust each initial detection region of the sample image based on the initial image features of the sample image, thereby making the final target detection region more effective in detecting image elements in the sample image. Furthermore, in each iteration, the initial prediction result of the detection region obtained in each iteration can be determined based on the initial image features of the sample image, and the target prediction result of the final target detection region can be determined based on the initial prediction result of the detection region obtained in each iteration, thereby improving the accuracy of the target prediction result of each target detection region. Based on this training method, an image element detection model for a preset type of image elements in an image can be accurately determined.

[0013] On the other hand, for the image to be processed, after the target detection region of the image to be processed is determined based on the trained image element detection model and the target prediction result used to indicate whether the image element detection model predicts whether each target detection region includes a preset type of image element, it is possible to further determine whether the image element corresponding to each target detection region is a preset type of image element based on the element features of the image element corresponding to each target detection region and the corresponding target prediction result, effectively ensuring the detection accuracy of the preset type of image element and having high applicability. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the training method of the image element detection model provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of a deep residual network provided in an embodiment of this application; Figure 4 This is a flowchart illustrating the process of determining the target prediction result provided in an embodiment of this application; Figure 5 This is a schematic diagram of a scenario for determining candidate detection regions provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of the adjustment network provided in an embodiment of this application; Figure 7 This is a schematic diagram of a scenario for determining the fusion prediction result provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of the result processing network provided in the embodiments of this application; Figure 9 This is a schematic diagram of a network structure of the initial model provided in the embodiments of this application; Figure 10 This is a schematic flowchart of the image element detection method provided in the embodiments of this application; Figure 11 This is a schematic diagram of an image element detection scenario provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of the initial element processing network provided in the embodiments of this application; Figure 13 This is a schematic diagram of the structure of the training device for the image element detection model provided in the embodiments of this application; Figure 14 This is a schematic diagram of the image element detection device provided in the embodiments of this application; Figure 15 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] The type detection method provided in this application embodiment is applicable to any image detection scenario in the field of image processing. For example, in the fields of intelligent transportation or mapping, the image element detection method provided in this application embodiment can be applied to Intelligent Traffic Systems (ITS) or Intelligent Vehicle Infrastructure Cooperative Systems (IVICS) to identify image elements of a preset type in road images, such as pedestrians and traffic lights, thereby providing data support for vehicle-road cooperation, traffic navigation, and road facility detection. Intelligent Transportation Systems (ITS), also known as Intelligent Transportation Systems, effectively integrate advanced technologies (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing, strengthening the connection between vehicles, roads, and users to form a comprehensive transportation system that ensures safety, improves efficiency, improves the environment, and saves energy.

[0018] Intelligent vehicle-road cooperative systems, or vehicle-road cooperative systems for short, represent a development direction for Intelligent Transportation Systems (ITS). These systems utilize advanced wireless communication and next-generation internet technologies to implement comprehensive, real-time dynamic information exchange between vehicles and infrastructure. Based on the collection and fusion of dynamic traffic information across all times and spaces, they enable proactive vehicle safety control and collaborative road management, fully realizing effective coordination between people, vehicles, and roads. This ensures traffic safety, improves traffic efficiency, and ultimately forms a safe, efficient, and environmentally friendly road traffic system.

[0019] For example, in the field of video editing, a video frame containing image elements of a preset type can be determined based on the image element detection method provided in the embodiments of this application, and then the video can be edited based on the video frame containing image elements of the preset type to obtain the required video (such as each video frame containing a car).

[0020] It should be noted that, in any image detection scenario, an image element detection model applicable to the image detection scenario needs to be trained first based on the image element detection model training method provided in the embodiments of this application. Then, based on the trained image element detection model, the image elements belonging to the preset type in the corresponding image detection scenario are determined based on the image element detection method provided in the embodiments of this application.

[0021] See Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application. For example... Figure 1As shown, in some feasible implementations, server 20 can train a model based on the initial image features of multiple sample images to obtain an image element prediction model that can predict image elements of a preset type included in any image.

[0022] Among them, server 20 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0023] In some feasible implementations, terminal 10 can process the image to be processed based on the type prediction model trained by server 20 to predict the type of each image element in the image to be processed.

[0024] The terminal 10 may include, but is not limited to, smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, in-vehicle terminals, smart wearable devices, etc., without limitation. Optionally, the operating system running on the terminal 10 may include, but is not limited to, Android, iOS, Linux, Windows, etc.

[0025] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided by the embodiment of this application. In actual applications, other application environments may also be included. For example, the training process of the image element detection model can be implemented on the terminal 10, and the process of determining the type of image elements based on the image element detection model can also be implemented on the server 20. No limitation is made here.

[0026] Terminal 10 and server 20 can be connected directly or indirectly via wired or wireless communication, without any restrictions.

[0027] In this embodiment, the training process of the image element detection model can be implemented based on machine learning in artificial intelligence. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions. Machine learning specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structures to continuously improve their performance, and thus implement the training process of the image element detection model in this embodiment.

[0028] See Figure 2 , Figure 2 This is a schematic flowchart of the training method for the image element detection model provided in the embodiments of this application. Figure 2 As shown, the training method for the image element detection model provided in this application embodiment may specifically include the following steps: Step S21: Input the initial image features of each sample image into the initial model to obtain the target prediction result of at least one target detection region of the sample image.

[0029] In some feasible implementations, after determining multiple sample images, features can be extracted from each sample image based on a pre-trained feature extraction network to obtain the initial image features of each sample image.

[0030] The pre-trained feature extraction network can be a deep residual network, such as the ResNet34 network. The specific network can be determined based on the actual application scenario requirements, and no restrictions are imposed here.

[0031] For example, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a deep residual network provided in an embodiment of this application. For each sample image, the sample image can be downsampled based on a downsampling convolutional layer, and the downsampled convolutional features can be processed based on multiple residual blocks to obtain intermediate features. The intermediate features are then further processed based on pooling layers and fully connected layers to obtain the initial image features of the sample image.

[0032] The network layer settings for each residual block can be determined based on the actual application scenario requirements, and there are no restrictions here.

[0033] In some feasible implementations, for each sample image, the initial image features of the sample image are input into the initial model to obtain at least one target detection region of the sample image and its corresponding target prediction result.

[0034] The target prediction result for each target detection region is used to indicate whether the model predicts that the target detection region includes image elements of a preset type.

[0035] For each sample image, any two target detection regions in the sample image can correspond to the same image element or different image elements. That is, each image element in the sample image can correspond to at least one target detection region.

[0036] For any target detection region in each sample image, the image elements corresponding to the target detection region can be all located within the target detection region or partially located within the target detection region; there is no restriction on this.

[0037] Among them, the target prediction results of different target detection regions corresponding to the same image element may be the same or different. For example, in the early stage of model training, there is a greater possibility that the target prediction results of different target detection regions corresponding to the same image element will be different, while in the later stage of model training, there is a greater possibility that the target prediction results of different target detection regions corresponding to the same image element will remain consistent.

[0038] The specific preset types mentioned above can be determined based on the actual application scenario requirements and are not limited here. For example, the preset types can be people, identification information of a specific range (such as container identification information), etc.

[0039] For details on how the initial model determines the target prediction results for each target region, please refer to [link / reference needed]. Figure 4 , Figure 4 This is a flowchart illustrating the process of determining the target prediction result provided in an embodiment of this application. Figure 4 The specific method for determining the target prediction result shown may include the following steps: Step S41: Based on the initial image features of each sample image, perform a preset number of iterative adjustments on at least one initial detection region of the sample image to obtain the corresponding target detection region.

[0040] In some feasible implementations, for each sample image, after the initial image features of the sample image are input into the initial model, the initial model can process the initial image features of each sample image based on the Region Proposal Network (PRN) to obtain at least one initial detection region of the sample image.

[0041] Specifically, for each sample image, since each feature position of the initial image features of the sample image corresponds to an image region in the sample image, after inputting the initial image features of the sample image into the initial model, multiple image regions of the sample image can be determined based on the initial image features of the sample image.

[0042] Furthermore, for each image region, any position of that image region (such as the upper left corner, the center position, etc.) can be used as an anchor point, and multiple target image regions can be determined from the corresponding sample images as candidate detection regions based on various aspect ratios and scaling values.

[0043] The specific mapping value and aspect ratio can be determined based on the actual application scenario requirements. For example, the scaling value can be 2 and the aspect ratio can be 2:1 (in which case the length of the candidate detection area is 4 and the width is 2), and there are no restrictions here.

[0044] See Figure 5 , Figure 5 This is a schematic diagram of a scenario for determining candidate detection regions provided in an embodiment of this application. For example... Figure 5 As shown, for any feature location of the initial image features of a sample image (as marked in the figure), multiple candidate detection regions of different sizes can be determined from the sample image based on the image region of that feature location with different aspect ratios and scaling values.

[0045] In this process, after determining all candidate detection regions corresponding to all sample images, each candidate detection region can be regarded as a candidate training sample used in this model training process.

[0046] Furthermore, for each sample image, after determining all candidate detection regions corresponding to that sample image, the image region of each image element in the sample image can be determined. The image region of each image element in the sample image is a region of a preset size that completely encompasses that image element; the specific size can be determined based on the actual application scenario requirements and is not limited here.

[0047] For each candidate detection region in each sample image, the intersection-union ratio (IU) between the candidate detection region and the image region of each image element in the sample image can be determined (hereinafter referred to as the first IU). If there is a first IU among the first IU corresponding to the candidate detection region that is greater than a first threshold, or if all the first IU corresponding to the candidate detection region are less than a second threshold, then the candidate detection region is determined as an initial detection region of the sample image.

[0048] The first threshold is greater than the second threshold, and the first and second thresholds can be determined based on the actual application scenario requirements. For example, the first threshold can be 0.5 and the second threshold can be 0.3, without any restrictions.

[0049] Specifically, the process of determining the initial detection region from each candidate detection region of a sample image can be viewed as the process of determining the training samples used in this model training process from each candidate training sample. Specifically, for each candidate detection region of each sample image, if any of the first intersection-union ratios (IURs) corresponding to the candidate detection region is greater than a first threshold, then the candidate detection region can be considered a positive sample; if all the first IURs corresponding to the candidate detection region are less than a second threshold, then the candidate detection region is considered a negative sample.

[0050] In some feasible implementations, for each sample image, during the process of iteratively adjusting the sample image and the initial detection region based on the initial image features of the sample image to obtain the corresponding target detection region, for each adjustment, the first detection region can be adjusted based on the initial image features of the sample image to obtain the corresponding second detection region and the initial prediction result corresponding to the second detection region.

[0051] The specific number of preset times can be determined based on the actual application scenario requirements, and no restrictions are imposed here.

[0052] Each initial prediction result is used to indicate whether the second detection region corresponding to the model prediction includes image elements of a preset type. Each adjustment of the corresponding first detection region is the second detection region obtained in the previous adjustment. The first adjustment of the corresponding first detection region is the initial detection region of the sample image. The second detection region obtained in the last adjustment is determined as the target detection region of the sample image.

[0053] The initial prediction result of each second detection region obtained in each adjustment can be represented as the probability that the second detection region includes image elements of a preset type. When the probability corresponding to the initial prediction result is greater than the fifth threshold, it means that the model predicts that the corresponding second detection region includes image elements of a preset type; otherwise, it means that the model predicts that the corresponding second detection region does not include image elements of a preset type.

[0054] In each adjustment, the position information of each first detection region of the sample image is mapped to the initial image features of the sample image to obtain the fused image features of the sample image. Then, based on the fused image features, the second detection region corresponding to each first detection region of the sample image and its corresponding initial prediction result are determined.

[0055] The initial model includes multiple cascaded adjustment networks, with each adjustment based on the corresponding first-level adjustment network. For example... Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of the adjustment network provided in an embodiment of this application. For example... Figure 6 The diagram illustrates an arbitrary adjustment network in the initial model. After obtaining the second detection regions based on the previous adjustment network, the initial image features of the sample image and the first detection regions obtained from the previous adjustment network are input into the feature fusion network within the adjustment network to obtain the fused image features of the sample image. Then, the feature processing network processes the fused image features to obtain the second detection region corresponding to each first detection region and its corresponding initial prediction result. Furthermore, the second detection region obtained in this process becomes the first detection region corresponding to the next adjustment network.

[0056] The network parameters for each level of network adjustment can be the same or different, depending on the actual application scenario requirements, and no restrictions are imposed here.

[0057] Step S42: Based on the initial prediction results of the second detection region obtained from each adjustment, determine the target prediction results of each target detection region of the sample image.

[0058] In some feasible implementations, for each sample image, after obtaining each target detection region of the sample image, for each initial detection region of the sample image, the initial prediction results of the second detection region corresponding to the initial detection region obtained each time can be fused to obtain the target prediction result of the target detection region corresponding to the initial detection region.

[0059] For each sample image, after obtaining each target detection region of the sample image, the weight sum of the initial prediction results of the second detection region corresponding to the initial detection region obtained in each adjustment can be determined, and this weight sum can be determined as the target prediction result of the target detection region corresponding to the initial detection region. Similarly, the target prediction result of each target detection region can be represented as the probability that the target detection region includes image elements of a preset type.

[0060] The weights corresponding to the second detection regions obtained from any two adjustments can be the same or different, and can be determined based on the actual application scenario requirements, without any restrictions here.

[0061] Optionally, when determining the target prediction result of each target detection region of the sample image, for each initial detection region of the sample image, the fusion prediction result corresponding to the initial detection region is determined based on the initial prediction result of the second detection region corresponding to the initial detection region obtained in each adjustment except the last adjustment.

[0062] Specifically, the weighted sum of the initial prediction results of the second detection region corresponding to the initial detection region obtained from each adjustment other than the last adjustment can be determined as the fusion prediction result corresponding to the initial detection region.

[0063] The weights corresponding to the second detection regions obtained from any two adjustments can be the same or different, and can be determined based on the actual application scenario requirements, without any restrictions here.

[0064] Furthermore, the initial detection region corresponding to the initial detection region and the prediction result, as well as the initial prediction result of the second detection region corresponding to the initial detection region obtained from the last adjustment, can be fused to obtain the target prediction result of the target detection region corresponding to the initial detection region.

[0065] Specifically, the target prediction result of the target detection region corresponding to the initial detection region can be determined by summing the weights of the fusion prediction result corresponding to the initial detection region and the initial prediction result of the second detection region corresponding to the initial detection region obtained by the last adjustment.

[0066] The weights of the fused prediction result and the initial prediction result of the second detection region corresponding to the initial detection region obtained from the last adjustment can be the same or different. The specific weights can be determined based on the actual application scenario requirements and are not restricted here.

[0067] Specifically, for the initial prediction results of the second detection region corresponding to the initial detection region obtained in each adjustment other than the last adjustment, in order to enhance the initial model's ability to recognize the initial prediction results, the initial prediction results can be further processed based on the result processing network to obtain the first prediction result. Then, the fusion prediction result of the initial detection region is determined by determining the first prediction results of the second detection region corresponding to the initial detection region obtained in each adjustment other than the last adjustment.

[0068] Each adjustment can use a different result processing network or the same result processing network, and the specific network parameters of each result processing network can be determined based on the actual application scenario requirements, without any restrictions here.

[0069] See Figure 7 , Figure 7This is a schematic diagram of a scenario for determining the fusion prediction result provided in an embodiment of this application. For each sample image, if the initial detection region of the sample image is adjusted N times to obtain the target detection region, then for each initial detection region, the initial prediction result of the second detection region corresponding to the initial detection region obtained from the first adjustment can be input into the result processing network to obtain the first prediction result corresponding to the adjustment, the initial prediction result of the second detection region corresponding to the initial detection region obtained from the second adjustment can be input into the result processing network to obtain the first prediction result corresponding to the adjustment, ..., and the initial prediction result of the second detection region corresponding to the initial detection region obtained from the (N-1)th adjustment can be input into the result processing network to obtain the first prediction result corresponding to the adjustment.

[0070] Furthermore, the first prediction result of the second detection region corresponding to the initial detection region, obtained from the first adjustment to the N-1th adjustment, can be fused to obtain the fused prediction result of the initial detection region.

[0071] Specifically, when processing the initial prediction results based on the result processing network, the prediction features used to characterize the initial prediction results can be processed based on the result processing network. The prediction features of the initial prediction results can be the input features of the fully connected layer that determines the initial prediction results.

[0072] See Figure 8 , Figure 8 This is a schematic diagram of the structure of the result processing network provided in an embodiment of this application. For example... Figure 8 As shown, each result processing network may include multiple convolutional layers and one fully connected layer, with skip connections between every two convolutional layers, and the network parameters of each result processing network are independent and not shared. After inputting each initial prediction result (or corresponding type feature) into the result processing network, the result processing network can output the corresponding first prediction result.

[0073] The following is combined Figure 9 The network structure of the initial model in the embodiments of this application will be further described. Figure 9 This is a schematic diagram of a network structure of the initial model provided in an embodiment of this application. For example... Figure 9 As shown, for each sample image, after determining the initial image features of the sample image, the initial image features of the sample image can be input into the initial model.

[0074] The initial model processes the initial image features based on the PRN network to obtain multiple initial detection regions, and then adjusts the initial detection regions based on the multi-level adjustment network to obtain the target detection region corresponding to each initial detection region.

[0075] The initial model maps the location information of the initial detection regions to initial image features to obtain fused image features. A first-level adjustment network processes this fused image to obtain the second detection region corresponding to each initial detection region and its corresponding initial prediction result. The location information of each second detection region obtained from the first-level adjustment network is mapped to the initial image features to obtain fused image features. A second-level adjustment network processes these fused image features to obtain new second detection regions and their corresponding initial prediction results. The location information of each second detection region obtained from the second-level adjustment network is mapped to the initial image features to obtain fused image features. A third-level adjustment network processes these fused image features to obtain new second detection regions and their corresponding initial prediction results. The location information of each second detection region obtained from the third-level adjustment network is mapped to the initial image features to obtain fused image features. A final-level adjustment network processes these fused image features to obtain the target detection region corresponding to each initial detection region and its corresponding initial prediction result.

[0076] For each initial detection region, the initial model can process the initial prediction results of the second detection region corresponding to the initial detection network obtained by the first-level adjustment network, the second-level adjustment network, and the third-level adjustment network to obtain the corresponding first prediction result through the result processing network corresponding to each adjustment network, and obtain the fusion prediction result corresponding to the initial detection region based on each first prediction result.

[0077] Furthermore, the initial model can fuse the initial prediction result of the second detection region corresponding to the initial detection region obtained from the fourth-level adjustment network with the fusion prediction result corresponding to the initial detection region to obtain the target prediction result of the target prediction region corresponding to the initial prediction result. Then, based on the target prediction result of each target detection region, it can be determined whether the initial model predicts that the target detection region actually includes image elements of a preset type.

[0078] Step S22: Determine the total training loss based on the target prediction results and the true classification results of each target detection region. Train the initial model based on each sample image until the total training loss meets the preset conditions, and stop training. Then, determine the model at the time of training stop as the image element detection model.

[0079] In this embodiment, each image element in each sample image is also labeled with a type label to characterize whether the corresponding image element is an image element of a preset type. Based on this, after determining each target detection region, the true classification result of each target detection region can be determined based on the type label of each image element in each sample image.

[0080] In this embodiment of the application, each target prediction result represents the probability that the corresponding target detection area includes image elements of a preset type. If the probability corresponding to the target prediction result is greater than the fifth threshold, it means that the model predicts that the corresponding target detection area includes image elements of a preset type; otherwise, it means that the model predicts that the corresponding second detection area does not include image elements of a preset type.

[0081] In some feasible implementations, when determining the total training loss, a first training loss can be determined based on the target prediction results and the true classification results of each target detection region. Simultaneously, the image region containing each image element in each sample image can be determined, and then a second training loss can be determined based on the image region containing each image element in each sample image and each target detection region.

[0082] The true classification result of each target detection region is used to indicate whether the corresponding target detection region actually includes image elements of a preset type.

[0083] Furthermore, the total training loss can be determined based on the first training loss and the second training loss. For example, the sum of the first training loss and the second training loss can be used as the total training loss, or the weighted sum of the first training loss and the second training loss can be used as the total training loss. The specific loss weights corresponding to the first training loss and the second training loss can be determined based on the actual application scenario requirements, and are not restricted here.

[0084] Specifically, when determining the first training loss, the cross-entropy loss between the target prediction result and the true classification result of each target detection region can be determined, and this cross-entropy loss can be used as the first training loss.

[0085] Optionally, when determining the first training loss, the cross-entropy loss between the target prediction result and the true classification result of each target detection region can also be determined, and this cross-entropy loss can be determined as the first sub-loss.

[0086] Simultaneously, the true classification result of each second detection region obtained in each adjustment can be determined. For example, the true classification result of each second detection region obtained in each adjustment can be determined based on the type label of each image element in each sample image. Furthermore, the second sub-loss can be determined based on the initial prediction result and the true classification result of each second detection region obtained in each adjustment.

[0087] Among them, the cross-entropy loss of the initial prediction result and the true classification result of each second detection region obtained in each adjustment can be determined, and this cross-entropy loss is determined as the second sub-loss.

[0088] The true classification result corresponding to each second detection region is used to indicate whether the second detection region actually includes image elements of a preset type.

[0089] Simultaneously, the true classification result of each initial detection region can be determined, such as based on the type label of each image element in each sample image. Furthermore, based on the initial prediction result and the true classification result of each initial detection region, a third sub-loss can be determined.

[0090] Among them, the cross-entropy loss of the initial prediction result and the true classification result of each initial detection region can be determined, and this cross-entropy loss is determined as the third sub-loss.

[0091] The true classification result corresponding to each initial detection region is used to indicate whether the initial detection region actually includes image elements of a preset type, and the initial prediction result of each initial detection region is used to indicate whether the model predicts whether the initial detection region includes image elements of a preset type.

[0092] Specifically, for each initial detection region, as mentioned above, among the first intersection-union ratios (IURs) of each initial detection region and the regions containing each image element in the corresponding sample image, there is either an IUR greater than a first threshold, or all IURs are less than a second threshold. Therefore, for each initial detection region, if the first IUR of the initial detection region and the image regions containing at least one image element in the corresponding sample image is greater than the first threshold, then the model predicts that the initial detection region includes image elements of a preset type. If the first IUR of the initial detection region and the image regions containing each image element in the corresponding sample image are all less than the second threshold, then the model predicts that the initial detection region does not include image elements of a preset type.

[0093] The first threshold is greater than the second threshold.

[0094] When determining the first training loss, it can be based on at least one of the second or third sub-loss, as well as the first sub-loss.

[0095] The first training loss can be determined by summing the sub-losses, such as by summing the first, second, and third sub-losses. Alternatively, the first training loss can be determined by summing the weights of the sub-losses, such as by summing the weights of the first, second, and third sub-losses. The weights of each sub-loss can be determined based on the requirements of the actual application scenario and are not restricted here.

[0096] Specifically, when determining the second training loss, the bounding box regression loss (smooth L1) of each target detection region and the image region where each image element in each sample image is located can be determined, and this loss can be used as the second training loss.

[0097] Optionally, when determining the second training loss, the bounding box regression loss of each target detection region and the image region where each image element in each sample image is located can also be determined, and this loss can be determined as the fourth sub-loss.

[0098] Simultaneously, the fifth sub-loss can be determined based on the second detection regions obtained from each adjustment other than the last adjustment and the image region where each image element in each sample image is located. For example, the bounding box regression loss of the second detection regions obtained from each adjustment other than the last adjustment and the image region where each image element in each sample image is located can be determined as the fifth sub-loss.

[0099] Simultaneously, a sixth sub-loss can be determined based on the image region where each initial detection region and each image element in each sample image are located. For example, the bounding box regression loss of each initial detection region and the image region where each image element in each sample image is located can be determined as the sixth sub-loss.

[0100] When determining the second training loss, it can be based on at least one of the fifth or sixth sub-loss and the fourth sub-loss.

[0101] The sum of all sub-losses can be used as the second training loss, for example, the sum of the fourth, fifth, and sixth sub-losses can be used as the first training loss. Alternatively, the sum of the weights of all sub-losses can be used as the second training loss, for example, the sum of the weights of the fourth, fifth, and sixth sub-losses can be used as the second training loss. The weights of each sub-loss can be determined based on the actual application scenario requirements and are not restricted here.

[0102] Specifically, for any second detection region in any adjustment process, the intersection-union ratio (IU) between this second detection region and the regions containing each image element in the corresponding sample image can be determined (hereinafter referred to as the second IU). If any of the second IU corresponding to this second detection region is greater than the third threshold, then this second detection region can be considered a positive sample; if all the second IU corresponding to this second detection region are less than the fourth threshold, then this second detection region can be considered a negative sample. Furthermore, the third and fourth thresholds corresponding to each adjustment process can be determined based on the actual application scenario requirements and are not restricted here.

[0103] In some feasible implementations, after determining the total training loss for each training session, the initial model can be iteratively trained repeatedly based on the initial image features of each sample image until the total training loss meets the preset conditions, at which point training stops, and the model at the point where training stops is determined as the image element detection model.

[0104] The aforementioned preset conditions can be either the total training loss reaching convergence or determined based on the actual application scenario requirements, and are not restricted here.

[0105] This application also provides an image element detection method, which can be found in the following embodiments: Figure 10 . Figure 10 This is a schematic flowchart of the image element detection method provided in the embodiments of this application. Figure 10 As shown, the image element detection method provided in this application embodiment may specifically include the following steps: Step S101: Determine the initial image features of the image to be processed, input the initial image features of the image to be processed into the image element detection model, and obtain at least one target detection region and its target prediction result.

[0106] In some feasible implementations, the specific implementation of the image element detection model in determining at least one target detection region and its target prediction result based on the initial image features of the image to be processed can be found in the embodiments of this application. Figure 2 The implementation method shown in step S21 will not be described again here.

[0107] The image element detection model is a model trained based on the image element detection model training method provided in the embodiments of this application.

[0108] Each target prediction result is used to indicate whether the target detection region predicted by the image element detection model includes image elements of a preset type.

[0109] like Figure 11 As shown, Figure 11 This is a schematic diagram of an image element detection scenario provided in an embodiment of this application. If the preset type of image element is a warning sign, then based on the image element detection model, it can be determined that... Figure 11 The image element detection model identifies multiple target detection regions and obtains a target prediction result for each target detection region. Each target prediction result indicates whether the image element corresponding to the target detection region predicted by the image element detection model is an image element of a preset type; that is, it instructs the image element detection model to determine whether the image element corresponding to the target detection region is a warning sign.

[0110] Step S102: Based on the target prediction results of each target detection region, determine whether the target detection region actually includes image elements of a preset type.

[0111] In some feasible implementations, each target prediction result can be represented as the probability that the corresponding target detection region includes image elements of a preset type. Based on this, for each target detection region, if the probability corresponding to the target prediction result is greater than a fifth threshold, it indicates that the image element detection model predicts that the corresponding target detection region includes image elements of a preset type; otherwise, it indicates that the image element detection model predicts that the corresponding second detection region does not include image elements of a preset type.

[0112] In some feasible implementations, for each target detection region, when determining whether the target detection region actually includes image elements of a preset type, the target element features of the image elements corresponding to the target detection region can also be determined, and then the similarity between the target element features and the preset element features of each preset type of image element in the feature set can be determined.

[0113] The aforementioned similarity can be cosine similarity or determined in other ways, and no restriction is imposed here.

[0114] In determining the image elements corresponding to the target detection region, the target detection region can be cropped from the image to be processed to obtain the image elements corresponding to the target detection region.

[0115] Each preset element feature and the target element feature corresponding to the target detection region are determined based on a pre-trained element processing network.

[0116] Specifically, for any image element among the image elements of each preset type and the image elements corresponding to the target detection region, the image element is scaled to a preset pixel size (e.g., 224 * 224 pixels) to obtain a scaled image element. The scaled image element is then processed based on a pre-trained element processing network to obtain element features. If the image element is an image element of a preset type, the determined element features are the preset element features of that preset type of image element; if the image element is an image element corresponding to the target detection region, the determined element features are the target element features of the image element corresponding to that target detection region.

[0117] Furthermore, for each preset target detection region, the maximum similarity between the corresponding target element features and the preset element features of each preset type of image element in the feature set can be determined. Based on the maximum similarity and the target prediction result of the target detection region, it can be determined whether the target detection region actually includes the preset type of image element.

[0118] If, based on the target prediction result of the target detection region, it is determined that the image element detection model predicts that the target detection region does not include image elements of a preset type, and the maximum similarity is less than the first similarity threshold, then it is determined that the target detection region does not actually include image elements of a preset type. If, based on the target prediction result of the target detection region, the image element detection model predicts that the target detection region includes image elements of a preset type, and the maximum similarity is greater than the second similarity threshold, then the target detection region is determined to include image elements of the preset type.

[0119] The first similarity threshold is less than or equal to the second similarity threshold. The specific value can be determined based on the actual application scenario requirements and is not restricted here.

[0120] In other cases, the image element corresponding to the target detection area can be sent to the review network, which will determine whether the image element is an image element of a preset type.

[0121] Optionally, if each target prediction result represents the probability that the corresponding target detection region includes image elements of a preset type, for each target detection region, if the probability corresponding to the target detection region is greater than or equal to the sixth threshold, then it is determined that the target detection region actually includes image elements of the preset type.

[0122] If the probability corresponding to the target detection region is less than the sixth threshold, it indicates that the target detection region may not include image elements of the preset type. At this time, the similarity between the target element features of the image element corresponding to the target detection region and the preset element features of each preset type of image element in the feature set can be determined. Based on the maximum similarity and the target prediction result of the target detection region, it can be determined whether the target detection region actually includes image elements of the preset type.

[0123] The sixth threshold is greater than the fifth threshold. For example, if the sixth threshold is 0.8, the fifth threshold is 0.6.

[0124] In other cases, the image element corresponding to the target detection area can be sent to the review network, which will determine whether the image element is an image element of a preset type.

[0125] In this embodiment of the application, the pre-trained element processing network described above can be trained in the following manner: Obtain a training sample set, which includes multiple sample image elements. Each sample image element is labeled with a type label, and each type label is used to characterize whether the corresponding sample image element is an image element of a preset type. All sample images are assumed to have the same preset pixel size, such as 224 * 224 pixels, and no restrictions are imposed here.

[0126] Each sample image element is input into the initial element processing network to obtain the sample element features of each sample image element. The sample element features are then input into the first fully connected layer to obtain the classification result of each sample image element. Each classification result is used to characterize whether the corresponding sample image element is predicted to be an image element of a preset type. Based on the actual type attribution result of each sample image element (whether it actually belongs to the preset type of image element) and the corresponding classification result, the total training loss is determined, and the initial element processing network and the first fully connected layer are trained based on each sample image element until the total training loss meets the stopping condition. When training stops, the initial element processing network at the time of training stop is determined as the pre-trained element processing network.

[0127] The initial element processing network can be formed by concatenating the feature processing network (excluding the fully connected layer) of an existing binary image element classification network with a second fully connected layer. The second fully connected layer has a preset dimension (e.g., 256 dimensions), and the first fully connected layer is the existing 2D fully connected layer of the image classification network. The feature dimension of the second fully connected layer is higher than that of the first fully connected layer, thus ensuring that the preset element features or target element features are fully covered when determining the preset element features or target element features based on the pre-trained element processing network.

[0128] See Figure 12 , Figure 12 This is a schematic diagram of the structure of the initial element processing network provided in an embodiment of this application. For example... Figure 12 As shown, the feature processing network and the first fully connected layer constitute the existing binary image element classification network. A second fully connected layer is added between the feature processing network and the first fully connected layer, and the feature processing network and the second fully connected layer are determined as the initial element processing network. The initial element processing network and the first fully connected layer are trained based on each sample image element, and the initial element processing network at the end of training is determined as the final pre-trained element processing network.

[0129] The image processing involved in the embodiments of this application can be implemented based on computer vision technology. Computer vision is a science that studies how to make machines "see". More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes to identify and measure targets, and then performs image processing, such as image cropping.

[0130] In this application embodiment, the relevant data during the model training process can be stored in a database, cloud storage, or blockchain. The specific storage location can be determined based on the actual application scenario requirements, and no restrictions are imposed here.

[0131] In short, a database can be viewed as an electronic filing cabinet—a place to store electronic files. In this application, it can be used to store data related to the model training process. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked using cryptographic methods. In this application, each data block in the blockchain can store data related to the model training process. Cloud storage is a new concept extended and developed from cloud computing. It refers to the collaborative storage of data related to the model training process by aggregating a large number of various types of storage devices (also called storage nodes) in a network through application software or application interfaces, using cluster applications, grid technology, and distributed storage file systems.

[0132] The data processing and computation processes involved in the embodiments of this application can be performed based on cloud computing and other methods. Cloud computing is a product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. Based on cloud computing, the efficiency of data processing and computation in the embodiments of this application can be improved.

[0133] The training method for the image element detection model provided in this application can iteratively adjust each initial detection region of the sample image based on the initial image features of the sample image, thereby making the final target detection region more effective in detecting image elements in the sample image. Furthermore, in each iteration, the initial prediction result of the detection region obtained in each iteration can be determined based on the initial image features of the sample image, and the target prediction result of the final target detection region can be determined based on the initial prediction result of the detection region obtained in each iteration, thereby improving the accuracy of the target prediction result of each target detection region. Based on this training method, an image element detection model for a preset type of image elements in an image can be accurately determined.

[0134] On the other hand, for the image to be processed, after the target detection region of the image to be processed is determined based on the trained image element detection model and the target prediction result used to indicate whether the image element detection model predicts whether each target detection region includes a preset type of image element, since the image element detection model may have prediction errors, it is possible to further determine whether the image element corresponding to each target detection region is a preset type of image element based on the element features of the image element corresponding to each target detection region and the corresponding target prediction result, which effectively ensures the detection accuracy of the preset type of image element and has high applicability.

[0135] See Figure 13 , Figure 13 This is a schematic diagram of the structure of the training device for the image element detection model provided in this application embodiment. The training device for the image element detection model provided in this application embodiment includes: The detection module 131 is used to input the initial image features of each sample image into the initial model to obtain the target prediction result of at least one target detection region of the sample image. Each of the above target prediction results is used to indicate whether the target detection region corresponding to the model prediction includes image elements of a preset type. Training module 132 is used to determine the total training loss based on the target prediction results and the true classification results of each of the above target detection regions, train the above initial model based on each of the above sample images until the total training loss meets the preset conditions and then stop training, and determine the model when training stops as the image element detection model. Each of the above true classification results is used to indicate whether the corresponding target detection region actually includes image elements of a preset type. The detection module 131 determines the prediction results of each of the above targets based on the following method: Based on the initial image features of each sample image, at least one initial detection region of the sample image is iteratively adjusted a predetermined number of times to obtain the corresponding target detection region, wherein each adjustment is performed in the following manner: Based on the initial image features of the sample image, each first detection region is adjusted to obtain a second detection region and its corresponding initial prediction result. Each initial prediction result is used to indicate whether the second detection region corresponding to the model prediction includes image elements of a preset type. Each adjustment of the corresponding first detection region is the second detection region obtained in the previous adjustment. The first adjustment of the corresponding first detection region is the aforementioned initial detection region. The second detection region obtained in the last adjustment is the target detection region of the sample image. Based on the initial prediction results of the second detection region obtained from each adjustment, the target prediction results of each target detection region of the sample image are determined.

[0136] In some feasible implementations, for each of the above sample images, the detection module 131 is configured to: Based on the initial image features of the sample image, each candidate detection region of the sample image is determined; For each of the above candidate detection regions, a first intersection-union ratio (IUU) is determined between the candidate detection region and the image region containing each image element in the sample image. If there is an IUU greater than a first threshold, or if all of the above IUU are less than a second threshold, then the candidate detection region is determined as an initial detection region of the sample image. The first threshold is greater than the second threshold.

[0137] In some feasible implementations, for each of the above sample images, the detection module 131 is configured to: The location information of each first detection region of the sample image is mapped to the initial image features of the sample image to obtain the fused image features; Based on the aforementioned fused image features, the second detection region corresponding to each first detection region of the sample image and its corresponding initial prediction result are determined.

[0138] In some feasible implementations, for each of the above sample images, the detection module 131 is configured to: For each initial detection region of the sample image, based on the initial prediction results of the second detection region corresponding to the initial detection region obtained in each adjustment except the last adjustment, the fusion prediction result corresponding to the initial detection region is determined. Based on the above fusion prediction result and the initial prediction result of the second detection region corresponding to the initial detection region obtained in the last adjustment, the target prediction result of the target detection region corresponding to the initial detection region is determined.

[0139] In some feasible implementations, the training module 132 described above is used for: The first training loss is determined based on the target prediction results and the true classification results of each of the above target detection regions. Determine the image region where each image element is located in each of the above sample images, and determine the second training loss based on the image region where each image element is located in each of the above sample images and each of the above target detection regions; The total training loss is determined based on the first training loss and the second training loss mentioned above.

[0140] In some feasible implementations, the training module 132 described above is used for: The first sub-loss is determined based on the target prediction results and the true classification results of each of the above-mentioned target detection regions. The first training loss is determined based on at least one of the second sub-loss or the third sub-loss, as well as the first sub-loss. When determining the second sub-loss, the training module 132 is used for: Determine the true classification result of each second detection region obtained from each adjustment; Based on the initial prediction results and the true classification results of each second detection region obtained from each adjustment, the aforementioned second sub-loss is determined; The third sub-loss mentioned above is determined, and the training module 132 mentioned above is used for: For each of the above initial detection regions, the initial prediction result of the initial detection region is determined based on the initial detection region and the image region where each image element in the corresponding sample image is located; The true classification result of each of the above initial detection regions is determined, and the third sub-loss is determined based on the initial prediction result and the true classification result of each of the above initial detection regions.

[0141] In some feasible implementations, for each of the aforementioned initial detection regions, the training module 132 is configured to: If the first intersection-union ratio of the initial detection region with the image region containing at least one image element in the corresponding sample image is greater than the first threshold, then the model predicts that the initial detection region includes image elements of a preset type. If the first intersection-union ratio of the initial detection region and the image regions where each image element in the corresponding sample image is located is less than the second threshold, then it is determined that the model predicts that the initial detection region does not include image elements of the preset type.

[0142] In some feasible implementations, the training module 132 described above is used for: The fourth sub-loss is determined based on each of the above-mentioned target detection regions and the image region where each image element in each of the above-mentioned sample images is located. The second training loss is determined based on at least one of the fifth sub-loss or the sixth sub-loss and the above-mentioned fourth sub-loss. The fifth sub-loss is determined by each second detection region obtained from each adjustment except the last adjustment and the image region where each image element in each of the above sample images is located. The sixth sub-loss is determined by each of the above initial detection regions and the image region where each image element in each of the above sample images is located.

[0143] In specific implementation, the training device for the aforementioned image element detection model can execute the above-described functions through its built-in functional modules. Figure 2 and / or Figure 4 For details on the implementation methods provided for each step, please refer to [link / reference]. Figure 2 and / or Figure 4 The implementation methods provided for each step are not elaborated here.

[0144] See Figure 14 , Figure 14 This is a schematic diagram of the image element detection device provided in an embodiment of this application. The image element detection device provided in an embodiment of this application includes: The prediction result determination module 141 is used to determine the initial image features of the image to be processed, input the initial image features of the image to be processed into the image element detection model, and obtain at least one target detection region and its target prediction result. Each of the above prediction results is used to indicate whether the target detection region predicted by the above image element detection model includes image elements of a preset type. The above image element detection model is trained based on the training method of the image element detection model provided in the embodiments of this application. The image element determination module 142 is used to determine whether the target detection region actually includes image elements of a preset type based on the target prediction result of each of the above target detection regions.

[0145] In some feasible implementations, for each of the above-mentioned target detection regions, the image element determination module 142 is configured to: Determine the target element features of the image elements corresponding to the target detection region; The similarity between the target element features and the preset element features of each preset type of image element in the feature set is determined. Each preset element feature and the target element feature is determined based on a pre-trained element processing network. Based on the maximum similarity and the target prediction result of the target detection region, it is determined whether the target detection region actually includes image elements of the preset type.

[0146] In some feasible implementations, for each of the above-mentioned target detection regions, the image element determination module 142 is configured to: If, based on the target prediction result of the target detection region, it is determined that the above image element detection model predicts that the target detection region does not include image elements of a preset type, and the above maximum similarity is less than the first similarity threshold, then it is determined that the target detection region does not actually include image elements of a preset type. If, based on the target prediction result of the target detection region, it is determined that the image element detection model predicts that the target detection region includes image elements of a preset type, and the maximum similarity is greater than the second similarity threshold, then it is determined that the target detection region actually includes image elements of the preset type, and the first similarity threshold is less than or equal to the second similarity threshold.

[0147] In specific implementation, the aforementioned image element detection device can perform the above-described actions through its built-in functional modules. Figure 10 For details on the implementation methods provided for each step, please refer to [link / reference]. Figure 10The implementation methods provided for each step will not be elaborated here.

[0148] See Figure 15 , Figure 15 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 15 As shown, the electronic device 1500 in this embodiment may include: a processor 1501, a network interface 1504, and a memory 1505. Furthermore, the electronic device 1500 may also include: an object interface 1503, and at least one communication bus 1502. The communication bus 1502 is used to implement communication between these components. The object interface 1503 may include a display screen and a keyboard; optionally, the object interface 1503 may also include a standard wired interface or a wireless interface. The network interface 1504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1505 may be a high-speed RAM or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory 1505 may also be at least one storage device located remotely from the aforementioned processor 1501. Figure 15 As shown, the memory 1505, which is a computer-readable storage medium, may include an operating system, a network communication module, an object interface module, and a device control application.

[0149] exist Figure 15 In the illustrated electronic device 1500, the network interface 1504 provides network communication functionality; the object interface 1503 is primarily used to provide an input interface for objects; and the processor 1501 can be used to call the device control application stored in the memory 1505 to achieve: The initial image features of each sample image are input into the initial model to obtain the target prediction result of at least one target detection region of the sample image. Each of the above target prediction results is used to indicate whether the target detection region corresponding to the model prediction includes image elements of a preset type. The total training loss is determined based on the target prediction results and the true classification results of each of the above target detection regions. The initial model is trained based on each of the above sample images until the total training loss meets the preset conditions. Training is stopped when the total training loss meets the preset conditions. The model at which training stops is determined as the image element detection model. Each of the above true classification results is used to indicate whether the corresponding target detection region actually includes image elements of the preset type. The processor 1501 determines the prediction results of each of the above targets based on the following method: Based on the initial image features of each sample image, at least one initial detection region of the sample image is iteratively adjusted a predetermined number of times to obtain the corresponding target detection region, wherein each adjustment is performed in the following manner: Based on the initial image features of the sample image, each first detection region is adjusted to obtain a second detection region and its corresponding initial prediction result. Each initial prediction result is used to indicate whether the second detection region corresponding to the model prediction includes image elements of a preset type. Each adjustment of the corresponding first detection region is the second detection region obtained in the previous adjustment. The first adjustment of the corresponding first detection region is the aforementioned initial detection region. The second detection region obtained in the last adjustment is the target detection region of the sample image. Based on the initial prediction results of the second detection region obtained from each adjustment, the target prediction results of each target detection region of the sample image are determined.

[0150] In some feasible implementations, for each of the above sample images, the processor 1501 is used to: Based on the initial image features of the sample image, each candidate detection region of the sample image is determined; For each of the above candidate detection regions, a first intersection-union ratio (IUU) is determined between the candidate detection region and the image region containing each image element in the sample image. If there is an IUU greater than a first threshold, or if all of the above IUU are less than a second threshold, then the candidate detection region is determined as an initial detection region of the sample image. The first threshold is greater than the second threshold.

[0151] In some feasible implementations, for each of the above sample images, the processor 1501 is used to: The location information of each first detection region of the sample image is mapped to the initial image features of the sample image to obtain the fused image features; Based on the aforementioned fused image features, the second detection region corresponding to each first detection region of the sample image and its corresponding initial prediction result are determined.

[0152] In some feasible implementations, for each of the above sample images, the processor 1501 is used to: For each initial detection region of the sample image, based on the initial prediction results of the second detection region corresponding to the initial detection region obtained in each adjustment except the last adjustment, the fusion prediction result corresponding to the initial detection region is determined. Based on the above fusion prediction result and the initial prediction result of the second detection region corresponding to the initial detection region obtained in the last adjustment, the target prediction result of the target detection region corresponding to the initial detection region is determined.

[0153] In some feasible implementations, the processor 1501 described above is used for: The first training loss is determined based on the target prediction results and the true classification results of each of the above target detection regions. Determine the image region where each image element is located in each of the above sample images, and determine the second training loss based on the image region where each image element is located in each of the above sample images and each of the above target detection regions; The total training loss is determined based on the first training loss and the second training loss mentioned above.

[0154] In some feasible implementations, the processor 1501 described above is used for: The first sub-loss is determined based on the target prediction results and the true classification results of each of the above-mentioned target detection regions. The first training loss is determined based on at least one of the second sub-loss or the third sub-loss, as well as the first sub-loss. When determining the second sub-loss, the processor 1501 is used to: Determine the true classification result of each second detection region obtained from each adjustment; Based on the initial prediction results and the true classification results of each second detection region obtained from each adjustment, the aforementioned second sub-loss is determined; When determining the aforementioned third sub-loss, the processor 1501 is used to: For each of the above initial detection regions, the initial prediction result of the initial detection region is determined based on the initial detection region and the image region where each image element in the corresponding sample image is located; The true classification result of each of the above initial detection regions is determined, and the third sub-loss is determined based on the initial prediction result and the true classification result of each of the above initial detection regions.

[0155] In some feasible implementations, for each of the aforementioned initial detection regions, the processor 1501 is used to: If the first intersection-union ratio of the initial detection region with the image region containing at least one image element in the corresponding sample image is greater than the first threshold, then the model predicts that the initial detection region includes image elements of a preset type. If the first intersection-union ratio of the initial detection region and the image regions where each image element in the corresponding sample image is located is less than the second threshold, then it is determined that the model predicts that the initial detection region does not include image elements of the preset type.

[0156] In some feasible implementations, the processor 1501 described above is used for: The fourth sub-loss is determined based on each of the above-mentioned target detection regions and the image region where each image element in each of the above-mentioned sample images is located. The second training loss is determined based on at least one of the fifth sub-loss or the sixth sub-loss and the above-mentioned fourth sub-loss. The fifth sub-loss is determined by each second detection region obtained from each adjustment except the last adjustment and the image region where each image element in each of the above sample images is located. The sixth sub-loss is determined by each of the above initial detection regions and the image region where each image element in each of the above sample images is located.

[0157] In some feasible implementations, the processor 1501 described above is used for: Determine the initial image features of the image to be processed, input the initial image features of the image to be processed into the image element detection model, and obtain at least one target detection region and its target prediction result; Each of the above prediction results is used to indicate whether the target detection region predicted by the above image element detection model includes image elements of a preset type. The above image element detection model is trained based on the training method of the image element detection model provided in the embodiments of this application. Based on the target prediction results of each of the above target detection regions, determine whether the target detection region actually includes image elements of a preset type.

[0158] In some feasible implementations, for each of the above-described target detection regions, the processor 1501 is used to: Determine the target element features of the image elements corresponding to the target detection region; The similarity between the target element features and the preset element features of each preset type of image element in the feature set is determined. Each preset element feature and the target element feature is determined based on a pre-trained element processing network. Based on the maximum similarity and the target prediction result of the target detection region, it is determined whether the target detection region actually includes image elements of the preset type.

[0159] In some feasible implementations, the processor 1501 described above is used for: If, based on the target prediction result of the target detection region, it is determined that the above image element detection model predicts that the target detection region does not include image elements of a preset type, and the above maximum similarity is less than the first similarity threshold, then it is determined that the target detection region does not actually include image elements of a preset type. If, based on the target prediction result of the target detection region, it is determined that the image element detection model predicts that the target detection region includes image elements of a preset type, and the maximum similarity is greater than the second similarity threshold, then it is determined that the target detection region actually includes image elements of the preset type, and the first similarity threshold is less than or equal to the second similarity threshold.

[0160] It should be understood that in some feasible implementations, the processor 1501 described above may be a central processing unit (CPU), which may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0161] In specific implementation, the aforementioned electronic device 1500 can perform the above-described actions through its built-in functional modules. Figure 2 , Figure 4 and / or Figure 10 For details on the implementation methods provided for each step, please refer to [link / reference]. Figure 2 , Figure 4 and / or Figure 10 The implementation methods provided for each step will not be elaborated here.

[0162] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement... Figure 2 , Figure 4 and / or Figure 10 The methods provided in each step are detailed in the implementation methods provided in the above steps, and will not be repeated here.

[0163] The aforementioned computer-readable storage medium can be an internal storage unit of the apparatus or electronic device provided in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device. The aforementioned computer-readable storage medium can also include magnetic disks, optical disks, read-only memory (ROM), or random access memory (RAM), etc. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the electronic device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0164] This application provides a computer program product, which includes a computer program that is executed by a processor. Figure 2 , Figure 4 and / or Figure 10 The methods provided for each step in the process.

[0165] The terms "first," "second," etc., used in the claims, description, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or electronic device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or electronic devices. References to "embodiment" herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The presentation of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The term "and / or" as used in this application's description and appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0166] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

[0167] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A training method for an image element detection model, characterized in that, The method includes: The initial image features of each sample image are input into the initial model to obtain the target prediction result of at least one target detection region of the sample image. Each target prediction result is used to indicate whether the target detection region corresponding to the model prediction includes image elements of a preset type. The total training loss is determined based on the target prediction results and the true classification results of each target detection region. The initial model is trained based on each sample image until the total training loss meets the preset conditions. Training is stopped when the total training loss meets the preset conditions. The model at the time of training stop is determined as the image element detection model. Each true classification result is used to indicate whether the corresponding target detection region actually includes image elements of a preset type. The initial model determines the prediction results for each target based on the following method: Based on the initial image features of each sample image, at least one initial detection region of the sample image is iteratively adjusted a predetermined number of times to obtain the corresponding target detection region, wherein each adjustment is performed in the following manner: Based on the initial image features of the sample image, each first detection region is adjusted to obtain a second detection region and its corresponding initial prediction result. Each initial prediction result is used to indicate whether the second detection region corresponding to the model prediction includes image elements of a preset type. Each adjustment of the corresponding first detection region is the second detection region obtained in the previous adjustment. The first adjustment of the corresponding first detection region is the initial detection region. The second detection region obtained in the last adjustment is the target detection region of the sample image. Based on the initial prediction results of the second detection region obtained from each adjustment, the target prediction results of each target detection region of the sample image are determined.

2. The method according to claim 1, characterized in that, For each of the sample images, at least one initial detection region of the sample image is determined based on the initial image features of the sample image, including: Based on the initial image features of the sample image, each candidate detection region of the sample image is determined; For each candidate detection region, a first intersection-union ratio (IUU) is determined between the candidate detection region and the image region containing each image element in the sample image. If there is an IUU greater than a first threshold, or if each of the first IUU is less than a second threshold, then the candidate detection region is determined as an initial detection region of the sample image, where the first threshold is greater than the second threshold.

3. The method according to claim 1, characterized in that, For each of the sample images, adjusting each first detection region based on the initial image features of that sample image to obtain a second detection region and its corresponding initial prediction result includes: The location information of each first detection region of the sample image is mapped to the initial image features of the sample image to obtain the fused image features; Based on the fused image features, the second detection region corresponding to each first detection region of the sample image and its corresponding initial prediction result are determined.

4. The method according to claim 1, characterized in that, For each of the sample images, determining the target prediction result for each target detection region of the sample image based on the initial prediction result of the second detection region obtained in each adjustment includes: For each initial detection region of the sample image, based on the initial prediction result of the second detection region corresponding to the initial detection region obtained in each adjustment except the last adjustment, the fusion prediction result corresponding to the initial detection region is determined. Based on the fusion prediction result and the initial prediction result of the second detection region corresponding to the initial detection region obtained in the last adjustment, the target prediction result of the target detection region corresponding to the initial detection region is determined.

5. The method according to claim 1, characterized in that, The determination of the total training loss based on the target prediction results and the true classification results of each target detection region includes: The first training loss is determined based on the target prediction results and the true classification results of each target detection region. Determine the image region where each image element is located in each of the sample images, and determine a second training loss based on the image region where each image element is located in each of the sample images and each of the target detection regions; The total training loss is determined based on the first training loss and the second training loss.

6. The method according to claim 5, characterized in that, The determination of the first training loss based on the target prediction results and the true classification results of each target detection region includes: A first sub-loss is determined based on the target prediction results and the true classification results of each target detection region; and a first training loss is determined based on at least one of the second sub-loss or the third sub-loss and the first sub-loss. Determining the second sub-loss includes: Determine the true classification result of each second detection region obtained from each adjustment; Based on the initial prediction results and the true classification results of each second detection region obtained from each adjustment, the second sub-loss is determined; Determining the third sub-loss includes: For each initial detection region, an initial prediction result is determined based on the initial detection region and the image region where each image element in the corresponding sample image is located. The true classification result of each initial detection region is determined, and the third sub-loss is determined based on the initial prediction result and the true classification result of each initial detection region.

7. The method according to claim 6, characterized in that, For each initial detection region, determining the initial prediction result for that initial detection region based on the initial detection region and the image region where each image element in the corresponding sample image is located includes: If the first intersection-union ratio of the initial detection region with the image region containing at least one image element in the corresponding sample image is greater than the first threshold, then the model predicts that the initial detection region includes image elements of a preset type. If the first intersection-union ratio of the initial detection region and the image regions where each image element in the corresponding sample image is located is less than the second threshold, then it is determined that the model predicts that the initial detection region does not include image elements of the preset type.

8. The method according to claim 5, characterized in that, The determination of the second training loss based on the image region where each image element in each of the sample images is located and each of the target detection regions includes: A fourth sub-loss is determined based on each of the target detection regions and the image region where each image element in each of the sample images is located; and a second training loss is determined based on at least one of the fifth sub-loss or the sixth sub-loss and the fourth sub-loss. The fifth sub-loss is determined by each second detection region obtained from each adjustment except the last adjustment and the image region where each image element in each sample image is located. The sixth sub-loss is determined by each initial detection region and the image region where each image element in each sample image is located.

9. An image element detection method, characterized in that, The method includes: Determine the initial image features of the image to be processed, input the initial image features of the image to be processed into the image element detection model, and obtain at least one target detection region and its target prediction result; Each of the prediction results is used to indicate whether the target detection region predicted by the image element detection model includes image elements of a preset type, and the image element detection model is trained based on the method described in any one of claims 1-8; Based on the target prediction result of each target detection region, it is determined whether the target detection region actually includes image elements of a preset type.

10. The method according to claim 9, characterized in that, For each target detection region, based on the target prediction result of that target detection region, it is determined whether the target detection region actually includes image elements of a preset type, including: Determine the target element features of the image elements corresponding to the target detection region; The similarity between the target element feature and the preset element feature of each preset type of image element in the feature set is determined, wherein each preset element feature and the target element feature are determined based on a pre-trained element processing network; Based on the maximum similarity and the target prediction result of the target detection region, it is determined whether the target detection region actually includes image elements of the preset type.

11. The method according to claim 10, characterized in that, The step of determining whether the target detection region actually includes image elements of a preset type based on the maximum similarity and the target prediction result of the target detection region includes: If, based on the target prediction result of the target detection region, it is determined that the image element detection model predicts that the target detection region does not include image elements of a preset type, and the maximum similarity is less than the first similarity threshold, then it is determined that the target detection region does not actually include image elements of the preset type. If, based on the target prediction result of the target detection region, it is determined that the image element detection model predicts that the target detection region includes image elements of a preset type, and the maximum similarity is greater than the second similarity threshold, then it is determined that the target detection region actually includes image elements of the preset type, and the first similarity threshold is less than or equal to the second similarity threshold.

12. A training device for an image element detection model, characterized in that, The device includes: The detection module is used to input the initial image features of each sample image into the initial model to obtain the target prediction result of at least one target detection region of the sample image. Each target prediction result is used to indicate whether the target detection region corresponding to the model prediction includes image elements of a preset type. The training module is used to determine the total training loss based on the target prediction results and the true classification results of each target detection region, train the initial model based on each sample image until the total training loss meets the preset conditions, and stop training when the model at the time of training stop is determined as the image element detection model. Each true classification result is used to indicate whether the corresponding target detection region actually includes image elements of a preset type. The detection module determines the prediction results for each target based on the following method: Based on the initial image features of each sample image, at least one initial detection region of the sample image is iteratively adjusted a predetermined number of times to obtain the corresponding target detection region, wherein each adjustment is performed in the following manner: Based on the initial image features of the sample image, each first detection region is adjusted to obtain a second detection region and its corresponding initial prediction result. Each initial prediction result is used to indicate whether the second detection region corresponding to the model prediction includes image elements of a preset type. Each adjustment of the corresponding first detection region is the second detection region obtained in the previous adjustment. The first adjustment of the corresponding first detection region is the initial detection region. The second detection region obtained in the last adjustment is the target detection region of the sample image. Based on the initial prediction results of the second detection region obtained from each adjustment, the target prediction results of each target detection region of the sample image are determined.

13. An image element detection device, characterized in that, The device includes: The prediction result determination module is used to determine the initial image features of the image to be processed, input the initial image features of the image to be processed into the image element detection model, and obtain at least one target detection region and its target prediction result; Each of the prediction results is used to indicate whether the target detection region predicted by the image element detection model includes image elements of a preset type, and the image element detection model is trained based on the method described in any one of claims 1-8; The image element determination module is used to determine whether the target detection region actually includes image elements of a preset type based on the target prediction result of each target detection region.

14. An electronic device, characterized in that, It includes a processor and a memory, which are interconnected; The memory is used to store computer programs; The processor is configured to perform the method as described in any one of claims 1 to 11 when the computer program is invoked.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method of any one of claims 1 to 11.

16. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images