Target detection model training method, target detection method, device and equipment

Abstract

The embodiment of the invention discloses a target detection model training method, a target detection method, devices and equipment. The target detection model comprises an image coding model, a feature coding model and a feature decoding model, and obtaining a target sample region, a target sample category and a sample description text corresponding to a to-be-detected sample object in a sample image; inputting the sample image into the image coding model, and outputting a region image feature corresponding to each sub-region in the sample image; inputting each region image feature and the sample description text feature into a feature coding model, and outputting a fusion feature corresponding to each sub-region; inputting the fusion feature corresponding to each sub-region into a feature decoding model, and outputting a target fusion feature corresponding to each sub-region; determining a prediction category probability corresponding to the to-be-detected sample object and a prediction sample area in the sample image according to each target fusion feature and the sample description text; and iterating the target detection model based on the prediction result and the target result.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a target detection model training method, a target detection method, a device and equipment. Background Art

[0002] With the development of artificial intelligence technology, artificial intelligence models can be used to detect some objects in images.

[0003] In the related art, a single artificial intelligence model is often used to achieve target detection of some objects in the image. However, a single artificial intelligence model has certain limitations. For example, the resolution of the image that can be input by a single artificial intelligence model is low, which will cause some tiny objects to be unable to be detected. In addition, the model performance of a single artificial intelligence model is limited, resulting in low detection accuracy for target detection of some objects in the image, or even failure to detect. Therefore, the artificial intelligence model in the related art has a technical problem that it cannot accurately detect objects in the image. Summary of the invention

[0004] The embodiments of the present application provide a target detection model training method, a target detection method, an apparatus and equipment. The trained target detection model can accurately detect the object to be detected and the area where the object to be detected exists in the image to be detected.

[0005] In order to achieve the above-mentioned object, an embodiment of the present application provides a method for training a target detection model, wherein the target detection model includes an image coding model, a feature coding model and a feature decoding model, and the method includes: Obtain the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image; Inputting the sample image into an image coding model, and outputting regional image features corresponding to each sub-region in the sample image; Determine the sample description text features corresponding to the sample description text, input each region image feature and the sample description text features into a feature encoding model, and output the fusion features corresponding to each sub-region; Inputting the fused features corresponding to each sub-region into the feature decoding model, and outputting the target fused features corresponding to each sub-region; Determine the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image according to each target fusion feature and the sample description text; A loss value corresponding to the target detection model is determined according to the predicted category probability, the target sample category, the predicted sample area and the target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated, and the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are returned until the loss value meets the preset loss condition, thereby obtaining a trained image coding model, a trained feature coding model and a trained feature decoding model.

[0006] In order to achieve the above-mentioned object, an embodiment of the present application provides, on the one hand, a target detection model training device, wherein the target detection model includes an image coding model, a feature coding model and a feature decoding model, and the device includes: An acquisition module is used to acquire a target sample area, a target sample category and a sample description text corresponding to a sample object to be detected in a sample image; An input module, used for inputting the sample image into an image coding model, and outputting regional image features corresponding to each sub-region in the sample image; A determination module, used to determine the sample description text features corresponding to the sample description text, and input each region image feature and the sample description text features into a feature coding model, and output a fusion feature corresponding to each sub-region; A fusion module, used for inputting the fusion features corresponding to each sub-region into the feature decoding model, and outputting the target fusion features corresponding to each sub-region; An output module, used to determine the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image according to each target fusion feature and the sample description text; An iterative module is used to determine the loss value corresponding to the target detection model according to the predicted category probability, the target sample category, the predicted sample area and the target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated to return to obtain the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image until the loss value meets the preset loss condition, thereby obtaining a trained image coding model, a trained feature coding model and a trained feature decoding model.

[0007] In some embodiments, the image coding model includes a first image coding sub-model and a second image coding sub-model; an input module, configured to: Inputting the sample image into a first image coding sub-model, and outputting first region image features corresponding to each sub-region in the sample image; Inputting the sample image into a second image coding sub-model, and outputting second region image features corresponding to each sub-region in the sample image; The first regional image feature corresponding to each sub-region is multiplied by a first weight, and the second regional image feature corresponding to each sub-region is multiplied by a second weight, and then added together to obtain the regional image feature corresponding to each sub-region.

[0008] In some embodiments, a determination module is used to: Performing text segmentation processing on the sample description text to obtain multiple sub-texts; Acquire a preset category name set, and generate an input text sequence by arranging the preset category name set and the multiple subtexts; The input text sequence is input into a pre-trained text encoding model, and the sample description text features corresponding to the sample description text are output.

[0009] In some embodiments, the output module is used to: Obtain the word vector corresponding to each word in the sample description text; Add the word vectors corresponding to each word and divide the sum by the number of words in the sample description text to obtain the sentence text feature; Perform an inner product operation on the sentence text feature and each target fusion feature to obtain a probability distribution corresponding to each sub-region belonging to each preset category; The predicted category probability corresponding to the sample object to be detected is determined according to the probability distribution, and the predicted sample area corresponding to the sample object to be detected is determined in each sub-area according to the predicted category probability.

[0010] In some embodiments, the iteration module includes a first loss submodule, a second loss submodule, and a third loss value module; A first loss submodule, used to determine a classification loss value corresponding to the target detection model according to the predicted category probability and the target sample category; A second loss submodule, used to determine a detection loss value corresponding to the target detection model according to the predicted sample area and the target sample area; A third loss submodule, configured to determine an intersection loss value corresponding to the target detection model according to an intersection-over-union ratio between the predicted sample region and the target sample region; The classification loss value, the detection loss value and the intersection loss value are added to obtain a loss value corresponding to the target detection model.

[0011] In some implementations, the first loss submodule is configured to: Normalizing the predicted category probabilities to obtain normalized predicted category probabilities; Determining a cross entropy loss value corresponding to the predicted category probability and the target sample category according to the normalized predicted category probability, the target sample category and a first preset hyperparameter; Determine a first weight value corresponding to the cross entropy loss value according to the target sample category, the normalized predicted category probability and a second preset hyperparameter; Determine a second weight value corresponding to the cross entropy loss value according to the first preset hyperparameter and the target sample category; Multiplying the first weight value, the second weight value, and the cross entropy loss value to obtain a target cross entropy loss value; A classification loss value corresponding to the target detection model is determined according to the total number of sub-regions in the sample image and the target cross entropy loss value.

[0012] In some implementations, the second loss submodule is configured to: Determine a plurality of first coordinates corresponding to the predicted sample area and a plurality of second coordinates corresponding to the target sample area; A difference calculation is performed on the first coordinates and the second coordinates associated with the multiple first coordinates and the multiple second coordinates to obtain a difference calculation result, and an absolute value corresponding to the difference calculation result is determined as a detection loss value corresponding to the target detection model.

[0013] In some implementations, the third loss submodule is configured to: Determine an intersection value and a union value between the prediction sample area and the target sample area; The intersection value is divided by the union value to obtain an intersection-and-union ratio, and an intersection loss value corresponding to the target detection model is determined according to the intersection-and-union ratio.

[0014] In order to achieve the above-mentioned object, an embodiment of the present application provides a target detection method on the one hand, which is applied to a trained target detection model, wherein the trained target detection model is a model trained by the target detection model training method provided by the embodiment of the present application, and the trained target detection model includes a trained image coding model, a trained feature coding model and a trained feature decoding model, and the method includes: Obtaining a task description text corresponding to an image to be detected and an object to be detected in the image to be detected; Inputting the image to be detected into the trained image coding model, and outputting the image features corresponding to each sub-region in the image to be detected; Determine the task description text feature corresponding to the task description text, input each image feature and the task description text feature into the trained feature encoding model, and output the first fusion feature corresponding to each sub-region; Inputting the first fused feature corresponding to each sub-region into the trained feature decoding model, and outputting the second fused feature corresponding to each sub-region; Determine the existence probability of the object to be detected in each sub-region according to each second fusion feature and the task description text; The sub-region corresponding to the highest existence probability in the image to be detected is determined as the existence region of the object to be detected.

[0015] In order to achieve the above-mentioned object, an embodiment of the present application provides, on the one hand, a target detection device, which is applied to a trained target detection model, wherein the trained target detection model is a model trained by the target detection model training method provided by the embodiment of the present application, and the trained target detection model includes a trained image coding model, a trained feature coding model and a trained feature decoding model, and the device includes: A first acquisition module is used to acquire the image to be detected and the task description text corresponding to the object to be detected in the image to be detected; A first input module, used for inputting the image to be detected into the trained image coding model, and outputting the image features corresponding to each sub-region in the image to be detected; A second input module is used to determine the task description text feature corresponding to the task description text, and input each image feature and the task description text feature into the trained feature encoding model, and output the first fusion feature corresponding to each sub-region; A third input module, used for inputting the first fused feature corresponding to each sub-region into the trained feature decoding model, and outputting the second fused feature corresponding to each sub-region; A first determination module, used to determine the existence probability of the object to be detected in each sub-region according to each second fusion feature and the task description text; The second determination module is used to determine the sub-region corresponding to the highest existence probability in the image to be detected as the existence region of the object to be detected.

[0016] In order to achieve the above-mentioned objectives, on the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the target detection model training method provided by the embodiment of the present application or the target detection method provided by the embodiment of the present application.

[0017] In order to achieve the above-mentioned objectives, on the one hand, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the target detection model training method provided in the embodiment of the present application or the target detection method provided in the embodiment of the present application is implemented.

[0018] In an embodiment of the present application, a target detection model includes an image coding model, a feature coding model, and a feature decoding model, by obtaining a target sample area, a target sample category, and a sample description text corresponding to a sample object to be detected in a sample image; inputting the sample image into the image coding model, and outputting a regional image feature corresponding to each sub-region in the sample image; determining a sample description text feature corresponding to the sample description text, and inputting each regional image feature and sample description text feature into the feature coding model, and outputting a fusion feature corresponding to each sub-region; inputting the fusion feature corresponding to each sub-region into the feature decoding model, and outputting a target fusion feature corresponding to each sub-region; determining a predicted category probability corresponding to the sample object to be detected and a predicted sample area in the sample image according to each target fusion feature and the sample description text; determining a loss value corresponding to the target detection model according to the predicted category probability, the target sample category, the predicted sample area, and the target sample area, when the loss value does not meet a preset loss condition, iterating the target detection model, and returning to obtain the target sample area, the target sample category, and the sample description text corresponding to the sample object to be detected in the sample image, until the loss value meets the preset loss condition, and obtaining a trained image coding model, a trained feature coding model, and a trained feature decoding model.

[0019] In this way, the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are first determined, and then the sample description text features of the sample description text are determined, and then the sample image is input into the image coding model, and the regional image features corresponding to each sub-region in the sample image are output, and each regional image feature and the sample description text feature are input into the feature coding model, and the fusion feature corresponding to each sub-region is output. The fusion feature contains the association relationship between the images and sample description texts of different sub-regions, and can associate and represent two different features in the same feature space, and then the fusion feature corresponding to each sub-region is input into the feature decoding model, and the target fusion feature corresponding to each sub-region is output. The target fusion feature can be understood as being generated to achieve specific downstream tasks, such as the fusion feature combined with the more specific keyword text generation of the sample object, and finally the target fusion feature and the sample description text can be used to determine the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image. Then, the loss value corresponding to the target detection model is determined according to the predicted category probability, target sample category, predicted sample area and target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated, and the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are returned until the loss value meets the preset loss condition, and the trained image coding model, trained feature coding model and trained feature decoding model are obtained. The trained target detection model can accurately detect the object to be detected and the area where the object to be detected exists in the image to be detected.

[0020] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a schematic diagram of a system framework corresponding to the target detection model training method and the target detection method provided in the embodiment of the present application; Figure 2 is a schematic diagram of an application scenario of a trained target detection model provided in an embodiment of the present application; Figure 3It is a flowchart of the target detection model training method provided in the embodiment of the present application; Figure 4 is a schematic diagram of the structure of the target detection model provided in the embodiment of the present application; Figure 5 is another flowchart of the target detection model training method provided in an embodiment of the present application; Figure 6 It is a flowchart of the target detection method provided in the embodiment of the present application; Figure 7 It is a structural schematic diagram of the target detection model training device provided in an embodiment of the present application; Figure 8 It is a structural schematic diagram of the target detection model training device provided in an embodiment of the present application; Fig. 9 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0024] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on image data and visual event data, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or jump to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0025] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution order. In addition, descriptions such as "first", "second" or "target" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0026] The target detection model training method and target detection method provided in the embodiments of the present application relate to the field of artificial intelligence technology. The target detection model training method and target detection method provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. Cloud servers; software can be applications that implement target detection model training methods and target detection methods, but are not limited to the above forms.

[0027] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0028] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations: Attention Mechanism: It is a commonly used technique in computer science and machine learning that can make the model more accurate and effective when processing sequence data. In traditional neural networks, the output of each neuron depends only on the output of all neurons in the previous layer, while in the attention mechanism, the output of each neuron not only depends on the output of all neurons in the previous layer, but can also be weighted according to different parts of the input data, that is, different weights are given to different parts. This allows the model to pay more attention to the key information in the input sequence, thereby improving the accuracy and efficiency of the model. In natural language processing, the attention mechanism is often used in tasks such as machine translation, speech recognition, and text summarization. Taking machine translation as an example, the attention mechanism can help the model better focus on the important parts of the source and target languages ​​during the translation process.

[0029] Normalized exponential function: Softmax function, in mathematics, especially probability theory and related fields, normalized exponential function. In fact, it is the logarithmic normalization of the gradient of a finite discrete probability distribution. The softmax function can convert a set of values ​​into a probability distribution. In this process, larger values ​​will be converted into larger probabilities, while smaller values ​​will be converted into smaller probabilities.

[0030] The above is the explanation of the relevant terms in this application. If other terms are involved in the subsequent content, they will be explained later.

[0031] First, let’s describe the technical problems existing in the relevant technologies: With the development of artificial intelligence technology, artificial intelligence models can be used to detect some objects in images.

[0032] In the related art, a single artificial intelligence model is often used to achieve target detection of some objects in the image. However, a single artificial intelligence model has certain limitations. For example, the resolution of the image that can be input by a single artificial intelligence model is low, which will cause some tiny objects to be unable to be detected. In addition, the model performance of a single artificial intelligence model is limited, resulting in low detection accuracy for target detection of some objects in the image, or even failure to detect. Therefore, the artificial intelligence model in the related art has a technical problem that it cannot accurately detect objects in the image.

[0033] In order to solve the above technical problems, an object detection model is provided in an embodiment of the present application. The object detection model includes an image coding model, a feature coding model and a feature decoding model. The target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are first determined, and then the sample description text features of the sample description text are determined. Then, the sample image is input into the image coding model, and the regional image features corresponding to each sub-region in the sample image are output. Each regional image feature and the sample description text feature are input into the feature coding model, and the fusion feature corresponding to each sub-region is output. The fusion feature includes the association relationship between the images and sample description texts of different sub-regions, and two different features can be associated and represented in the same feature space. Then, the fusion feature corresponding to each sub-region is input into the feature decoding model, and the target fusion feature corresponding to each sub-region is output. The target fusion feature can be understood as being generated to achieve a specific downstream task, such as the fusion feature combined with the more specific keyword text generation of the sample object. Finally, the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image can be determined through the target fusion feature and the sample description text. Then, the loss value corresponding to the target detection model is determined according to the predicted category probability, target sample category, predicted sample area and target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated, and the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are returned until the loss value meets the preset loss condition, and the trained image coding model, trained feature coding model and trained feature decoding model are obtained. The trained target detection model can accurately detect the object to be detected and the area where the object to be detected exists in the image to be detected.

[0034] The training process and application process of the target detection model will be explained later.

[0035] See also Figure 1 , Figure 1 1 is a schematic diagram of a system framework corresponding to the target detection model training method and the target detection method provided in the embodiment of the present application. The target detection model training method and the target detection method provided in the embodiment of the present application can be applied to the system framework.

[0036] The terminal 140 or the server 110 may be a device for executing a target detection model training method or a target detection method.

[0037] The terminal 140 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The embodiments of the present application can be applied to various scenarios, including but not limited to target detection, image classification, etc. In addition, it can be a single device or a collection of multiple devices. For example, multiple desktop computers are interconnected through a local area network, share a display, etc. to work together, and together constitute a terminal 140. The terminal 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0038] The server 110 refers to a computer system that can provide certain services to the terminal 140. Compared with the ordinary terminal 140, the server 110 has higher requirements in terms of stability, security, performance, etc. The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0039] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway can also provide filtering and security functions. The message sent by the terminal 140 to the server 110 must be sent to the corresponding server 110 through the gateway 120. The message sent by the server 110 to the terminal 140 must also be sent to the corresponding terminal 140 through the gateway 120.

[0040] In the embodiment of the present application, the target detection model trained by the target detection model training method can be applied to various scenarios such as target detection and image classification, and there is no limitation on the scenarios in which the trained target detection model in the present application is applied.

[0041] See also Figure 2 , Figure 2 It is a schematic diagram of an application scenario of the trained target detection model provided in an embodiment of the present application.

[0042] like Figure 2As shown, the trained target detection model can be applied to target detection scenarios. For example, if the image to be detected contains clouds and the sun, then the task description text corresponding to the image to be detected is obtained, and the task description text is: "Please find the sun in the image". Then the image to be detected and the task description text are input into the trained target detection model. The trained target detection model can find the target object "sun" in the image to be detected according to the task description text, and can generate a bounding box corresponding to the "sun". The area contained in the bounding box is the area where the "sun" exists.

[0043] The target detection model includes an image encoding model, a feature encoding model, and a feature decoding model. The training method corresponding to the trained target detection model can be specifically trained in the following ways: Obtain the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image; Input the sample image into the image coding model, and output the regional image features corresponding to each sub-region in the sample image; Determine the sample description text features corresponding to the sample description text, input each region image feature and the sample description text feature into the feature encoding model, and output the fusion feature corresponding to each sub-region; Input the fusion features corresponding to each sub-region into the feature decoding model, and output the target fusion features corresponding to each sub-region; Determine the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image according to each target fusion feature and the sample description text; The loss value corresponding to the target detection model is determined according to the predicted category probability, the target sample category, the predicted sample area and the target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated, and the target sample area, the target sample category and the sample description text corresponding to the sample object to be detected in the sample image are returned until the loss value meets the preset loss condition, thereby obtaining the trained image coding model, the trained feature coding model and the trained feature decoding model.

[0044] For a more detailed understanding of the target detection model training method provided in the embodiment of the present application, please refer to Figure 3 , Figure 3 : is a flow chart of the target detection model training method provided in the embodiment of the present application. The target detection model includes an image encoding model, a feature encoding model and a feature decoding model. The target detection model training method may include the following steps: Step 210, obtaining a target sample area, a target sample category and a sample description text corresponding to the sample object to be detected in the sample image; Step 220: input the sample image into the image coding model, and output the regional image features corresponding to each sub-region in the sample image; Step 230: determine the sample description text features corresponding to the sample description text, input each region image feature and the sample description text feature into a feature encoding model, and output a fusion feature corresponding to each sub-region; Step 240: input the fusion feature corresponding to each sub-region into the feature decoding model, and output the target fusion feature corresponding to each sub-region; Step 250: Determine the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image according to each target fusion feature and the sample description text; Step 260, determine the loss value corresponding to the target detection model according to the predicted category probability, the target sample category, the predicted sample area and the target sample area. When the loss value does not meet the preset loss condition, iterate the target detection model, and return to obtain the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image until the loss value meets the preset loss condition, and obtain the trained image coding model, the trained feature coding model and the trained feature decoding model.

[0045] Steps 210 to 260 will be described in detail below.

[0046] In step 210, a target sample area, a target sample category and a sample description text corresponding to a sample object to be detected in a sample image are obtained.

[0047] Among them, the training of the target detection model requires the use of some sample images, which are used as training data. The sample images can be images with a resolution of 1024x1024.

[0048] Taking a sample image as an example, the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image can be obtained. In the sample image, a sample object to be detected can be predetermined, and the sample object to be detected occupies an area in the sample image, namely the target sample area. The sample object to be detected corresponds to a target sample category, for example, if the sample object to be detected is a cat, the target sample category is a cat. The sample object to be detected corresponds to a sample description text, for example, the sample description text is a relevant description of the sample object to be detected in the sample image, for example, the description text is: "On a sunny lawn, there is a white cat lying in the sun."

[0049] It should be noted that the sample objects to be detected in the sample image, the target sample areas, target sample categories and sample description texts corresponding to the sample objects to be detected are all predetermined, and the subsequent target sample areas and target sample categories can be used as label data.

[0050] In step 220, the sample image is input into the image coding model, and the regional image features corresponding to each sub-region in the sample image are output.

[0051] Please also read Figure 4 , Figure 4 : is a schematic diagram of the structure of the target detection model provided in the embodiment of the present application. Among them, the target detection model includes a pre-trained text model, an image encoding model, a feature encoding model and a feature decoding model. Among them, the image encoding model, the feature encoding model and the feature decoding model are models that need to be iterated during the training process of the target detection model.

[0052] The sample image can be input into the image coding model to output the regional image features corresponding to each sub-region in the sample image. For example, if there are multiple objects in the sample image, the image coding model can encode the sub-region where each object is located, thereby obtaining the regional image features corresponding to each sub-region.

[0053] In some embodiments, the image coding model includes a first image coding sub-model and a second image coding sub-model; inputting a sample image into the image coding model, and outputting a regional image feature corresponding to each sub-region in the sample image, including: (1.1) Inputting the sample image into the first image coding sub-model, and outputting the first region image features corresponding to each sub-region in the sample image; (1.2) inputting the sample image into the second image coding sub-model, and outputting the second region image features corresponding to each sub-region in the sample image; (1.3) Multiply the first regional image feature corresponding to each sub-region by the first weight, multiply the second regional image feature corresponding to each sub-region by the second weight, and add the results to obtain the regional image feature corresponding to each sub-region.

[0054] Among them, the first image coding submodel and the second image coding submodel can be different types of models. For example, the first image coding submodel is the visual coding model ViT of the APE model, and the second image coding submodel is the visual coding model ViT-18B of the EVA-CLIP-18B model. The two models have different model structures and model parameters.

[0055] The sample image may be input into the first image coding sub-model, and the first region image features corresponding to each sub-region in the sample image may be output. For example, the sample image may be input into the visual coding model ViT model, and the first region image features Vape corresponding to each sub-region in the sample image may be output.

[0056] The sample image can be input into the second image coding sub-model, and the second region image features corresponding to each sub-region in the sample image can be output. For example, the sample image can be input into the visual coding model ViT-18B, and the first region image features Vvit18b corresponding to each sub-region in the sample image can be output.

[0057] Finally, the first regional image feature corresponding to each sub-region is multiplied by the first weight, and the second regional image feature corresponding to each sub-region is multiplied by the second weight, and then added together to obtain the regional image feature corresponding to each sub-region. For example, the calculation formula is as follows: .in is the regional image feature, 1-W is the first weight, and W is the second weight. The first weight and the second weight are learnable, that is, during the iteration of the image coding model, the first weight and the second weight will change with the iteration of the image coding model.

[0058] As can be seen from the above, in the present application, by combining the features outputted by the first image coding sub-model and the second image coding sub-model respectively, the regional image features corresponding to each sub-region generated in this way can more accurately express the image information in the image corresponding to each sub-region. Compared with the related art that only uses a single visual coding model to obtain image features of different regions in the image, the regional image features in the present application have richer feature information, which is beneficial to the detection of sample objects by the target detection model.

[0059] In step 230, the sample description text features corresponding to the sample description text are determined, and each region image feature and the sample description text feature are input into a feature encoding model, and a fusion feature corresponding to each sub-region is output.

[0060] The sample description text needs to be converted to generate sample description text features, which are vectors under a certain spatial dimension. Then, the image features of each region and the sample description text features are input into the feature encoding model, and the fused features corresponding to each sub-region are output.

[0061] In some implementations, determining a sample description text feature corresponding to the sample description text includes: (1.1) Perform text segmentation on the sample description text to obtain multiple sub-texts; (1.2) Obtaining a preset category name set, and arranging the preset category name set and a plurality of subtexts to generate an input text sequence; (1.3) Input the input text sequence into the pre-trained text encoding model and output the sample description text features corresponding to the sample description text.

[0062] The sample description text may be segmented to obtain multiple subtexts. For example, a sentence may be segmented according to a period, a question mark, or an exclamation mark, by first finding the positions of these punctuation marks in the text, and then extracting the content between the two punctuation marks as a subtext.

[0063] For example, first determine some key words, such as "people", "vehicles", and "animals" in the picture description, and then find the position of these keywords in the text, and treat the content before each keyword as a sub-text, and the content after the keyword until the next keyword as another sub-text.

[0064] Then, a preset category name set is obtained, and the input text sequence is generated by arranging the preset category name set and the multiple subtexts. The preset category name set includes multiple preset categories, and the preset categories can be categories corresponding to different people and objects in the picture. For example, the preset categories include "cat", "dog", "child", "man", "woman", "elderly", "tree", "flower", etc. These preset categories can all be collected in the preset category name set.

[0065] Each preset category in the preset category name set can be used as a category text, and these category texts and multiple sub-texts are arranged to generate an input text sequence. For example, each category text and each sub-text are connected in series to form a long text, and the long text is the input text sequence.

[0066] Finally, the input text sequence is input into the pre-trained text encoding model, and the sample description text features corresponding to the sample description text are output. For example, the pre-trained model can extract features from the input text sequence to generate sample description text features under a certain spatial dimension.

[0067] From the above, it can be seen that in the present application, by combining the preset category name set and the sub-text in the sample description text, an input text sequence containing multiple categories is generated. The advantage of this is that the generated sample description text features can contain more information about the preset categories, which is beneficial for the target detection model to learn the preset categories.

[0068] After obtaining the sample description text features, each region image feature and sample description text features can be input into the feature encoding model to output the fused features corresponding to each sub-region. The feature encoding model has a self-attention mechanism, which allows the feature encoding model to weigh the importance of each element in the sequence when processing sequence data. For the input sample description text features and each region image features, it can simultaneously pay attention to the information of these two different modalities.

[0069] Specifically, when the sample description text features and each region image features are input into the feature encoding model, they are first converted into vector representations suitable for processing by the feature encoding model. These vectors are concatenated or combined in some way as the input sequence of the feature encoding model. The feature encoding model then performs self-attention processing to obtain the corresponding fusion features of the two. This fusion feature contains the semantic association and complementary information between text and image, and can be better used for subsequent tasks such as classification and generation.

[0070] In step 240, the fused features corresponding to each sub-region are input into the feature decoding model, and the target fused features corresponding to each sub-region are output.

[0071] Among them, after the fusion features output by the encoding model are input into the feature decoding model, the target fusion features are more targeted features generated through the decoding process based on the fusion features. It is usually generated to achieve specific downstream tasks, such as in the target detection task, it can detect the image area of ​​the sample object to be detected in the sample image that matches the task description text.

[0072] In some implementations, an object query feature for querying a sample object to be detected may be obtained, and the object query feature may be a task description text feature generated based on the task description text. The task description text feature and the fusion feature corresponding to each sub-region are then input into a feature decoding model, and a target fusion feature corresponding to each sub-region is output.

[0073] The object query feature is like a "clue" for the feature decoding model to search for the sample object to be detected in the sample image. It will guide the feature decoding model to focus on different areas in the image by fusing features, and try to find areas that match the object query feature. For example, when faced with the prompt "find the cat in the image", the object query feature will prompt the feature decoding model to search for areas with cat features in the sample image, and by constantly adjusting the focus, it will eventually accurately identify the location and category of the cat.

[0074] The target fusion features can be understood as the final output object embeddings. These object embeddings contain rich information, covering the key elements such as the features, positions, and categories of various objects in the image. They are an important basis for subsequent regional image and text alignment and prediction. When processing complex scene images containing multiple objects, object embeddings can effectively encode the unique attributes of each object and help the target detection model distinguish different objects.

[0075] In step 250, the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image are determined according to each target fusion feature and the sample description text.

[0076] Among them, based on the generated target fusion features, the target detection model will further predict a series of key information, including scores, bounding boxes, and masks. The scores are used to evaluate the degree of match between each predicted object and the given prompt, reflecting the model's confidence in the accuracy of the prediction; the bounding box accurately locates the position of the target object in the image, and clarifies its specific range in the image in the form of coordinates; the mask is used to segment the contour of the target object in more detail. In instance segmentation and semantic segmentation tasks, the mask can clearly define the boundaries of the object, which is crucial for processing image scenes with complex foreground and background interweaving.

[0077] For example, when analyzing an image of a city street scene containing people, vehicles, and buildings, the object detection model uses the predicted scores to determine the likelihood of each detected object matching the corresponding prompt (such as "pedestrian", "car", "building"). The bounding box accurately defines the position of each object, and the mask accurately outlines their contours.

[0078] In some embodiments, determining the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image according to each target fusion feature and the sample description text includes: (1.1) Obtain the word vector corresponding to each word in the sample description text; (1.2) Add the word vectors corresponding to each word and divide it by the number of words in the sample description text to obtain the sentence text features; (1.3) Perform an inner product operation on the sentence text feature and each target fusion feature to obtain the probability distribution corresponding to each sub-region belonging to each preset category; (1.4) Determine the predicted category probability corresponding to the sample object to be detected according to the probability distribution, and determine the predicted sample area corresponding to the sample object to be detected in each sub-area according to the predicted category probability.

[0079] Among them, the word vector corresponding to each word in the sample description text can be obtained, and then the word vector corresponding to each word is added and divided by the number of words in the sample description text to obtain the sentence text feature. Among them, the sentence feature can be used to reduce the complexity of subsequent calculations and improve calculation efficiency. By decomposing the sample description text into independent concepts and aggregating them into sentence-level embeddings, the model can receive a large number of vocabulary and sentence prompts in a single forward pass. This method greatly improves the scale of model prompts, enabling it to cope with complex and diverse visual tasks, such as accurately identifying various targets in images containing many objects and scenes.

[0080] Then, the inner product operation is performed on the sentence text feature and each target fusion feature to obtain the probability distribution corresponding to each sub-region belonging to each preset category. This probability distribution can be understood as the score obtained by the target fusion feature mentioned above.

[0081] The predicted category probability corresponding to the sample object to be detected is determined according to the probability distribution, and the predicted sample area corresponding to the sample object to be detected is determined in each sub-area according to the predicted category probability. For example, the highest probability distribution (score) can be determined as the predicted category probability, and the sub-area corresponding to the highest probability distribution can be determined as the predicted sample area corresponding to the sample object to be detected, and it is considered that the sample object to be detected is most likely to exist in the predicted sample area. For example, if the sample object to be detected is "cat", then it is considered that the category corresponding to the predicted sample area is "cat", and "cat" exists in the predicted sample area. The predicted sample area can be represented by a bounding box, such as a bounding box in the sample image, and the bounding box has certain coordinate information.

[0082] In step 260, the loss value corresponding to the target detection model is determined based on the predicted category probability, the target sample category, the predicted sample area and the target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated, and the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are returned until the loss value meets the preset loss condition, thereby obtaining the trained image coding model, the trained feature coding model and the trained feature decoding model.

[0083] Among them, after obtaining the predicted category probability and predicted sample area corresponding to the sample object to be detected, the predicted value needs to be verified to determine the difference between the predicted value and the true value. For example, if the difference is a loss value, the loss value can be compared with the preset loss condition to determine whether the target detection model is trained.

[0084] When the loss value meets the preset loss condition, and after subsequent input of different sample images, the loss value corresponding to the target detection model still meets the preset loss condition, it means that the training of the target detection model is completed, and the trained target detection model is obtained. The trained target detection model includes a trained image encoding model, a trained feature encoding model and a trained feature decoding model.

[0085] When the loss value does not meet the preset loss condition, the target detection model is iterated, and the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are returned until the loss value meets the preset loss condition, and the trained image encoding model, the trained feature encoding model and the trained feature decoding model are obtained.

[0086] In some implementations, determining a loss value corresponding to the target detection model according to the predicted category probability, the target sample category, the predicted sample area, and the target sample area includes: (1.1) Determine the classification loss value corresponding to the target detection model based on the predicted category probability and the target sample category; (1.2) Determine the detection loss value corresponding to the target detection model based on the predicted sample area and the target sample area; (1.3) Determine the intersection loss value corresponding to the target detection model based on the intersection-over-union ratio between the predicted sample area and the target sample area; (1.4) Add the classification loss, detection loss, and intersection loss to get the loss value corresponding to the target detection model.

[0087] Among them, the classification loss value corresponding to the target detection model is first determined based on the predicted category probability and the target sample category. The classification loss value is mainly obtained based on FocalLoss and is used to distinguish the foreground area from the background area.

[0088] Then, the detection loss value corresponding to the target detection model is determined based on the predicted sample area and the target sample area. The detection loss value is mainly used to measure the difference between the predicted sample area and the target sample area, so as to determine the prediction accuracy of the target detection model for the predicted sample area.

[0089] Then, the intersection loss value corresponding to the target detection model is determined based on the intersection-over-union ratio between the predicted sample area and the target sample area. The intersection loss value is mainly based on the GIOU (Generalized Intersection over Union)-loss, which is mainly used to measure the similarity between two shapes (usually bounding boxes, such as in target detection tasks). The predicted sample area and the target sample area can be considered as the area composed of the bounding box.

[0090] Finally, the classification loss value, detection loss value, and intersection loss value are added to obtain the loss value corresponding to the target detection model.

[0091] The advantage of doing this is that the loss value corresponding to the target detection model can be measured from multiple different types of loss dimensions, which can increase the accuracy of determining the loss value of the target detection model and help improve the training efficiency of the target detection model.

[0092] In some implementations, determining a classification loss value corresponding to the target detection model according to the predicted category probability and the target sample category includes: (1.1.1) Normalize the predicted category probability to obtain the normalized predicted category probability; (1.1.2) determining a corresponding cross entropy loss value between the predicted category probability and the target sample category according to the normalized predicted category probability, the target sample category and the first preset hyperparameter; (1.1.3) determining a first weight value corresponding to the cross entropy loss value according to the target sample category, the normalized predicted category probability and the second preset hyperparameter; (1.1.4) Determining a second weight value corresponding to the cross entropy loss value according to the first preset hyperparameter and the target sample category; (1.1.5) Multiply the first weight value, the second weight value and the cross entropy loss value to obtain the target cross entropy loss value; (1.1.6) Determine the classification loss value corresponding to the target detection model based on the total number of sub-regions in the sample image and the target cross entropy loss value.

[0093] First, the predicted category probability is normalized to obtain the normalized predicted category probability. For example, the sigmoid function is used to normalize the predicted category probability. The specific calculation formula is as follows: Among them, pred_logits is the predicted category probability, which can be understood as the probability representation of each sub-region belonging to each preset category, and prob is the normalized predicted category probability.

[0094] Then, the corresponding cross entropy loss value between the predicted category probability and the target sample category is determined according to the normalized predicted category probability, the target sample category and the first preset hyperparameter. Specifically, the calculation formula is as follows: .in is the cross entropy loss value, targets is the representation corresponding to the target sample category, which can be obtained by one-hot encoding of the target sample category. is the first preset hyperparameter, and its corresponding value is 0.25.

[0095] Then, the first weight value corresponding to the cross entropy loss value is determined according to the target sample category, the normalized predicted category probability and the second preset hyperparameter. The first weight value can be specifically expressed as ,in is the second preset hyperparameter, and its value can be 2.0. The specific calculation method is: .

[0096] Then, the second weight value corresponding to the cross entropy loss value is determined according to the first preset hyperparameter and the target sample category. The specific calculation method is: ,in is the second weight value.

[0097] Then multiply the first weight value, the second weight value and the cross entropy loss value to get the target cross entropy loss value. The specific calculation method is: . loss_l is the target cross entropy loss value. The cross entropy loss can be adjusted by the first weight value and the second weight value to obtain a more accurate target cross entropy loss value.

[0098] Finally, the classification loss value corresponding to the target detection model is determined based on the total number of sub-regions in the sample image and the target cross entropy loss value. The specific calculation formula is: . Where num_boxes is the total number of sub-regions in the sample image, i is the i-th sub-region, is the target cross entropy loss value corresponding to the i-th sub-region, is the classification loss value.

[0099] The advantage of doing this is that the classification loss value can provide the necessary supervision signal for multi-task learning, thereby accurately guiding the training of the target detection model.

[0100] In some implementations, determining a detection loss value corresponding to the target detection model according to the predicted sample area and the target sample area includes: (1.2.1) Determine a plurality of first coordinates corresponding to the prediction sample region and a plurality of second coordinates corresponding to the target sample region; (1.2.2) Performing difference calculation on the first coordinates and the second coordinates associated with each other among the plurality of first coordinates and the plurality of second coordinates to obtain a difference calculation result, and determining an absolute value corresponding to the difference calculation result as a detection loss value corresponding to the target detection model.

[0101] The predicted sample region and the target sample region can be understood as regions respectively formed by two bounding boxes, and a plurality of first coordinates are obtained in the bounding box corresponding to the predicted sample region, and a plurality of second coordinates are obtained in the bounding box corresponding to the target sample region.

[0102] A difference calculation is performed on the associated first coordinates and second coordinates among the multiple first coordinates and the multiple second coordinates to obtain a difference calculation result, and an absolute value corresponding to the difference calculation result is determined as a detection loss value corresponding to the target detection model.

[0103] For example, if the bounding box is a rectangle that contains four vertices: the upper left, lower left, upper right, and lower right, the difference between the first coordinate and the second coordinate corresponding to each vertex can be calculated to obtain the difference calculation result. The absolute value corresponding to the difference calculation result is determined as the detection loss value corresponding to the target detection model. The specific calculation formula is: .in, To detect the loss value, is the target sample area, is the prediction sample area.

[0104] The advantage of doing this is that it is possible to determine the detection loss between the predicted sample area and the target sample area to measure the loss value of the target detection model, thereby improving the accuracy of the determined loss value.

[0105] In some implementations, determining an intersection loss value corresponding to the target detection model according to an intersection-over-union ratio between the predicted sample region and the target sample region includes: (1.3.1) Determine the intersection value and union value between the prediction sample area and the target sample area; (1.3.2) Divide the intersection value by the union value to get the intersection-union ratio, and determine the intersection loss value corresponding to the target detection model based on the intersection-union ratio.

[0106] Among them, the intersection value and the union value between the predicted sample area and the target sample area are determined. The size of the intersection area between the predicted sample area and the target sample area can be determined, thereby obtaining the intersection value, which is expressed as IntersectionArea. The size of the union area between the predicted sample area and the target sample area can be determined, thereby obtaining the intersection value, which is expressed as UnionArea.

[0107] Then divide the intersection value by the union value to get the intersection-union ratio, which is calculated as: ,in It is the intersection and comparison.

[0108] Then, the intersection loss value corresponding to the target detection model is determined according to the intersection-union ratio. The specific calculation method is: .in, is the intersection loss value.

[0109] Finally, the classification loss value, detection loss value and intersection loss value are added to obtain the loss value corresponding to the target detection model.

[0110] Among them, when the loss value meets the preset loss condition, and after subsequent input of different sample images, the loss value corresponding to the target detection model still meets the preset loss condition, it means that the training of the target detection model is completed, and the trained target detection model is obtained. The trained target detection model includes a trained image encoding model, a trained feature encoding model and a trained feature decoding model.

[0111] When the loss value does not meet the preset loss condition, the target detection model is iterated, such as adjusting the parameters of the target detection model, and returning to obtain the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image, until the loss value meets the preset loss condition, and the trained image coding model, the trained feature coding model and the trained feature decoding model are obtained. The trained image coding model includes the trained first image coding sub-model and the trained second image coding sub-model.

[0112] The trained target detection model can accurately detect the object to be detected and the area where the object to be detected exists in the image to be detected.

[0113] In an embodiment of the present application, a target detection model includes an image coding model, a feature coding model, and a feature decoding model, by obtaining a target sample area, a target sample category, and a sample description text corresponding to a sample object to be detected in a sample image; inputting the sample image into the image coding model, and outputting a regional image feature corresponding to each sub-region in the sample image; determining a sample description text feature corresponding to the sample description text, and inputting each regional image feature and sample description text feature into the feature coding model, and outputting a fusion feature corresponding to each sub-region; inputting the fusion feature corresponding to each sub-region into the feature decoding model, and outputting a target fusion feature corresponding to each sub-region; determining a predicted category probability corresponding to the sample object to be detected and a predicted sample area in the sample image according to each target fusion feature and the sample description text; determining a loss value corresponding to the target detection model according to the predicted category probability, the target sample category, the predicted sample area, and the target sample area, when the loss value does not meet a preset loss condition, iterating the target detection model, and returning to obtain the target sample area, the target sample category, and the sample description text corresponding to the sample object to be detected in the sample image, until the loss value meets the preset loss condition, and obtaining a trained image coding model, a trained feature coding model, and a trained feature decoding model.

[0114] In this way, the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are first determined, and then the sample description text features of the sample description text are determined, and then the sample image is input into the image coding model, and the regional image features corresponding to each sub-region in the sample image are output, and each regional image feature and the sample description text feature are input into the feature coding model, and the fusion feature corresponding to each sub-region is output. The fusion feature contains the association relationship between the images and sample description texts of different sub-regions, and can associate and represent two different features in the same feature space, and then the fusion feature corresponding to each sub-region is input into the feature decoding model, and the target fusion feature corresponding to each sub-region is output. The target fusion feature can be understood as being generated to achieve specific downstream tasks, such as the fusion feature combined with the more specific keyword text generation of the sample object, and finally the target fusion feature and the sample description text can be used to determine the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image. Then, the loss value corresponding to the target detection model is determined according to the predicted category probability, target sample category, predicted sample area and target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated, and the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are returned until the loss value meets the preset loss condition, and the trained image coding model, trained feature coding model and trained feature decoding model are obtained. The trained target detection model can accurately detect the object to be detected and the area where the object to be detected exists in the image to be detected.

[0115] See also Figure 5 , Figure 5 : is another flow chart of the target detection model training method provided by the embodiment of the present application. The target detection model training method may include: Step 301, obtaining a target sample area, a target sample category and a sample description text corresponding to a sample object to be detected in a sample image; Step 302: input the sample image into the first image coding sub-model, and output the first region image features corresponding to each sub-region in the sample image; Step 303: input the sample image into the second image coding sub-model, and output the second region image features corresponding to each sub-region in the sample image; Step 304: multiply the first regional image feature corresponding to each sub-region by the first weight, multiply the second regional image feature corresponding to each sub-region by the second weight, and add the results to obtain the regional image feature corresponding to each sub-region; Step 305: determine the sample description text features corresponding to the sample description text, input each region image feature and the sample description text feature into a feature encoding model, and output a fusion feature corresponding to each sub-region; Step 306: Input the fusion feature corresponding to each sub-region into the feature decoding model, and output the target fusion feature corresponding to each sub-region; Step 307: Determine the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image according to each target fusion feature and the sample description text; Step 308: Determine the classification loss value corresponding to the target detection model according to the predicted category probability and the target sample category; Step 309: Determine the detection loss value corresponding to the target detection model according to the predicted sample area and the target sample area; Step 310: Determine the intersection loss value corresponding to the target detection model according to the intersection-over-union ratio between the predicted sample area and the target sample area; Step 311, add the classification loss value, the detection loss value and the intersection loss value to obtain the loss value corresponding to the target detection model. When the loss value does not meet the preset loss condition, iterate the target detection model, and return to obtain the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image until the loss value meets the preset loss condition, and obtain the trained image coding model, the trained feature coding model and the trained feature decoding model.

[0116] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the above target detection model training method, which will not be repeated here.

[0117] See also Figure 6 , Figure 6 1 is a flow chart of a target detection method provided in an embodiment of the present application. The target detection method is applied to a trained target detection model, which is a model trained by a target detection model training method, and the trained model includes a trained image coding model, a trained feature coding model, and a trained feature decoding model.

[0118] Among them, the target detection method includes: Step 410: Obtain the image to be detected and the task description text corresponding to the object to be detected in the image to be detected; Step 420: input the image to be detected into the trained image coding model, and output the image features corresponding to each sub-region in the image to be detected; Step 430: determine the task description text features corresponding to the task description text, and input each image feature and the task description text feature into the trained feature encoding model, and output the first fusion feature corresponding to each sub-region; Step 440: input the first fused feature corresponding to each sub-region into the trained feature decoding model, and output the second fused feature corresponding to each sub-region; Step 450: Determine the existence probability of the object to be detected in each sub-region according to each second fusion feature and the task description text; Step 460: Determine the sub-region corresponding to the highest existence probability in the image to be detected as the existence region of the object to be detected.

[0119] Steps 410 to 460 will be described in detail below.

[0120] In step 410, the image to be detected and the task description text corresponding to the object to be detected in the image to be detected are obtained.

[0121] The image to be detected may be an image with a resolution of 1024x1024. The image to be detected may contain an object to be detected, for example, a person. A task description text may be obtained, and the task description text may be: "Please confirm whether a person exists".

[0122] In step 420, the image to be detected is input into the trained image coding model, and the image features corresponding to each sub-region in the image to be detected are output.

[0123] Then the image to be detected can be input into the trained image coding model to output the image features corresponding to each sub-region in the image to be detected. The trained image coding model includes a trained first image coding sub-model and a trained second image coding sub-model.

[0124] The image to be detected can be input into the trained first image coding sub-model and the trained second image coding sub-model respectively to obtain the target first image features corresponding to each sub-region and the target second image features corresponding to each sub-region output by the trained first image coding sub-model and the trained second image coding sub-model respectively.

[0125] Finally, the first result is obtained by multiplying the first image feature of the target by the first weight of the target. The second result is obtained by multiplying the second image feature of the target by the second weight. Finally, the first result and the second result are added to obtain the image feature corresponding to each sub-region in the detection image.

[0126] In step 430, the task description text features corresponding to the task description text are determined, and each image feature and the task description text feature are input into the trained feature encoding model, and the first fusion feature corresponding to each sub-region is output.

[0127] The task description text features corresponding to the task description text can be determined, and each image feature and the task description text feature are input into the trained feature encoding model to output the first fusion feature corresponding to each sub-region. The first fusion feature combines the image information in the image to be detected and the text information in the task description text to achieve feature enhancement for each sub-region.

[0128] In step 440, the first fused feature corresponding to each sub-region is input into the trained feature decoding model, and the second fused feature corresponding to each sub-region is output.

[0129] The first fused features corresponding to each sub-region are input into the trained feature decoding model, and the second fused features corresponding to each sub-region are output. The second fused features can be understood as the object embeddings of the final output. These object embeddings contain rich information, covering key elements such as the features, positions, and categories of various objects in the image, and are an important basis for subsequent regional image and text alignment and prediction. When processing complex scene images containing multiple objects, object embedding can effectively encode the unique attributes of each object and help the target detection model distinguish between different objects.

[0130] In step 450, the existence probability of the object to be detected in each sub-region is determined according to each second fused feature and the task description text.

[0131] Among them, the task description text can be input into the pre-trained text model to generate sentence-level features, and finally each second fusion feature and the sentence-level feature are aligned, and then the aligned sentence-level feature and each second fusion feature are multiplied to obtain the existence probability of the object to be detected in each sub-area.

[0132] In step 460, the sub-region corresponding to the highest existence probability in the image to be detected is determined as the existence region of the object to be detected.

[0133] The sub-region corresponding to the highest existence probability in the image to be detected can be determined as the existence region of the object to be detected. Finally, the object in the existence region is determined as the object finally detected.

[0134] From the above, it can be seen that the trained target detection model in the present application can accurately detect the object to be detected and the area where the object to be detected exists in the image to be detected.

[0135] See also Figure 7 , Figure 7 : is a schematic diagram of the structure of the target detection model training device provided in the embodiment of the present application. The target detection model training device can execute the above-mentioned target detection model training method.

[0136] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0137] The target detection model includes an image coding model, a feature coding model and a feature decoding model. The target detection model training device 500 includes: An acquisition module 510 is used to acquire a target sample area, a target sample category and a sample description text corresponding to a sample object to be detected in a sample image; An input module 520, used to input a sample image into an image coding model and output a regional image feature corresponding to each sub-region in the sample image; A determination module 530 is used to determine the sample description text features corresponding to the sample description text, and input each region image feature and the sample description text feature into a feature coding model, and output a fusion feature corresponding to each sub-region; A fusion module 540 is used to input the fusion feature corresponding to each sub-region into the feature decoding model, and output the target fusion feature corresponding to each sub-region; An output module 550 is used to determine the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image according to each target fusion feature and the sample description text; The iteration module 560 is used to determine the loss value corresponding to the target detection model according to the predicted category probability, the target sample category, the predicted sample area and the target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated to return the target sample area, the target sample category and the sample description text corresponding to the sample object to be detected in the sample image until the loss value meets the preset loss condition, thereby obtaining the trained image coding model, the trained feature coding model and the trained feature decoding model.

[0138] In some embodiments, the image coding model includes a first image coding sub-model and a second image coding sub-model; an input module 520 is used to: Inputting the sample image into the first image coding sub-model, and outputting the first region image features corresponding to each sub-region in the sample image; Inputting the sample image into the second image coding sub-model, and outputting the second region image features corresponding to each sub-region in the sample image; The first regional image feature corresponding to each sub-region is multiplied by the first weight, and the second regional image feature corresponding to each sub-region is multiplied by the second weight, and then added together to obtain the regional image feature corresponding to each sub-region.

[0139] In some implementations, the determination module 530 is configured to: Perform text segmentation processing on the sample description text to obtain multiple sub-texts; Obtain a preset category name set, and generate an input text sequence by arranging the preset category name set and a plurality of subtexts; The input text sequence is input into the pre-trained text encoding model, and the sample description text features corresponding to the sample description text are output.

[0140] In some embodiments, the output module 550 is used to: Get the word vector corresponding to each word in the sample description text; Add the word vectors corresponding to each word and divide it by the number of words in the sample description text to obtain the sentence text features; Perform inner product operation on the sentence text feature and each target fusion feature to obtain the probability distribution corresponding to each sub-region belonging to each preset category; The predicted category probability corresponding to the sample object to be detected is determined according to the probability distribution, and the predicted sample area corresponding to the sample object to be detected is determined in each sub-area according to the predicted category probability.

[0141] In some embodiments, the iteration module 560 includes a first loss submodule, a second loss submodule, and a third loss value module; The first loss submodule is used to determine the classification loss value corresponding to the target detection model according to the predicted category probability and the target sample category; The second loss submodule is used to determine the detection loss value corresponding to the target detection model according to the predicted sample area and the target sample area; The third loss submodule is used to determine the intersection loss value corresponding to the target detection model according to the intersection-over-union ratio between the predicted sample area and the target sample area; Add the classification loss value, detection loss value, and intersection loss value to get the loss value corresponding to the target detection model.

[0142] In some implementations, the first loss submodule is configured to: Normalize the predicted category probability to obtain the normalized predicted category probability; Determine a cross entropy loss value corresponding to the predicted category probability and the target sample category according to the normalized predicted category probability, the target sample category and the first preset hyperparameter; Determine a first weight value corresponding to the cross entropy loss value according to the target sample category, the normalized predicted category probability and the second preset hyperparameter; Determine a second weight value corresponding to the cross entropy loss value according to the first preset hyperparameter and the target sample category; Multiply the first weight value, the second weight value, and the cross entropy loss value to obtain a target cross entropy loss value; The classification loss value corresponding to the target detection model is determined based on the total number of sub-regions in the sample image and the target cross entropy loss value.

[0143] In some implementations, the second loss submodule is configured to: Determine a plurality of first coordinates corresponding to the prediction sample area and a plurality of second coordinates corresponding to the target sample area; A difference calculation is performed on the associated first coordinates and second coordinates among the multiple first coordinates and the multiple second coordinates to obtain a difference calculation result, and an absolute value corresponding to the difference calculation result is determined as a detection loss value corresponding to the target detection model.

[0144] In some implementations, the third loss submodule is configured to: Determine the intersection value and the union value between the prediction sample area and the target sample area; The intersection value is divided by the union value to obtain the intersection-union ratio, and the intersection loss value corresponding to the target detection model is determined based on the intersection-union ratio.

[0145] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the above target detection model training method, which will not be repeated here.

[0146] In an embodiment of the present application, the target detection model includes an image coding model, a feature coding model and a feature decoding model. The acquisition module 510 acquires the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image; the input module 520 inputs the sample image into the image coding model, and outputs the regional image features corresponding to each sub-region in the sample image; the determination module 530 determines the sample description text features corresponding to the sample description text, and inputs each regional image feature and sample description text feature into the feature coding model, and outputs the fused features corresponding to each sub-region; the fusion module 540 inputs the fused features corresponding to each sub-region into the feature decoding model, and outputs the fused features corresponding to each sub-region. The target fusion features corresponding to the domain; the output module 550 determines the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image according to each target fusion feature and the sample description text; the iteration module 560 determines the loss value corresponding to the target detection model according to the predicted category probability, the target sample category, the predicted sample area and the target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated, and the target sample area, the target sample category and the sample description text corresponding to the sample object to be detected in the sample image are returned until the loss value meets the preset loss condition, thereby obtaining the trained image coding model, the trained feature coding model and the trained feature decoding model.

[0147] In this way, the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are first determined, and then the sample description text features of the sample description text are determined, and then the sample image is input into the image coding model, and the regional image features corresponding to each sub-region in the sample image are output, and each regional image feature and the sample description text feature are input into the feature coding model, and the fusion feature corresponding to each sub-region is output. The fusion feature contains the association relationship between the images and sample description texts of different sub-regions, and can associate and represent two different features in the same feature space, and then the fusion feature corresponding to each sub-region is input into the feature decoding model, and the target fusion feature corresponding to each sub-region is output. The target fusion feature can be understood as being generated to achieve specific downstream tasks, such as the fusion feature combined with the more specific keyword text generation of the sample object, and finally the target fusion feature and the sample description text can be used to determine the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image. Then, the loss value corresponding to the target detection model is determined according to the predicted category probability, target sample category, predicted sample area and target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated, and the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are returned until the loss value meets the preset loss condition, and the trained image coding model, trained feature coding model and trained feature decoding model are obtained. The trained target detection model can accurately detect the object to be detected and the area where the object to be detected exists in the image to be detected.

[0148] See also Figure 8 , Figure 8 is a schematic diagram of the structure of the target detection device provided in the embodiment of the present application. The target detection device can execute the above target detection method.

[0149] The target detection device 600 is applied to a trained target detection model, where the trained target detection model is a model trained by the target detection model training method provided by an embodiment of the present application, and the trained model includes a trained image coding model, a trained feature coding model, and a trained feature decoding model. The device includes: A first acquisition module 610 is used to acquire the image to be detected and the task description text corresponding to the object to be detected in the image to be detected; A first input module 620, used to input the image to be detected into the trained image coding model, and output the image features corresponding to each sub-region in the image to be detected; The second input module 630 is used to determine the task description text feature corresponding to the task description text, and input each image feature and the task description text feature into the trained feature encoding model, and output the first fusion feature corresponding to each sub-region; A third input module 640 is used to input the first fused feature corresponding to each sub-region into the trained feature decoding model, and output the second fused feature corresponding to each sub-region; A first determination module 650 is used to determine the existence probability of the object to be detected in each sub-region according to each second fusion feature and the task description text; The second determination module 660 is used to determine the sub-region corresponding to the highest existence probability in the image to be detected as the existence region of the object to be detected.

[0150] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the above target detection method, which will not be repeated here.

[0151] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned target detection model training method or target detection method when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0152] See also Fig. 9 , Fig. 9 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes: The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the target detection model training method or target detection method of the embodiment of this application; Input / output interface 703, used to implement information input and output; Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); A bus 705 that transmits information between various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704); The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0153] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned target detection model training method or target detection method.

[0154] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0155] In an embodiment of the present application, a target detection model includes an image coding model, a feature coding model, and a feature decoding model, by obtaining a target sample area, a target sample category, and a sample description text corresponding to a sample object to be detected in a sample image; inputting the sample image into the image coding model, and outputting a regional image feature corresponding to each sub-region in the sample image; determining a sample description text feature corresponding to the sample description text, and inputting each regional image feature and sample description text feature into the feature coding model, and outputting a fusion feature corresponding to each sub-region; inputting the fusion feature corresponding to each sub-region into the feature decoding model, and outputting a target fusion feature corresponding to each sub-region; determining a predicted category probability corresponding to the sample object to be detected and a predicted sample area in the sample image according to each target fusion feature and the sample description text; determining a loss value corresponding to the target detection model according to the predicted category probability, the target sample category, the predicted sample area, and the target sample area, when the loss value does not meet a preset loss condition, iterating the target detection model, and returning to obtain the target sample area, the target sample category, and the sample description text corresponding to the sample object to be detected in the sample image, until the loss value meets the preset loss condition, and obtaining a trained image coding model, a trained feature coding model, and a trained feature decoding model.

[0156] In this way, the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are first determined, and then the sample description text features of the sample description text are determined, and then the sample image is input into the image coding model, and the regional image features corresponding to each sub-region in the sample image are output, and each regional image feature and the sample description text feature are input into the feature coding model, and the fusion feature corresponding to each sub-region is output. The fusion feature contains the association relationship between the images and sample description texts of different sub-regions, and can associate and represent two different features in the same feature space, and then the fusion feature corresponding to each sub-region is input into the feature decoding model, and the target fusion feature corresponding to each sub-region is output. The target fusion feature can be understood as being generated to achieve specific downstream tasks, such as the fusion feature combined with the more specific keyword text generation of the sample object, and finally the target fusion feature and the sample description text can be used to determine the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image. Then, the loss value corresponding to the target detection model is determined according to the predicted category probability, target sample category, predicted sample area and target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated, and the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are returned until the loss value meets the preset loss condition, and the trained image coding model, trained feature coding model and trained feature decoding model are obtained. The trained target detection model can accurately detect the object to be detected and the area where the object to be detected exists in the image to be detected.

[0157] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0158] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0159] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0161] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0162] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0163] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0164] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0166] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0167] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A target detection model training method, characterized in that: The target detection model includes an image encoding model, a feature encoding model and a feature decoding model, and the method includes: Obtain the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image; Inputting the sample image into an image coding model, and outputting regional image features corresponding to each sub-region in the sample image; Determine the sample description text features corresponding to the sample description text, input each region image feature and the sample description text features into a feature encoding model, and output the fusion features corresponding to each sub-region; Inputting the fused features corresponding to each sub-region into the feature decoding model, and outputting the target fused features corresponding to each sub-region; Determine the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image according to each target fusion feature and the sample description text; A loss value corresponding to the target detection model is determined according to the predicted category probability, the target sample category, the predicted sample area and the target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated, and the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image are returned until the loss value meets the preset loss condition, thereby obtaining a trained image coding model, a trained feature coding model and a trained feature decoding model.

2. The target detection model training method according to claim 1, characterized in that: The image coding model comprises a first image coding sub-model and a second image coding sub-model; The step of inputting the sample image into an image coding model and outputting a regional image feature corresponding to each sub-region in the sample image comprises: Inputting the sample image into a first image coding sub-model, and outputting first region image features corresponding to each sub-region in the sample image; Inputting the sample image into a second image coding sub-model, and outputting second region image features corresponding to each sub-region in the sample image; The first regional image feature corresponding to each sub-region is multiplied by a first weight, and the second regional image feature corresponding to each sub-region is multiplied by a second weight, and then added together to obtain the regional image feature corresponding to each sub-region.

3. The target detection model training method according to claim 1, characterized in that: The determining of the sample description text feature corresponding to the sample description text includes: Performing text segmentation processing on the sample description text to obtain multiple sub-texts; Acquire a preset category name set, and generate an input text sequence by arranging the preset category name set and the multiple subtexts; The input text sequence is input into a pre-trained text encoding model, and the sample description text features corresponding to the sample description text are output.

4. The target detection model training method according to claim 1, characterized in that: The step of determining the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image according to each target fusion feature and the sample description text includes: Obtain the word vector corresponding to each word in the sample description text; Add the word vectors corresponding to each word and divide the sum by the number of words in the sample description text to obtain the sentence text feature; Perform an inner product operation on the sentence text feature and each target fusion feature to obtain a probability distribution corresponding to each sub-region belonging to each preset category; The predicted category probability corresponding to the sample object to be detected is determined according to the probability distribution, and the predicted sample area corresponding to the sample object to be detected is determined in each sub-area according to the predicted category probability.

5. The target detection model training method according to claim 1, characterized in that: The determining the loss value corresponding to the target detection model according to the predicted category probability, the target sample category, the predicted sample area and the target sample area includes: Determine a classification loss value corresponding to the target detection model according to the predicted category probability and the target sample category; Determine a detection loss value corresponding to the target detection model according to the predicted sample area and the target sample area; Determining an intersection loss value corresponding to the target detection model according to an intersection-over-union ratio between the predicted sample area and the target sample area; The classification loss value, the detection loss value and the intersection loss value are added to obtain a loss value corresponding to the target detection model.

6. The target detection model training method according to claim 5, characterized in that: The step of determining the classification loss value corresponding to the target detection model according to the predicted category probability and the target sample category includes: Normalizing the predicted category probabilities to obtain normalized predicted category probabilities; Determining a cross entropy loss value corresponding to the predicted category probability and the target sample category according to the normalized predicted category probability, the target sample category and a first preset hyperparameter; Determine a first weight value corresponding to the cross entropy loss value according to the target sample category, the normalized predicted category probability and a second preset hyperparameter; Determine a second weight value corresponding to the cross entropy loss value according to the first preset hyperparameter and the target sample category; Multiplying the first weight value, the second weight value, and the cross entropy loss value to obtain a target cross entropy loss value; A classification loss value corresponding to the target detection model is determined according to the total number of sub-regions in the sample image and the target cross entropy loss value.

7. The target detection model training method according to claim 5, characterized in that: The determining, according to the predicted sample area and the target sample area, a detection loss value corresponding to the target detection model includes: Determine a plurality of first coordinates corresponding to the predicted sample area and a plurality of second coordinates corresponding to the target sample area; A difference calculation is performed on the first coordinates and the second coordinates associated with the multiple first coordinates and the multiple second coordinates to obtain a difference calculation result, and an absolute value corresponding to the difference calculation result is determined as a detection loss value corresponding to the target detection model.

8. The target detection model training method according to claim 5, characterized in that: The determining, according to the intersection-over-union ratio between the predicted sample region and the target sample region, an intersection loss value corresponding to the target detection model comprises: Determine an intersection value and a union value between the prediction sample area and the target sample area; The intersection value is divided by the union value to obtain an intersection-and-union ratio, and an intersection loss value corresponding to the target detection model is determined according to the intersection-and-union ratio.

9. A target detection model training device, characterized in that: The target detection model includes an image coding model, a feature coding model and a feature decoding model, and the device includes: An acquisition module is used to acquire a target sample area, a target sample category and a sample description text corresponding to a sample object to be detected in a sample image; An input module, used for inputting the sample image into an image coding model, and outputting regional image features corresponding to each sub-region in the sample image; A determination module, used to determine the sample description text features corresponding to the sample description text, and input each region image feature and the sample description text features into a feature coding model, and output a fusion feature corresponding to each sub-region; A fusion module, used for inputting the fusion features corresponding to each sub-region into the feature decoding model, and outputting the target fusion features corresponding to each sub-region; An output module, used to determine the predicted category probability corresponding to the sample object to be detected and the predicted sample area in the sample image according to each target fusion feature and the sample description text; An iterative module is used to determine the loss value corresponding to the target detection model according to the predicted category probability, the target sample category, the predicted sample area and the target sample area. When the loss value does not meet the preset loss condition, the target detection model is iterated to return to obtain the target sample area, target sample category and sample description text corresponding to the sample object to be detected in the sample image until the loss value meets the preset loss condition, thereby obtaining a trained image coding model, a trained feature coding model and a trained feature decoding model.

10. A target detection method, characterized in that: The trained target detection model is applied to a trained target detection model, wherein the trained target detection model is a model trained by the target detection model training method according to any one of claims 1 to 8, wherein the trained target detection model includes a trained image coding model, a trained feature coding model and a trained feature decoding model, and the method includes: Obtaining a task description text corresponding to an image to be detected and an object to be detected in the image to be detected; Inputting the image to be detected into the trained image coding model, and outputting the image features corresponding to each sub-region in the image to be detected; Determine the task description text feature corresponding to the task description text, input each image feature and the task description text feature into the trained feature encoding model, and output the first fusion feature corresponding to each sub-region; Inputting the first fused feature corresponding to each sub-region into the trained feature decoding model, and outputting the second fused feature corresponding to each sub-region; Determine the existence probability of the object to be detected in each sub-region according to each second fusion feature and the task description text; The sub-region corresponding to the highest existence probability in the image to be detected is determined as the existence region of the object to be detected.

11. A target detection device, characterized in that: The trained target detection model is applied to a trained target detection model, wherein the trained target detection model is a model trained by the target detection model training method according to any one of claims 1 to 8, wherein the trained target detection model includes a trained image coding model, a trained feature coding model and a trained feature decoding model, and the device includes: A first acquisition module is used to acquire the image to be detected and the task description text corresponding to the object to be detected in the image to be detected; A first input module, used for inputting the image to be detected into the trained image coding model, and outputting the image features corresponding to each sub-region in the image to be detected; A second input module is used to determine the task description text feature corresponding to the task description text, and input each image feature and the task description text feature into the trained feature encoding model, and output the first fusion feature corresponding to each sub-region; A third input module, used for inputting the first fused feature corresponding to each sub-region into the trained feature decoding model, and outputting the second fused feature corresponding to each sub-region; A first determination module, used to determine the existence probability of the object to be detected in each sub-region according to each second fusion feature and the task description text; The second determination module is used to determine the sub-region corresponding to the highest existence probability in the image to be detected as the existence region of the object to be detected.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute the target detection model training method described in any one of claims 1 to 8 or the target detection method described in claim 10.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the target detection model training method described in any one of claims 1 to 8 or the target detection method described in claim 10 is implemented.

Images