Target detection method and device and storage medium

Abstract

The invention discloses a target detection method and device and a storage medium, and the method comprises the steps: obtaining target prompt features corresponding to a plurality of pieces of target prompt information, mapping the target prompt features to a feature matrix, and obtaining a standardized prompt feature corresponding to the plurality of target prompt features in a unified manner; obtaining a coding result corresponding to the to-be-detected image to obtain an image feature, and decoding the image feature and the standardized prompt feature to obtain a weight parameter matched with the image feature; performing weighted calculation on the standardized prompt features by using the weight parameters, and taking a calculation result as target prompt features matched with the image features; and performing target detection on the to-be-detected image based on the image feature and the target prompt feature matched with the image feature to obtain a target detection result corresponding to the to-be-detected image. The generalization and stability of target detection can be improved through the target prompt information, infinite increase of prompt features can be avoided, storage resources and computing resources are saved, and the model training cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a target detection method, device and storage medium. Background Art

[0002] With the rapid construction of intelligent systems, the reliability, scalability and generalization of intelligent systems have become increasingly important, and the application direction of intelligent algorithms has gradually expanded from solving closed set problems to solving open set problems.

[0003] Among the technologies related to target detection, open-set tasks mostly adopt multimodal large model solutions. For example, the multimodal expressions of visual and language texts are aligned, and the generalization of language texts is used to improve the scalability and generalization of visual algorithms to achieve target detection.

[0004] However, as the number of targets is infinitely increased over time, and different forms of the same type of target may be judged as multiple targets, the target feature pool will expand infinitely and the model training time will increase. Summary of the invention

[0005] In order to solve the above technical problems, the present application at least provides a target detection method, device and storage medium.

[0006] The first aspect of the present application provides a target detection method, which includes: obtaining encoding results corresponding to multiple target prompt information to obtain multiple target prompt features, mapping the multiple target prompt features to a feature matrix, and obtaining standardized prompt features that uniformly correspond to the multiple target prompt features; wherein the target prompt information includes images and / or texts; obtaining the encoding result corresponding to the image to be detected to obtain image features, decoding the image features and the standardized prompt features to obtain weight parameters that match the image features; using the weight parameters to perform weighted calculation on the standardized prompt features, and using the calculation results as target prompt features that match the image features; performing target detection on the image to be detected based on the image features and the target prompt features that match the image features, and obtaining target detection results corresponding to the image to be detected.

[0007] In one embodiment, the image features and the standardized prompt features are decoded to obtain weight parameters that match the image features, including: decoding the image features and the standardized prompt features to obtain weight parameters of each matrix element in the standardized prompt features relative to each matrix element in the feature matrix to be solved; using the weight parameters to perform weighted calculation on the standardized prompt features, and using the calculation results as target prompt features that match the image features, including: based on the weight parameters of each matrix element in the standardized prompt features relative to the matrix elements in the feature matrix to be solved, weighted summation calculation is performed on the values ​​of each matrix element in the standardized prompt features, and the calculation results are used as the values ​​of the matrix elements in the feature matrix to be solved; combining the values ​​of each matrix element in the feature matrix to be solved to obtain the target prompt features that match the image features.

[0008] In one embodiment, a feature standardization network and a standardized decoder are pre-trained; multiple target prompt features are mapped into a feature matrix to obtain standardized prompt features that uniformly correspond to the multiple target prompt features, including: multiple target prompt features are input into the feature standardization network to obtain standardized prompt features output by the feature standardization network; image features and standardized prompt features are decoded to obtain weight parameters that match the image features, including: image features and standardized prompt features are input into the standardized decoder to obtain weight parameters that match the image features output by the standardized decoder.

[0009] In one embodiment, a plurality of prompt encoding networks are pre-trained; encoding results corresponding to a plurality of target prompt information are obtained to obtain a plurality of target prompt features, including: inputting the target prompt information into a plurality of prompt encoding networks respectively, obtaining initial encoding results output by each prompt encoding network respectively; fusing each initial encoding result to obtain the target prompt feature corresponding to the target prompt information.

[0010] In one embodiment, the target prompt information includes a prompt image, and the prompt encoding network includes a multimodal model and a text encoding model; the target prompt information is respectively input into a plurality of prompt encoding networks to obtain initial encoding results output by each prompt encoding network, including: inputting the prompt image into the multimodal model of the prompt encoding network to obtain a description text of the prompt image extracted by the multimodal model; inputting the description text into the text encoding model of the prompt encoding network to obtain text encoding features output by the text encoding model, and using the text encoding features as the initial encoding results output by the encoding network.

[0011] In one embodiment, the method also includes: obtaining the prompt accuracy corresponding to each target prompt information; if the prompt accuracy of the target prompt information is less than or equal to a preset accuracy threshold, the target prompt information is used as special target prompt information; obtaining the image and / or text associated with the target corresponding to the special target prompt information, and obtaining additional prompt content corresponding to the special target prompt information; performing target detection on the image to be detected based on image features and target prompt features matched with image features, and obtaining a target detection result corresponding to the image to be detected, including: performing target detection on the image to be detected based on image features, target prompt features matched with image features, and additional prompt content corresponding to the special target prompt information, and obtaining a target detection result corresponding to the image to be detected.

[0012] In one embodiment, target detection is performed on the image to be detected based on image features, target prompt features matched by the image features, and additional prompt content corresponding to the special target prompt information to obtain a target detection result corresponding to the image to be detected, including: obtaining an encoding result corresponding to the additional prompt content to obtain the additional prompt feature; calculating the matching degree between the image features corresponding to the image to be detected and the additional prompt feature; if the matching degree is greater than a preset matching degree threshold, target detection is performed on the image to be detected based on the image features, the target prompt features matched by the image features, and the additional prompt feature to obtain a target detection result corresponding to the image to be detected.

[0013] In one embodiment, a target detection decoder is pre-trained; target detection is performed on the image to be detected based on image features and target prompt features that match the image features to obtain a target detection result corresponding to the image to be detected, including: obtaining an encoding result corresponding to a question description text of the image to be detected to obtain question text features; cascading question text features, image features, and target prompt features that match the image features to obtain cascade features; and inputting the image features and the cascade features into the target detection decoder to obtain a target detection result output by the target detection decoder.

[0014] According to a second aspect of the present application, there is provided a target detection device, which includes: a feature standardization module, which is used to obtain encoding results corresponding to multiple target prompt information to obtain multiple target prompt features, map the multiple target prompt features into a feature matrix, and obtain standardized prompt features that uniformly correspond to the multiple target prompt features; wherein the target prompt information includes images and / or texts; a feature decoding module, which is used to obtain encoding results corresponding to the image to be detected to obtain image features, decode the image features and the standardized prompt features, and obtain weight parameters that match the image features; a feature calculation module, which is used to perform weighted calculation on the standardized prompt features using the weight parameters, and use the calculation results as target prompt features that match the image features; a target detection module, which is used to perform target detection on the image to be detected based on the image features and the target prompt features that match the image features, and obtain target detection results corresponding to the image to be detected.

[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned target detection method.

[0016] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned target detection method when the program instructions are executed by a processor.

[0017] The above scheme obtains multiple target prompt features by obtaining the encoding results corresponding to multiple target prompt information respectively, maps the multiple target prompt features into a feature matrix, and obtains standardized prompt features that uniformly correspond to the multiple target prompt features; obtains the encoding result corresponding to the image to be detected to obtain the image feature, decodes the image feature and the standardized prompt feature, and obtains the weight parameter that matches the image feature; uses the weight parameter to perform weighted calculation on the standardized prompt feature, and uses the calculation result as the target prompt feature that matches the image feature; performs target detection on the image to be detected based on the image feature and the target prompt feature that matches the image feature, and obtains the target detection result corresponding to the image to be detected, which can not only improve the generalization and stability of target detection through the target prompt information, but also avoid the unlimited increase of prompt features, save storage resources and computing resources, and reduce the model training cost.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0020] Figure 1is a schematic diagram of a solution implementation environment shown in an exemplary embodiment of the present application; Figure 2 is a flow chart of a target detection method shown in an exemplary embodiment of the present application; Figure 3 is a schematic diagram of feature standardization and feature decoding shown in an exemplary embodiment of the present application; Figure 4 is a schematic diagram of encoding target prompt information according to an exemplary embodiment of the present application; Figure 5 is a schematic diagram of target detection shown in an exemplary embodiment of the present application; Figure 6 is a block diagram of a target detection device shown in an exemplary embodiment of the present application; Figure 7 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application; Figure 8 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0022] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0023] The term "and / or" in this article is only an association information describing the associated objects, indicating that there may be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] The target detection method provided in the embodiments of the present application is described below.

[0025] Please refer to Figure 1 , Figure 1 1 is a schematic diagram of a solution implementation environment shown in an exemplary embodiment of the present application. The solution implementation environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 are connected to each other for communication.

[0026] The number of the terminal 110 may be one or more. The terminal 110 may be a camera, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, etc., but is not limited thereto.

[0027] Server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0028] In one example, the server 120 may perform target detection processing on the image to be detected obtained from the terminal 110 to obtain a target detection result. The server 120 may store the target detection result locally, transmit it back to the terminal 110, or transmit it to other terminals.

[0029] In one example, a client running a target application is installed in the terminal 110. For example, the target application may be an application that provides a target detection function. The target detection process is performed on the image to be detected based on the target application to obtain a target detection result. The server 120 may be a background server of the target application, which is used to provide background services for the client of the target application.

[0030] In the target detection method provided in the embodiment of the present application, the execution subject of each step can be the terminal 110, such as the client of the target application installed and running in the terminal 110, or the server 120, or the terminal 110 and the server 120 can interact and cooperate to execute, that is, part of the steps of the method are executed by the terminal 110 and the other steps are executed by the server 120.

[0031] See also Figure 2 , Figure 2 is a flowchart of a target detection method shown in an exemplary embodiment of the present application. The target detection method can be applied to Figure 1 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0032] like Figure 2 As shown, the target detection method at least includes steps S210 to S240, which are described in detail as follows: Step S210: Obtain encoding results corresponding to a plurality of target prompt information to obtain a plurality of target prompt features, map the plurality of target prompt features into a feature matrix, and obtain standardized prompt features corresponding to the plurality of target prompt features.

[0033] The target prompt information includes an image containing the target and / or text used to describe the target.

[0034] For example, collect the images to be detected corresponding to the target detection results with abnormalities in the target detection scene, such as collect the images to be detected corresponding to the target detection results with inaccurate recognition reported by users, use these images to be detected as abnormal recognition result images, and obtain target prompt information based on the abnormal recognition result images. For example, the user's feedback content may identify the abnormal cause of the recognition abnormality (such as misidentification or missed recognition) and the abnormal area, extract the abnormal area from the abnormal recognition result image, and obtain the target prompt information by combining the abnormal cause and the abnormal area; the image content of the abnormal area may also be converted into a text description to obtain the target prompt information; of course, the abnormal recognition result image may also be directly used as the target prompt information, and this application does not limit this.

[0035] The target prompt information is encoded to obtain the target prompt feature of the target prompt information.

[0036] In actual application scenarios, target prompt information will gradually increase. If the target prompt features are directly stored, it will lead to infinite expansion of the feature pool, infinitely increase the demand for device storage resources, increase the time consumption of subsequent target prompt information comparison encoding, and reduce target recognition efficiency.

[0037] In order to solve the above problems, the present application maps multiple target prompt features into a feature matrix to obtain standardized prompt features that uniformly correspond to the multiple target prompt features. The standardized prompt features can subsequently be decoded and estimated based on the image features corresponding to the image to be detected to obtain target prompt features that match the image to be detected.

[0038] It should be noted that the relevant parameters used in the mapping process need to be determined through iterative training based on training samples.

[0039] It should be noted that the target prompt feature can be a feature vector obtained by directly encoding the target prompt information; the target prompt feature can also be a standardized prompt feature implied by a historical time period. For example, new target prompt information is received at the current time, and the encoding result of the new target prompt information is obtained to obtain the new target prompt feature, and the previous standardized prompt feature is used as the historical standardized prompt feature. The new target prompt feature and the historical standardized prompt feature are encoded and mapped to a feature matrix of a fixed size to obtain the current latest standardized prompt feature and replace the previous standardized prompt feature.

[0040] Step S220: Obtain the encoding result corresponding to the image to be detected to obtain image features, decode the image features and the standardized prompt features, and obtain weight parameters matching the image features.

[0041] The image to be detected may be an image pre-stored in the server or an image uploaded by the terminal.

[0042] For example, an image acquisition device is connected to a server via a network, and the image acquisition device acquires image information of the environment in real time, and uploads the acquired environment image to the server as an image to be detected.

[0043] The image to be detected is encoded to obtain the image features corresponding to the image to be detected.

[0044] Then, the standardized prompt features are decoded according to the image features to obtain weight parameters matching the image features.

[0045] That is, the weight parameters are jointly determined by the network parameters used in the decoding process and the image features corresponding to the image to be detected.

[0046] Step S230: performing weighted calculation on the standardized prompt feature using the weight parameter, and using the calculation result as the target prompt feature that matches the image feature.

[0047] The standardized prompt features are weighted according to the weight parameters obtained by decoding and matching the image features, so that the standardized prompt features can be converted into target prompt features that match the image features.

[0048] Exemplarily, by decoding the image features and the standardized prompt features, the weight parameters of each matrix element in the standardized prompt features relative to each matrix element in the feature matrix to be solved are obtained. The feature matrix to be solved is a matrix of a preset size, which contains multiple matrix elements that need to be solved.

[0049] For example, the characteristic matrix to be solved is expressed as , the standardized hint feature is expressed as , n is the number of matrix elements. For the kth matrix element to be solved in the feature matrix to be solved , decode to get the i-th matrix element in the standardized prompt feature Relative matrix elements The weight parameter is expressed as , and so on, decoding obtains the relative matrix element of each matrix element in the standardized prompt feature The weight parameter of .

[0050] Then, based on the weight parameter of each matrix element in the standardized prompt feature relative to the matrix element in the feature matrix to be solved, the value of each matrix element in the standardized prompt feature is weighted and summed, and the calculation result is used as the value of the matrix element in the feature matrix to be solved; combined with the value of each matrix element in the feature matrix to be solved, the target prompt feature that matches the image feature is obtained.

[0051] Specifically, each matrix element in the standardized hint feature is relative to the matrix element The weight parameter is calculated to obtain the matrix element The relevant formula for the value of can be expressed as the following formula 1: (Formula 1) The value of each matrix element in the feature matrix to be solved is calculated based on Formula 1, so as to obtain the target prompt feature matching the image feature according to the calculated value of each matrix element in the feature matrix to be solved.

[0052] It should be noted that the above embodiments are only schematic illustrations. In actual application scenarios, the feature sizes of the feature matrix to be solved and the standardized prompt feature may be different, or the weight parameters of the matrix elements in the decoded standardized prompt feature relative to the matrix elements in the corresponding positions in the feature matrix to be solved may be obtained, so that each matrix element in the standardized prompt feature is directly weighted calculated through the weight parameter to obtain the target prompt feature that matches the image feature. The specific implementation method can be flexibly set according to the adopted network structure, the amount of target prompt information, the type of image to be detected, etc., and this application does not limit this.

[0053] For example, see Figure 3 , Figure 3 A schematic diagram of feature standardization and feature decoding is shown for an exemplary embodiment of the present application. Figure 3 As shown, a feature normalization network and a normalized decoder are pre-trained; multiple target cue features are input into the feature normalization network to obtain normalized cue features output by the feature normalization network; subsequently, image features and normalized cue features are input into the normalized decoder to obtain target cue features output by the normalized decoder that match the image features.

[0054] For example, the standardized decoder is represented as , the image feature is expressed as , the standardized hint feature is expressed as , then according to the image features and the standardized hint features, the estimated target hint features are decoded as .

[0055] Optionally, the feature normalization network and the normalized decoder are trained using training samples, and the training samples include sample target prompt information and sample input images. The model training loss can be obtained by calculating the difference between the actual output of the normalized decoder during the training process and the sample target prompt information, so as to adjust the parameters of the feature normalization network and the normalized decoder according to the model training loss until the loss converges.

[0056] Step S240: performing target detection on the image to be detected based on the image features and the target prompt features matched with the image features, and obtaining a target detection result corresponding to the image to be detected.

[0057] The target detection is realized by combining the image features and the target prompt features matched with the image features to obtain the target detection result.

[0058] The present application encodes multiple target prompt features into a standardized prompt feature, which can not only improve the generalization and stability of target detection through target prompt information, but also avoid the unlimited increase of prompt features, save storage resources and computing resources, and if new target prompt information is added subsequently, it is mainly necessary to train and fine-tune the network used to generate standardized prompt features and the network used to decode standardized prompt features, thereby reducing the model training cost.

[0059] Next, some embodiments of the present application are described in detail.

[0060] In some implementations, multiple prompt encoding networks are pre-trained; in step S210, encoding results corresponding to multiple target prompt information are obtained to obtain multiple target prompt features, including: Step S211: inputting the target prompt information into a plurality of prompt coding networks respectively, and obtaining initial coding results outputted by each prompt coding network respectively.

[0061] It should be noted that the network structure and / or network parameters of each hint coding network may be different.

[0062] Each cue encoding network performs feature encoding on the input target cue information respectively, and the encoding result output by each cue encoding network is used as the initial encoding result.

[0063] Step S212: fuse each initial encoding result to obtain the target prompt feature corresponding to the target prompt information.

[0064] For example, the target prompt information includes a prompt image, and the prompt encoding network includes a multimodal model and a text encoding model. Figure 4 , Figure 4 FIG. 1 is a schematic diagram showing an exemplary embodiment of the present application for encoding target prompt information, such as Figure 4 As shown, the prompt image Q is input into the multimodal model of M prompt encoding networks respectively, and the description text of the prompt image extracted by the multimodal model is obtained; the description text is input into the text encoding model of M prompt encoding networks respectively, and the text encoding features output by the text encoding model are obtained. The M text encoding features finally obtained are used as the initial encoding results output by the encoding network, which are expressed as the initial encoding result set .

[0065] Then, each initial encoding result of the initial encoding result set is fused.

[0066] For example, each initial encoding result can be directly added to obtain the target prompt feature corresponding to the prompt image Q as Alternatively, each initial coding result may be cascaded and the cascaded result may be subjected to dimensionality reduction processing, such as reducing the feature dimension of the cascaded result through a linear layer to normalize the M initial coding results into a feature matrix of a fixed length, thereby obtaining the target prompt feature corresponding to the prompt image in the end.

[0067] Among them, the multimodal model and text encoding model are open source models or models that have been fine-tuned and trained for specific tasks.

[0068] The above embodiment encodes and fuses the target prompt information respectively through multiple prompt encoding networks, which can improve the generalization of the target prompt features finally extracted and improve the accuracy of subsequent target detection.

[0069] Optionally, the target prompt information includes misidentified target information and missed target information, the misidentified target information and missed target information are divided to obtain a misidentified target information set and a missed target information set respectively, and each misidentified target information in the misidentified target information set is uniformly encoded to obtain a positive sample standardized prompt feature , uniformly encode each missed target information in the missed target information set to obtain the negative sample standardized prompt feature , combined with the image features corresponding to the image to be detected and the normalized prompt features of the positive sample Decode to obtain the positive sample target prompt features that match the image features, combined with the image features corresponding to the image to be detected and the negative sample normalized prompt features Decoding is performed to obtain negative sample target prompt features that match the image features, and target detection is performed based on the image features and the image features and the corresponding positive sample target prompt features and negative sample target prompt features to obtain a target detection result corresponding to the image to be detected.

[0070] In order to distinguish the positive sample target hint features from the negative sample target hint features during the target detection process, a label position may be added at the front or the back of the positive sample target hint features and the negative sample target hint features.

[0071] The accuracy of target recognition is improved by constructing standardized prompt features for misidentified target information and missed target information respectively.

[0072] In some embodiments, the method also includes: obtaining the prompt accuracy corresponding to each target prompt information; if the prompt accuracy of the target prompt information is less than or equal to a preset accuracy threshold, then using the target prompt information as special target prompt information; obtaining the image and / or text associated with the target corresponding to the special target prompt information, and obtaining additional prompt content corresponding to the special target prompt information.

[0073] Exemplarily, it may be possible to collect evaluation feedback from users on target detection results within a preset time period, analyze the evaluation feedback, and obtain the accuracy rate corresponding to the target prompt information used for each target detection result within the preset time period.

[0074] If the prompt accuracy of the target prompt information is less than or equal to the preset accuracy threshold, it means that it is still difficult to accurately identify the target corresponding to the target prompt information relying solely on the target prompt information. Therefore, the target prompt information is used as special target prompt information, and images and / or texts associated with the target corresponding to the special target prompt information are obtained, such as images containing the target or texts describing the target, and these information are used as additional prompt content corresponding to the special target prompt information.

[0075] The additional prompt content may be directly stored, or the additional information may be encoded and the encoding result may be stored for subsequent comparison and calculation.

[0076] Furthermore, target detection is performed on the image to be detected based on the image features, target prompt features matched by the image features, and additional prompt content corresponding to the special target prompt information, to obtain a target detection result corresponding to the image to be detected.

[0077] Specifically, an encoding result corresponding to the additional prompt content is obtained to obtain an additional prompt feature; the matching degree between the image feature corresponding to the image to be detected and the additional prompt feature is calculated; if the matching degree is greater than a preset matching degree threshold, target detection is performed on the image to be detected based on the image feature, the target prompt feature matched by the image feature and the additional prompt feature to obtain a target detection result corresponding to the image to be detected.

[0078] Exemplarily, the cosine distance, Euclidean distance, etc. between feature vectors may be calculated to obtain the matching degree between the image feature corresponding to the image to be detected and the additional prompt feature.

[0079] If the matching degree is greater than a preset matching degree threshold, target detection is performed on the image to be detected based on the comprehensive image features, target prompt features matched by the image features, and additional prompt features matched, to obtain a target detection result corresponding to the image to be detected.

[0080] The preset matching degree threshold may be pre-set or flexibly calculated. For example, the prompt accuracy of the target prompt information corresponding to each additional prompt feature is obtained, and based on the prompt accuracy of the target prompt information corresponding to each additional prompt feature, the preset matching degree threshold corresponding to each additional prompt feature is set respectively, and the prompt accuracy and the preset matching degree threshold are positively correlated, that is, the higher the prompt accuracy, the higher the preset matching degree threshold, and the lower the prompt accuracy, the lower the preset matching degree threshold, so as to improve the accuracy of target detection.

[0081] In some embodiments, in addition to selecting matching additional prompt features for target detection, target detection can also be performed directly based on all additional prompt features, that is, target detection is performed in combination with image features, target prompt features matched by image features, and all additional prompt features. This application does not limit this.

[0082] In some real-time methods, a target detection decoder is pre-trained; target detection is performed on the image to be detected based on image features and target prompt features matched with the image features to obtain a target detection result corresponding to the image to be detected, including: obtaining an encoding result corresponding to a question description text of the image to be detected to obtain question text features; cascading question text features, image features, and target prompt features matched with the image features to obtain cascade features; inputting the image features and the cascade features into the target detection decoder to obtain a target detection result output by the target detection decoder.

[0083] Specifically, see Figure 5 , Figure 5 A schematic diagram of target detection is shown as an exemplary embodiment of the present application. Figure 5As shown in the figure, the target detection model contains an image encoder, a target detection decoder, and a prompt encoder. It receives a target detection request, which carries an image to be detected and a problem description text. The image to be detected is input into the image encoder to obtain the image features corresponding to the image to be detected. Then, the image features corresponding to the image to be detected are obtained using Figure 3 The network structure shown in the figure obtains the target prompt feature that matches the image feature, inputs the question description text, additional prompt content (image or text) and the target prompt feature into the prompt encoder, obtains the question text feature, additional prompt feature and the encoded target prompt feature, cascades the image feature with the question text feature, additional prompt feature and the encoded target prompt feature to obtain the cascade feature; inputs the image feature and the cascade feature into the target detection decoder, obtains the target detection result output by the target detection decoder, and the target detection result is used to mark the position and type of each target in the image to be detected.

[0084] Among them, the above-mentioned network structure can be implemented based on convolutional neural networks (CNN), recurrent neural networks (RNN), Transformer networks, etc., and this application does not limit this.

[0085] The target detection method provided by the present application obtains multiple target prompt features by obtaining encoding results corresponding to multiple target prompt information respectively, maps the multiple target prompt features into a feature matrix, and obtains standardized prompt features that uniformly correspond to the multiple target prompt features; obtains the encoding result corresponding to the image to be detected to obtain image features, decodes the image features and the standardized prompt features, and obtains weight parameters that match the image features; uses the weight parameters to perform weighted calculation on the standardized prompt features, and uses the calculation results as target prompt features that match the image features; performs target detection on the image to be detected based on the image features and the target prompt features that match the image features, and obtains the target detection result corresponding to the image to be detected, which can not only improve the generalization and stability of target detection through target prompt information, but also avoid the unlimited increase of prompt features, save storage resources and computing resources, and reduce model training costs.

[0086] Figure 6 FIG. 1 is a block diagram of a target detection device shown in an exemplary embodiment of the present application. Figure 6 As shown, the exemplary target detection device 600 includes: The feature standardization module 610 is used to obtain the encoding results corresponding to the multiple target prompt information to obtain multiple target prompt features, map the multiple target prompt features to a feature matrix, and obtain standardized prompt features corresponding to the multiple target prompt features; wherein the target prompt information includes an image and / or text; The feature decoding module 620 is used to obtain the encoding result corresponding to the image to be detected to obtain the image feature, decode the image feature and the standardized prompt feature, and obtain the weight parameter matching the image feature; A feature calculation module 630 is used to perform weighted calculation on the standardized prompt feature using the weight parameter, and use the calculation result as the target prompt feature that matches the image feature; The target detection module 640 is used to perform target detection on the image to be detected based on the image features and the target prompt features matched with the image features, and obtain the target detection result corresponding to the image to be detected.

[0087] It should be noted that the target detection device provided in the above embodiment and the target detection method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In practical applications, the target detection device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0088] See also Figure 7 , Figure 7 7 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 700 includes a memory 701 and a processor 702, and the processor 702 is used to execute program instructions stored in the memory 701 to implement the steps in any of the above target detection method embodiments. In a specific implementation scenario, the electronic device 700 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 700 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0089] Specifically, the processor 702 is used to control itself and the memory 701 to implement the steps in any of the above-mentioned target detection method embodiments. The processor 702 may also be referred to as a central processing unit (CPU). The processor 702 may be an integrated circuit chip having the ability to process signals. The processor 702 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 702 may be implemented by an integrated circuit chip.

[0090] See also Figure 8 , Figure 8 The computer-readable storage medium 800 stores program instructions 810 that can be executed by a processor, and the program instructions 810 are used to implement the steps in any of the above target detection method embodiments.

[0091] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0092] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0093] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0094] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

Claims

1. A target detection method, characterized in that: The method comprises: Obtaining encoding results corresponding to a plurality of target prompt information respectively to obtain a plurality of target prompt features, mapping the plurality of target prompt features to a feature matrix, and obtaining standardized prompt features uniformly corresponding to the plurality of target prompt features; wherein the target prompt information includes an image containing a target and / or text for describing the target; Obtaining the encoding result corresponding to the image to be detected to obtain image features, decoding the image features and the standardized prompt features to obtain weight parameters matching the image features; Using the weight parameter to perform weighted calculation on the standardized prompt feature, and using the calculation result as the target prompt feature that matches the image feature; Based on the image features and the target prompt features that match the image features, target detection is performed on the image to be detected to obtain a target detection result corresponding to the image to be detected.

2. The method according to claim 1, characterized in that The decoding process of the image feature and the standardized prompt feature to obtain a weight parameter matching the image feature includes: Decoding the image features and the standardized prompt features to obtain weight parameters of each matrix element in the standardized prompt features relative to each matrix element in the feature matrix to be solved; The step of performing weighted calculation on the standardized prompt feature by using the weight parameter and taking the calculation result as the target prompt feature matched with the image feature comprises: Based on the weight parameter of each matrix element in the standardized prompt feature relative to the matrix element in the feature matrix to be solved, a weighted sum calculation is performed on the value of each matrix element in the standardized prompt feature, and the calculation result is used as the value of the matrix element in the feature matrix to be solved; The value of each matrix element in the feature matrix to be solved is combined to obtain a target prompt feature that matches the image feature.

3. The method according to claim 1, characterized in that A feature normalization network and a normalization decoder are pre-trained; the mapping of the plurality of target prompt features to a feature matrix to obtain normalized prompt features uniformly corresponding to the plurality of target prompt features comprises: Inputting the plurality of target prompt features into the feature normalization network to obtain normalized prompt features output by the feature normalization network; The decoding process of the image feature and the standardized prompt feature to obtain a weight parameter matching the image feature includes: The image features and the standardized prompt features are input into the standardized decoder to obtain the weight parameters matched with the image features output by the standardized decoder.

4. The method according to claim 1, characterized in that: Pre-trained with multiple cue encoding networks; The step of obtaining the encoding results corresponding to the plurality of target prompt information to obtain a plurality of target prompt features includes: Inputting the target prompt information into a plurality of prompt encoding networks respectively, and obtaining initial encoding results output by each prompt encoding network respectively; Each initial encoding result is fused to obtain a target prompt feature corresponding to the target prompt information.

5. The method according to claim 4, characterized in that The target prompt information includes a prompt image, and the prompt encoding network includes a multimodal model and a text encoding model; the target prompt information is input into a plurality of prompt encoding networks respectively to obtain initial encoding results output by each prompt encoding network respectively, including: Inputting the prompt image into the multimodal model of the prompt encoding network to obtain the description text of the prompt image extracted by the multimodal model; The description text is input into the text encoding model of the prompt encoding network to obtain the text encoding features output by the text encoding model, and the text encoding features are used as the initial encoding results output by the encoding network.

6. The method according to claim 1, characterized in that The method further comprises: Obtain the prompt accuracy corresponding to each target prompt information; If the prompt accuracy of the target prompt information is less than or equal to a preset accuracy threshold, the target prompt information is used as special target prompt information; Acquire an image and / or text associated with the target corresponding to the special target prompt information, and obtain additional prompt content corresponding to the special target prompt information; The performing target detection on the image to be detected based on the image feature and the target prompt feature matched with the image feature to obtain the target detection result corresponding to the image to be detected includes: Based on the image features, the target prompt features matched by the image features and the additional prompt content corresponding to the special target prompt information, target detection is performed on the image to be detected to obtain a target detection result corresponding to the image to be detected.

7. The method according to claim 6, characterized in that The performing target detection on the image to be detected based on the image features, the target prompt features matched by the image features, and the additional prompt content corresponding to the special target prompt information to obtain the target detection result corresponding to the image to be detected includes: Obtaining the encoding result corresponding to the additional prompt content to obtain the additional prompt feature; Calculating the matching degree between the image feature corresponding to the image to be detected and the additional prompt feature; If the matching degree is greater than a preset matching degree threshold, target detection is performed on the image to be detected based on the image features, the target prompt features matched by the image features, and the additional prompt features to obtain a target detection result corresponding to the image to be detected.

8. The method according to claim 1, characterized in that A target detection decoder is pre-trained; the target detection is performed on the image to be detected based on the image features and the target prompt features matched with the image features to obtain the target detection result corresponding to the image to be detected, including: Obtaining the encoding result corresponding to the question description text of the image to be detected, and obtaining the question text feature; Cascading the question text feature, the image feature, and the target prompt feature matched with the image feature to obtain a cascade feature; The image features and the cascade features are input into an object detection decoder to obtain an object detection result output by the object detection decoder.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1 to 8.

Images