Image detection method, model training method, and related device

Abstract

Embodiments of the present application disclose an image detection method, a model training method and related equipment; after obtaining a to-be-detected image and a target description text, the embodiments of the present application determine a text hidden state feature and a text output mapping feature of a target text feature of the target description text by using an image detection model, then determine an image hidden state feature and an image output mapping feature of a to-be-detected image feature of the to-be-detected image according to the text hidden state feature, determine an adjusted text feature and an adjusted text hidden state feature of the text output mapping feature according to the image hidden state feature, determine an adjusted image feature of the image output mapping feature according to the adjusted text hidden state feature, and finally detect a target position corresponding to the target detection object from the to-be-detected image according to the adjusted text feature and the adjusted image feature; by using the scheme, the accuracy and efficiency of an open vocabulary detection task can be improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an image detection method, a model training method, and related equipment. Background Technology

[0002] Object detection is a core problem in computer vision. It involves identifying all objects of interest in an image and determining their category and location. For example, in industrial AI quality inspection, it can be used to detect and classify defects in images of industrial parts. Traditional object detection models are trained on closed datasets, and the categories of objects they can detect need to be predefined. This limits the technology's capabilities in industrial AI quality inspection, as many new defects cannot be predefined in this context.

[0003] To overcome this limitation, object detection techniques can be transformed into open-vocabulary detection techniques, which require the model to detect objects beyond a predefined set of categories. However, open-vocabulary detection models in related technologies are limited by neck feature fusion mechanisms, resulting in a limited guided receptive field and secondary complexity, leading to low detection accuracy and efficiency. Summary of the Invention

[0004] This application provides an image detection method, a model training method, and related equipment, which can improve the accuracy and efficiency of open vocabulary detection tasks.

[0005] In a first aspect, embodiments of this application provide an image detection method, including:

[0006] Acquire the image to be detected, and the target description text corresponding to the target detection object in the image to be detected;

[0007] An image detection model is used to determine the target text features in the state space, including the hidden text state features and the text output mapping features.

[0008] Based on the text hidden state features, determine the image hidden state features and image output mapping features of the image to be detected in the state space;

[0009] Based on the image hidden state features, the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space are determined, and based on the adjusted text hidden state features, the adjusted image features of the image output mapping features in the state space are determined.

[0010] Based on the adjusted text features and the adjusted image features, the target location corresponding to the target detection object is detected from the image to be detected.

[0011] Accordingly, embodiments of this application provide an image detection device, including:

[0012] The first acquisition unit is used to acquire the image to be detected and the target description text corresponding to the target detection object in the image to be detected.

[0013] The first text feature determination unit is used to use an image detection model to determine the text hidden state features and text output mapping features of the target text in the state space of the target description text.

[0014] The first image feature determination unit is used to determine the image hidden state features and image output mapping features of the image to be detected in the state space based on the text hidden state features.

[0015] The first adjustment unit is used to determine the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space based on the image hidden state features, and to determine the adjusted image features of the image output mapping features in the state space based on the adjusted text hidden state features.

[0016] The detection unit is used to detect the target location corresponding to the target object from the image to be detected based on the adjusted text features and the adjusted image features.

[0017] In some embodiments, the first text feature determination unit may be specifically used to obtain the first text input mapping parameters and the first text output mapping parameters of the first text state space network, and adjust the target text features according to the first text input mapping parameters to obtain text input mapping features; obtain the first text state features of the first text state space network at the current time, and fuse the first text state features with the text input mapping features to obtain the text hidden state features; and adjust the text hidden state features according to the first text output mapping parameters to obtain the text output mapping features.

[0018] In some embodiments, the first image feature determination unit may be specifically used to perform feature segmentation on the image features to be detected using the first image state space network to obtain a first image feature and a second image feature; determine the image hidden state feature and the first image output mapping feature in the state space based on the text hidden state feature; and fuse the first image output mapping feature and the second image feature to obtain the image output mapping feature.

[0019] In some embodiments, the first image feature determination unit may be specifically used to obtain the first image state space parameters of the first image state space network and the first image state features of the first image state space network at the current time; adjust the first image state space parameters according to the text hiding state features to obtain the adjusted first image state space parameters; and adjust the first image features based on the adjusted first image state space parameters and the first image state features to obtain the image hiding state features and the first image output mapping features.

[0020] In some embodiments, the first image feature determination unit may be specifically used to adjust the first image state transition parameters according to the text hidden state features to obtain adjusted first image state transition parameters; adjust the first image input mapping parameters according to the text hidden state features to obtain adjusted first image input mapping parameters; adjust the first image output mapping parameters according to the text hidden state features to obtain adjusted first image output mapping parameters; and use the adjusted first image state transition parameters, the adjusted first image input mapping parameters, and the adjusted first image output mapping parameters as the adjusted first image state space parameters.

[0021] In some embodiments, the first image feature determination unit may be specifically configured to: adjust the first image state feature according to the adjusted first image state transition parameter to obtain the adjusted first image state feature; adjust the first image feature according to the adjusted first image input mapping parameter to obtain the adjusted first image feature; fuse the adjusted first image state feature and the adjusted first image feature to obtain the image hidden state feature; and determine the first image output mapping feature based on the adjusted first image output mapping parameter and the image hidden state feature.

[0022] In some embodiments, the first adjustment unit may be specifically used to obtain the second text state space parameters of the second text state space network; adjust the second text state space parameters according to the image hidden state features to obtain the adjusted second text state space parameters; and adjust the text output mapping features according to the adjusted second text state space parameters to obtain the adjusted text features and the adjusted text hidden state features.

[0023] In some embodiments, the first adjustment unit may be specifically used to adjust the text output mapping feature according to the adjusted second text input mapping parameter to obtain the adjusted text output mapping feature; obtain the second text state feature of the second text state space network at the current time, and adjust the second text state feature according to the adjusted second text state transition parameter to obtain the adjusted second text state feature; fuse the adjusted text output mapping feature with the adjusted second text state feature to obtain the adjusted text hidden state feature; and determine the adjusted text feature based on the adjusted second text output mapping parameter and the adjusted text hidden state feature.

[0024] In some embodiments, the first adjustment unit may be specifically used to perform feature segmentation on the image output mapping features using the second image state space network to obtain a first image output feature and a second image output feature; adjust the first image output feature according to the adjusted text hidden state feature to obtain the adjusted image output mapping feature of the first image output feature in the state space; and fuse the adjusted image output mapping feature with the second image output feature to obtain the adjusted image feature.

[0025] In some embodiments, the first adjustment unit may be specifically used to obtain the second image state space parameters of the second image state space network and the second image state features of the second image state space network at the current time; adjust the second image state space parameters according to the adjusted text hiding state features to obtain the adjusted second image state space parameters; and adjust the first image output features based on the adjusted second image state space parameters and the second image state features to obtain the adjusted image output mapping features.

[0026] In some embodiments, the first adjustment unit may be specifically used to adjust the second image state transition parameters according to the adjusted text hidden state features to obtain adjusted second image state transition parameters; adjust the second image input mapping parameters according to the adjusted text hidden state features to obtain adjusted second image input mapping parameters; adjust the second image output mapping parameters according to the adjusted text hidden state features to obtain adjusted second image output mapping parameters; and use the adjusted second image state transition parameters, the adjusted second image input mapping parameters, and the adjusted second image output mapping parameters as the adjusted second image state space parameters.

[0027] In some embodiments, the first adjustment unit may be specifically configured to adjust the second image state features according to the adjusted second image state transition parameters to obtain adjusted second image state features; adjust the first image output features according to the adjusted second image input mapping parameters to obtain adjusted first image output features; fuse the adjusted second image state features and the adjusted first image output features to obtain adjusted image hidden state features; and adjust the adjusted image hidden state features according to the adjusted second image output mapping parameters to obtain adjusted image output mapping features of the first image output features in the state space.

[0028] In some embodiments, the detection unit may be specifically configured to: identify candidate positions corresponding to the target detection object from the image to be detected based on the adjusted image features; filter object features corresponding to the candidate positions from the adjusted image features; compare the object features with the adjusted text features; and identify the target position corresponding to the target detection object from the candidate positions based on the comparison results.

[0029] Secondly, embodiments of this application also provide a model training method, including:

[0030] Obtain image samples and corresponding text description data for the image samples, wherein the image samples include location annotation data corresponding to at least one labeled object;

[0031] A preset image detection model is used to determine the text hidden state features and text output mapping features of the text description data in the state space;

[0032] Based on the text hidden state features, determine the image hidden state features and image output mapping features of the image sample in the state space;

[0033] Based on the image hidden state features, the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space are determined, and based on the adjusted text hidden state features, the adjusted image features of the image output mapping features in the state space are determined.

[0034] Based on the adjusted text features and the adjusted image features, predict the location prediction data of the labeled object;

[0035] Based on the adjusted text features, the text features, the location prediction data, and the location annotation data, the preset image detection model is trained to obtain the image detection model.

[0036] Accordingly, embodiments of this application also provide an image detection device, including:

[0037] The second acquisition unit is used to acquire image samples and text description data corresponding to the image samples, wherein the image samples include location annotation data corresponding to at least one labeled object;

[0038] The second text feature determination unit is used to determine the text hidden state features and text output mapping features of the text description data in the state space using a preset image detection model.

[0039] The second image feature determination unit is used to determine the image hidden state features and image output mapping features of the image sample in the state space based on the text hidden state features.

[0040] The second adjustment unit is used to determine the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space based on the image hidden state features, and to determine the adjusted image features of the image output mapping features in the state space based on the adjusted text hidden state features.

[0041] The prediction unit is used to predict the location prediction data of the labeled object based on the adjusted text features and the adjusted image features;

[0042] The training unit is used to train the preset image detection model based on the adjusted text features, the text features, the location prediction data, and the location annotation data to obtain the image detection model.

[0043] In some embodiments, the training unit may specifically be used to compare the adjusted text features with the text features and determine the text feature loss based on the comparison result; compare the location prediction data with the location annotation data and determine the location prediction loss based on the comparison result; and converge the preset image detection model based on the text feature loss and the location prediction loss to obtain the image detection model.

[0044] Furthermore, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the image detection method and model training method provided in embodiments of this application.

[0045] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the image detection method and model training method provided in embodiments of this application.

[0046] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the image detection methods and model training methods provided in embodiments of this application.

[0047] This application embodiment acquires an image to be detected and target description text corresponding to the target detection object in the image to be detected. Then, using an image detection model, it determines the text hidden state features and text output mapping features of the target text features in the state space. Next, based on the text hidden state features, it determines the image hidden state features and image output mapping features of the image to be detected in the state space. Then, based on the image hidden state features, it determines the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space, and based on the adjusted text hidden state features, it determines the adjusted image features of the image output mapping features in the state space. Finally, based on the adjusted text features and adjusted image features, it detects the target location corresponding to the target detection object from the image to be detected. This scheme utilizes an image detection model to map the target text features of the target description text into the state space to obtain text hidden state features, and to map the image features of the image to be detected into the state space to obtain image hidden state features. Thus, the text hidden state features and image hidden state features serve as intermediaries for feature fusion between different modal features (text features and image features), sequentially achieving three-stage feature fusion from text to image, from image to text, and then from text to image, resulting in adjusted text features and adjusted image features. Subsequently, the image detection model uses the adjusted text features and adjusted image features obtained from the three-stage feature fusion to predict the target location corresponding to the target object. This process not only improves the ability to capture the global receptive field during feature fusion but also enhances the fine-grainedness of feature fusion, improving the prediction accuracy of the image detection model in open vocabulary detection tasks. Furthermore, since feature fusion in the state space has linear complexity, it significantly reduces the computational load of the image detection model and improves its prediction efficiency. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram illustrating an application scenario of the model training method provided in the embodiments of this application;

[0050] Figure 2 This is a flowchart illustrating the model training method provided in the embodiments of this application;

[0051] Figure 3 This is a schematic diagram of an image sample provided in an embodiment of this application;

[0052] Figure 4 This is a data processing diagram of the model training method provided in an embodiment of this application;

[0053] Figure 5 This is a schematic diagram of a state-space model;

[0054] Figure 6 This is a schematic diagram of data processing in the first image state space network provided in the embodiments of this application;

[0055] Figure 7 This is a schematic diagram of data processing in a parallel first state space sub-network provided in an embodiment of this application;

[0056] Figure 8 This is another data processing diagram of the model training method provided in the embodiments of this application;

[0057] Figure 9 This is a schematic diagram of data processing in the second image state space network provided in the embodiments of this application;

[0058] Figure 10 This is a schematic diagram of data processing in a parallel second state space sub-network provided in an embodiment of this application;

[0059] Figure 11 This is a schematic diagram illustrating an application scenario of the image detection method provided in the embodiments of this application;

[0060] Figure 12 This is a schematic flowchart of the image detection method provided in the embodiments of this application;

[0061] Figure 13 This is a data processing schematic diagram of the image detection method provided in the embodiments of this application;

[0062] Figure 14 This is another data processing schematic diagram of the image detection method provided in the embodiments of this application;

[0063] Figure 15 This is a schematic diagram illustrating the target locations of some defects detected by an image detection model in industrial artificial intelligence quality inspection, as provided in an embodiment of this application.

[0064] Figure 16 This is another schematic flowchart of the image detection method provided in the embodiments of this application;

[0065] Figure 17 This is a schematic diagram of the structure of the image detection device provided in the embodiments of this application;

[0066] Figure 18 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application;

[0067] Figure 19 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0068] The technical solutions described below, with reference to the accompanying drawings, will be clearly and completely described. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0069] This application provides an image detection method, a model training method, and related equipment. The related equipment may include an image detection device, a model training device, an electronic device, a computer program product, and a computer-readable storage medium. The image detection device and the model training device may be integrated into an electronic device, which may be a server or a user terminal, etc.

[0070] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud-preset databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN) acceleration services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0071] Before introducing the image detection method, this application embodiment first describes the training process of the image detection model used in the image detection method. The image detection model involved in the image detection method of this application embodiment is trained based on a preset image detection model. Figure 1 A schematic diagram illustrating an application scenario of the model training method provided in this application embodiment is shown. For example... Figure 1As shown, taking an example where the model training device is integrated into an electronic device, and the electronic device is a server, the server can acquire image samples and corresponding text description data. The image samples include location annotation data corresponding to at least one labeled object. Using a preset image detection model, the server determines the text hidden state features and text output mapping features of the text description data in the state space. Based on the text hidden state features, the server determines the image hidden state features and image output mapping features of the image samples in the state space. Based on the image hidden state features, the server determines the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space, and based on the adjusted text hidden state features, the server determines the adjusted image features of the image output mapping features in the state space. Based on the adjusted text features and adjusted image features, the server predicts the location prediction data of the labeled object. Based on the adjusted text features, text features, location prediction data, and location annotation data, the server trains the preset image detection model to obtain the image detection model.

[0072] It is understood that, in the specific embodiments of this application, image samples, text description data corresponding to the image samples, and other related data are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0073] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0074] This embodiment will be described from the perspective of a model training device, which can be integrated into an electronic device, such as a server or a terminal. The terminal can include tablet computers, laptops, personal computers (PCs), wearable devices, virtual reality devices, or other smart devices that can generate image files.

[0075] A model training method includes: acquiring image samples and corresponding text description data, wherein the image samples include location annotation data corresponding to at least one labeled object; using a pre-set image detection model, determining the text hidden state features and text output mapping features of the text description data in the state space; determining the image hidden state features and image output mapping features of the image samples in the state space based on the text hidden state features; determining the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space based on the image hidden state features, and determining the adjusted image features of the image output mapping features in the state space based on the adjusted text hidden state features; predicting the location prediction data of the labeled object based on the adjusted text features and adjusted image features; and training the pre-set image detection model based on the adjusted text features, text features, location prediction data, and location annotation data to obtain an image detection model.

[0076] Figure 2 A schematic flowchart of the model training method provided in an embodiment of this application is shown. Figure 2 As shown, the specific process of this model training method is as follows:

[0077] 101. Obtain image samples and corresponding text description data. Image samples include location annotation data corresponding to at least one labeled object.

[0078] In this embodiment, the image sample and its corresponding text description data can be understood as an image-text pair. The image sample is the image in the image-text pair, and the corresponding text description data is the text in the image-text pair. The image sample can be a single image or a video frame contained in a video. The image sample may include location annotation data corresponding to at least one labeled object. This labeled object can be understood as the object that needs to be identified by the image detection model during the training phase. The labeled object can be an object with detection or recognition value in any scenario. For example, the labeled object can be a person, musical instrument, vehicle, house, object, etc. in a target detection scenario, or various defects or flaws in an industrial quality inspection scenario, or objects that need to be detected or recognized in other scenarios, which will not be listed here. The location annotation data can be understood as the actual location region of the labeled object in the image sample, and the location region can be marked with a bounding box.

[0079] Text description data corresponding to an image sample can be understood as text data generated by describing the image content (including but not limited to labeled objects) contained in the image sample. The text length, vocabulary type, or text content of the text description data can be adjusted according to the actual situation of the image sample. For example, for an image sample, text language of different lengths can be used to describe it, resulting in text description data. Longer text description data may provide a more detailed description of the image sample, while shorter text description data may provide a more concise description. For example, for an image sample, its corresponding text description data can include nouns, prepositions, conjunctions, etc., or it can use only nouns as the vocabulary type of the text description data. Furthermore, for an image sample, different text content can be used as the text description data for different labeled objects, so that each text description data has a different focus for the image sample. The image sample can also be combined with different text description data for model training.

[0080] Figure 3 A schematic diagram of an image sample provided in an embodiment of this application is shown. For example... Figure 3 As shown, the image sample includes two labeled objects: a butterfly and a cat. For each labeled object, a bounding box can be used to define its location, representing the corresponding positional annotation data. Specifically, bounding box 1 can represent the positional annotation data for the butterfly, and bounding box 2 can represent the positional annotation data for the cat. Furthermore, the image sample also includes corresponding textual description data. For example, the textual description data can be expressed as: "A kitten is chasing a butterfly on the grass." It can also be expressed as: "A cat is chasing a butterfly." Alternatively, it can be expressed as: "Cat, butterfly." It is understood that the textual description data can also be expressed in other ways, and this embodiment does not limit its expression.

[0081] In this embodiment of the application, the image samples and the corresponding text description data can be pre-stored in a database. The server can directly call the database to obtain the image samples and the corresponding text description data stored therein.

[0082] 102. Using a pre-defined image detection model, determine the text features of the text description data in the state space, including the text hidden state features and the text output mapping features.

[0083] A pre-defined image detection model can be understood as a network model that can be used to perform open-vocabulary detection tasks. Open-vocabulary detection is a task in the field of computer vision that aims to detect objects of any category in an image, including new categories that did not appear in the training data (i.e., "open vocabulary"). Open-vocabulary detection can overcome the limitation of traditional object detection models that can only recognize predefined fixed categories (closed vocabulary), greatly improving the model's generalization ability and practical application scope.

[0084] In this embodiment, text features can be extracted from the text description data. Since image detection targets or identifies objects such as people or objects in image samples, and these objects are generally represented by nouns, the text description data can first be segmented to obtain text word fragments. Then, target text word fragments of noun type are selected from these fragments, and feature extraction is performed on these target text word fragments to obtain text features. This method removes redundant information from the text description data, resulting in text features specific to the objects in the image sample.

[0085] In this embodiment, a CLIP (Contrastive Language-Image Pre-Training) text encoder can be used to extract features from the text description data, thereby obtaining text features. The CLIP text encoder can understand the semantics of the text description data and convert it into feature vectors for subsequent data processing in the network model.

[0086] After determining the text features of the text description data, a preset image detection model can be used to convert them into feature representations in the state space. That is, the preset image detection model is used to determine the text hidden state features and text output mapping features corresponding to the text features in the state space.

[0087] Figure 4 A data processing schematic diagram of the model training method provided in an embodiment of this application is shown. For example... Figure 4 As shown, the preset image detection model may include a first text state space network. The first text state space network can be understood as a state space model that processes text data. There can be various types of state space models, such as the Mamba model. The basic idea of ​​a state space model (SSM) is to describe the evolution of a system through a hidden "state," and the observed data (i.e., the output of the state space model) is based on this hidden state.

[0088] Figure 5 A schematic diagram of a state-space model is shown. For example... Figure 5 As shown, without omitting the direct transfer matrix, for a continuous input signal u(t)∈R, the state-space model uses hidden states h(t)∈R. E Mapping this to a continuous output signal y(t)∈R, the process can be represented by state equations and output equations:

[0089] State equation: h'(t)=A*h(t)+B*u(t)

[0090] Output equation: y(t)=C*h(t)

[0091] Where E is the state expansion factor of the state-space model, and A∈R E×E It is the state transition matrix, B∈R E×1 Let C ∈ R be the input mapping matrix. 1×E Let h'(t) be the output mapping matrix, and h'(t) be the derivative of h(t). According to the above formula, the input signal u(t) influences the output signal y(t) by affecting the hidden state h(t). Based on the state-space model, Mamba further introduces a selective scan algorithm, making A, B, and C functions of the input sequence.

[0092] It is understood that the networks included in the preset image detection model provided in this application embodiment (such as the first text state space network, the first image state space network, the second text state space network, and the second image state space network) all involve the network structure of state space networks. In specific descriptions, the parameter values ​​of the state transition matrix A, the input mapping matrix B, and the output mapping matrix C in the state space model are different in different networks.

[0093] Based on this, a preset image detection model is used to determine the text features of the text description data in the state space, including the text hidden state features and text output mapping features. This can include: obtaining the first text input mapping parameters and the first text output mapping parameters of the first text state space network, and adjusting the text features according to the first text input mapping parameters to obtain the text input mapping features; obtaining the first text state features of the first text state space network at the current time, and fusing the first text state features with the text input mapping features to obtain the text hidden state features; and adjusting the text hidden state features according to the first text output mapping parameters to obtain the text output mapping features.

[0094] In this context, the text features of the text description data can be understood as the input signal u(t) of the first text state space network, the first text input mapping parameter can be understood as the input mapping matrix B of the first text state space network, and the first text output mapping parameter can be understood as the output mapping matrix C of the first text state space network.

[0095] The text input mapping feature obtained by adjusting the text features according to the first text input mapping parameters is B*u(t) in the state equation. In other words, the process of adjusting the text features according to the first text input mapping parameters can be understood as: performing matrix multiplication between the first text input mapping parameters and the text features to obtain the text input mapping feature.

[0096] It is understandable that, in the state equation, when performing matrix operations between the input signal u(t) and the input mapping matrix B, it is necessary to ensure that the input signal u(t) ∈ R. 1×E Therefore, in the process of extracting text features from text description data, it is necessary to consider the matrix dimension of the input mapping matrix B in the first text state space network to determine the feature dimension corresponding to the text features.

[0097] According to the state equation, h(t) represents the hidden state obtained at the current time, and h'(t) is the derivative of h(t). That is, h'(t) can represent the influence of the input signal u(t) on the hidden state based on h(t). Since the hidden state h(t) changes continuously with the input signal in the state-space model, taking time t as the input signal being input into the state equation at the current time as an example, h(t) in the state equation represents the hidden state corresponding to the previous time t-1 (i.e., the hidden state obtained at the current time t before the input signal u(t) is input into the state equation). Based on this, after calculating h'(t) by inputting the input signal u(t) into the state equation, h'(t) needs to be fused with h(t) (the fusion method can be matrix addition) to obtain the updated hidden state at the current time. This updated hidden state can then be used as the hidden state obtained at the next time t+1, and the hidden state update calculation for the next time step can be performed accordingly.

[0098] In this embodiment, the first text state feature is the hidden state obtained at the current moment. Based on this, fusing the first text state feature with the text input mapping feature to obtain the text hidden state feature specifically includes: obtaining the first text state transition parameters of the first text state space network; adjusting the first text state feature according to the first text state transition parameters (e.g., matrix multiplication) to obtain the adjusted first text state feature; fusing the adjusted first text state feature with the text input mapping feature (e.g., matrix addition) to obtain the first hidden state change feature at the current moment; and fusing the first hidden state change feature with the first text state feature (e.g., matrix addition) to obtain the text hidden state feature.

[0099] It is understandable that when adjusting hidden features (including hidden features of text type and hidden features of image type) in the future, the above method can be used to calculate the adjusted hidden features.

[0100] After obtaining the text hidden state features, the text hidden state features can be adjusted using the first text output mapping parameter according to the output equation to obtain the text output mapping features. The text output mapping features are the output signal y(t) in the output equation.

[0101] 103. Based on the text hidden state features, determine the image hidden state features and image output mapping features of the image samples in the state space.

[0102] After determining the textual hidden state features of the textual features in the state space using the first textual state space network, these textual hidden state features can be used as an intermediary for the first feature fusion between different modal features (textual features and image features). Specifically, based on the textual hidden state features, the image hidden state features and image output mapping features of the image features of the image samples in the state space can be determined.

[0103] In this embodiment, image features can be extracted from image samples. Specifically, the backbone network of the YOLO (You Only Look Once) series of models can be used to extract features from the image samples, resulting in multi-scale image features. Then, based on the text hidden state features, the image hidden state features and image output mapping features in the state space of these multi-scale image features are determined.

[0104] like Figure 4 As shown, the preset image detection model may further include a first image state space network. The first image state space network can be understood as a state space model that aggregates text to image features. Therefore, the first image state space network also has various matrices of a state space model (such as state transition matrix, input mapping matrix, and output mapping matrix), and features can also be processed according to state equations and output equations in the first image state space network.

[0105] Based on this, according to the text hidden state features, the image hidden state features and image output mapping features of the image sample in the state space are determined, which may include: using a first image state space network to perform feature segmentation on the image features to obtain first image features and second image features; determining the image hidden state features and first image output mapping features of the first image features in the state space according to the text hidden state features; and fusing the first image output mapping features and the second image features to obtain image output mapping features.

[0106] Figure 6 This illustration shows a data processing diagram in a first image state space network provided in an embodiment of this application. For example... Figure 6 As shown, the network structure of the first image state space network integrates the network structure of Cross Stage Partial Connections (CSP) and the state space model. Cross Stage Partial Connections, as a network structure design method, can maintain model performance while reducing computational cost. It divides the features (graph) into two branches, applies complex convolutions or other operations to one branch, while the other branch retains the original features, and finally merges the two.

[0107] In this embodiment, image features can be input into a first image state space network. The first image state space network can perform feature segmentation on the image features to obtain two branches, namely, first image features and second image features. Then, feature processing is performed on one branch (first image features) to obtain image hidden state features and first image output mapping features; then, the first image output mapping features and the other branch (second image features) are fused to obtain image output mapping features.

[0108] The first image state space network includes a parallel state space sub-network, which is referred to as the parallel first state space sub-network for easy distinction. During the feature processing of one of its branches (first image features), the parallel first state space sub-network can be used to determine the image hidden state features and the first image output mapping features of the first image features in the state space based on the text hidden state features.

[0109] In this embodiment of the application, determining the image hidden state features and the first image output mapping features of the first image features in the state space based on the text hidden state features may include: obtaining the first image state space parameters of the first image state space network and the first image state features of the first image state space network at the current time; adjusting the first image state space parameters according to the text hidden state features to obtain the adjusted first image state space parameters; and adjusting the first image features based on the adjusted first image state space parameters and the first image state features to obtain the image hidden state features and the first image output mapping features.

[0110] The first image state space parameters of the first image state space network are the same as the state space parameters in the parallel first state space sub-network. The first image state space parameters may include first image state transition parameters, first image input mapping parameters, and first image output mapping parameters. Specifically, the first image state transition parameters are the state transition matrix (denoted by A) in the network (either the first image state space network or the parallel first state space sub-network), the first image input mapping parameters are the input mapping matrix (denoted by B) in the network (either the first image state space network or the parallel first state space sub-network), and the first image output mapping parameters are the output mapping matrix (denoted by C) in the network (either the first image state space network or the parallel first state space sub-network).

[0111] The first image state feature of the first image state space network at the current time can be understood as the hidden state that the network (the first image state space network or the parallel first state space sub-network) has obtained at the current time.

[0112] After obtaining the state transition parameters of the first image, the state space parameters of the first image can be adjusted according to the hidden state features of the text to obtain the adjusted state space parameters of the first image. Then, the features of the first image can be processed according to the state equation and output equation of the state space model to obtain the hidden state features of the image and the output mapping features of the first image.

[0113] Specifically, adjusting the state space parameters of the first image based on the text-hidden state features to obtain adjusted state space parameters of the first image may include: adjusting the state transition parameters of the first image based on the text-hidden state features to obtain adjusted state transition parameters of the first image; adjusting the input mapping parameters of the first image based on the text-hidden state features to obtain adjusted input mapping parameters of the first image; adjusting the output mapping parameters of the first image based on the text-hidden state features to obtain adjusted output mapping parameters of the first image; and using the adjusted state transition parameters, the adjusted input mapping parameters, and the adjusted output mapping parameters of the first image as the adjusted state space parameters of the first image.

[0114] Figure 7 A schematic diagram of data processing in a parallel first state space sub-network provided in an embodiment of this application is shown. Figure 7 As shown, the text hidden state features can be incorporated into the first image state transition parameters, the first image input mapping parameters, and the first image output mapping parameters to adjust these parameters, thereby obtaining the adjusted first image state space parameters. The adjusted first image state space parameters include the adjusted first image state transition parameters, the adjusted first image input mapping parameters, and the adjusted first image output mapping parameters.

[0115] In this embodiment, there are several ways to adjust the first image state transition parameters based on the text hidden state features. For example, the feature values ​​in the text hidden state features can be used as weight parameters to weight the first image state transition parameters, thereby obtaining the adjusted first image state transition parameters. Another example is to treat the text hidden state features as one matrix and the first image state transition parameters as another matrix, and then perform matrix multiplication on the two matrices; the result is the adjusted first image state transition parameters, and so on.

[0116] Similarly, the method for adjusting the input mapping parameters of the first image based on the text hiding state features can refer to the above method, and the method for adjusting the output mapping parameters of the first image based on the text hiding state features can also refer to the above method. These will not be elaborated on in the embodiments of this application.

[0117] In this embodiment, the parameters of the first image state space (including the first image state transition parameters, the first image input mapping parameters, and the first image output mapping parameters) are adjusted according to the text hidden state features. This process can use the text hidden state features as input conditions to adjust the various parameters in the first image state space network in parallel, thereby improving the feature fusion degree of subsequent text-image.

[0118] As mentioned above, after obtaining the adjusted first image state space parameters, the first image features can be processed according to the state equation and output equation of the state space model to obtain image hidden state features and first image output mapping features. In this embodiment, adjusting the first image features based on the adjusted first image state space parameters and first image state features to obtain image hidden state features and first image output mapping features may include: adjusting the first image state features according to the adjusted first image state transition parameters to obtain adjusted first image state features; adjusting the first image features according to the adjusted first image input mapping parameters to obtain adjusted first image features; fusing the adjusted first image state features and the adjusted first image features to obtain image hidden state features; and determining the first image output mapping features based on the adjusted first image output mapping parameters and image hidden state features.

[0119] According to the data processing method of state equation, the state features of the first image can be adjusted according to the state transition parameters of the first image after adjustment to obtain the adjusted state features of the first image. Then, the features of the first image can be adjusted according to the input mapping parameters of the first image after adjustment to obtain the adjusted features of the first image. Finally, the adjusted state features of the first image and the adjusted features of the first image can be fused to obtain the hidden state features of the image.

[0120] The method of adjusting the first image state features based on the adjusted first image state transition parameters can include: performing a matrix multiplication operation between the adjusted first image state transition parameters and the first image state features, with the result being the adjusted first image state features. Similarly, the method of adjusting the first image features based on the adjusted first image input mapping parameters can include: performing a matrix multiplication operation between the adjusted first image input mapping parameters and the first image features, with the result being the adjusted first image features. The method of fusing the adjusted first image state features and the adjusted first image features can include: performing a matrix addition operation between the adjusted first image state features and the adjusted first image features, with the result being the image hidden state features.

[0121] According to the data processing method of the output equation, the adjusted first image output mapping parameters and the image hidden state features can be multiplied by matrix, and the result is the first image output mapping features.

[0122] In step 103, the text hidden state features are introduced into the first image state space network and injected into the first image state space parameters of the first image state space network to adjust the first image state space parameters. This adjusts the first image state space parameters to incorporate the relevant features of the text description data. Then, the relevant features of the text description data are passed to the image hidden state features and image output mapping features through the relevant calculation rules of the subsequent state space model to complete the first feature fusion. The first feature fusion is a feature fusion from text to image, which can improve the alignment ability of text and image from the perspective of the global receptive field, laying the foundation for the preset image detection model to complete the subsequent prediction task. Furthermore, the state space model is used as the calculation basis for fusion in the first feature fusion process. Since the state space model has linear complexity, its O(N+1) complexity is significantly lower than the unavoidable quadratic complexity of feature fusion in related technologies (such as Transformer), thus significantly reducing the computational load of the model and improving the prediction efficiency of the model.

[0123] Furthermore, as mentioned earlier, the image features obtained after feature extraction from image samples can be multi-scale image features. Based on this, the image hidden state features and image output mapping features of the multi-scale image features of the image samples in the state space can be determined according to the text hidden state features.

[0124] Figure 8 This illustration shows another data processing diagram of the model training method provided in an embodiment of this application. For example... Figure 8As shown, multi-scale image features can include multiple image features of different scales, and these features can be extracted using convolution methods of different scales. Furthermore, for image features of different scales, the hidden state features and output mapping features of the image features at that scale in the state space can be determined based on the text hidden state features.

[0125] For any image feature at any scale, a first image state space network corresponding to that scale can be used to calculate its hidden state features and output mapping features in the state space. The specific processing method is the same as described above, which determines the hidden state features and output mapping features of the image sample in the state space based on the hidden state features of the text. For example, taking any image feature at any scale as an example, the first image state space network corresponding to that scale can be used to segment its features, obtaining a first image feature and a second image feature; based on the hidden state features of the text, the hidden state features and output mapping features of the first image feature in the state space can be determined; then, the first output mapping features and the second image feature are fused to obtain the output mapping features.

[0126] The method for determining the image hidden state features and the first image output mapping features in the state space based on the text hidden state features is the same as the aforementioned processing method. The method for fusing the first image output mapping features and the second image features to obtain the image output mapping features is also the same as the aforementioned processing method, and will not be elaborated upon here.

[0127] After obtaining the hidden state features corresponding to image features at different scales, these hidden state features can be fused to obtain the hidden state features of the image sample.

[0128] In this embodiment, text hidden state features are introduced into the first image state space network corresponding to each of the multi-scale image features, and then injected into the first image state space parameters of the first image state space network. This adjusts the first image state space parameters corresponding to different scale image features, ensuring that the adjusted first image state space parameters for different scale image features incorporate relevant features of the text description data. Furthermore, the relevant features of the text description data are passed to the image hidden state features and image output mapping features through the relevant calculation rules of the subsequent state space model, completing the first feature fusion. Figure 8By using a filled-pattern arrow to indicate the first feature fusion between the first text state space network and the first image state space network, multi-scale image features can be fused at the pixel level rather than the channel level, improving the fine-grainedness of the feature fusion process. Furthermore, determining the image hidden state features and image output mapping features based on the multi-scale image features using the above method can also improve the text-image alignment capability from the perspective of the global receptive field, laying the foundation for the preset image detection model to complete subsequent prediction tasks. Moreover, using the state space model as the computational basis for fusion in the first feature fusion process, due to the linear complexity of the state space model, its O(N+1) complexity is significantly lower than the unavoidable quadratic complexity in feature fusion in related techniques (such as Transformer), thus significantly reducing the computational load of the model and improving its prediction efficiency.

[0129] 104. Based on the image hidden state features, determine the adjusted text features and adjusted text hidden state features in the state space of the text output mapping features, and based on the adjusted text hidden state features, determine the adjusted image features in the state space of the image output mapping features.

[0130] In this embodiment of the application, the process of determining the adjusted text features and the adjusted text hidden state features in the state space based on the image hidden state features can be understood as follows: using the image hidden state features as an intermediary for the second feature fusion between different modal features (text features and image features) to achieve the second feature fusion, which is the feature fusion from image to text.

[0131] Following this, the process of determining the adjusted image features in the state space based on the adjusted text hidden state features can be understood as: using the adjusted text hidden state features as an intermediary for the third feature fusion between different modal features (text features and image features), thereby realizing the third feature fusion from text to image.

[0132] like Figure 4 and Figure 8 As shown in the embodiments of this application, the preset image detection model may further include a second text state space network. The second text state space network can be understood as a state space model that aggregates image to text features. Therefore, the second text state space network has various matrices of a state space model (such as state transition matrices, input mapping matrices, and output mapping matrices), and features can also be processed according to state equations and output equations in the second text state space network. Furthermore, the second text state space network is used to implement a second feature fusion. Figure 8A blank arrow is used to indicate the second feature fusion between the first image state space network and the second text state space network.

[0133] Based on this, according to the image hidden state features, the adjusted text features and adjusted text hidden state features in the state space of the text output mapping features can be determined, which may include: obtaining the second text state space parameters of the second text state space network; adjusting the second text state space parameters according to the image hidden state features to obtain the adjusted second text state space parameters; and adjusting the text output mapping features according to the adjusted second text state space parameters to obtain the adjusted text features and adjusted text hidden state features.

[0134] The second text state space parameters can include the second text input mapping parameters and the second text output mapping parameters. The second text input mapping parameters can be understood as the input mapping matrix B of the second text state space network, and the second text output mapping parameters can be understood as the output mapping matrix C of the second text state space network.

[0135] In this embodiment, the adjusted second text state space parameters may include adjusted second text state transition parameters, adjusted second text input mapping parameters, and adjusted second text output mapping parameters. Based on this, adjusting the text output mapping features according to the adjusted second text state space parameters to obtain adjusted text features and adjusted text hidden state features may include: adjusting the text output mapping features according to the adjusted second text input mapping parameters to obtain adjusted text output mapping features; obtaining the second text state features of the second text state space network at the current time, and adjusting the second text state features according to the adjusted second text state transition parameters to obtain adjusted second text state features; fusing the adjusted text output mapping features and the adjusted second text state features to obtain adjusted text hidden state features; and determining the adjusted text features based on the adjusted second text output mapping parameters and the adjusted text hidden state features.

[0136] The adjusted text output mapping feature obtained by adjusting the text output mapping feature according to the adjusted second text input mapping parameter is B*u(t) in the state equation. In other words, the process of adjusting the text output mapping feature according to the adjusted second text input mapping parameter can be understood as: performing matrix multiplication between the adjusted second text input mapping parameter and the text output mapping feature to obtain the adjusted text output mapping feature.

[0137] Similar to the first text state feature in the aforementioned first text state space, since the hidden state h(t) in the state space model changes continuously with the input signal, according to the state equation, after calculating h'(t) by inputting the input signal u(t) into the state equation, h'(t) needs to be fused with h(t) (the fusion method can be matrix addition) to obtain the updated hidden state at the current time. In this embodiment, the second text state feature is the hidden state that the second text state space network has generated at the current time. Based on this, the adjusted second text state feature is fused with the adjusted text output mapping feature to obtain the adjusted text hidden state feature. Specifically, this may include: performing a matrix addition operation on the adjusted second text state feature and the adjusted text output mapping feature to obtain the second hidden state change feature of the adjusted second text state feature at the current time; and performing a matrix addition operation on the second hidden state change feature and the adjusted text output mapping feature to obtain the adjusted text hidden state feature.

[0138] After obtaining the adjusted text hidden state features, the adjusted text features can be determined based on the output equation, the adjusted second text output mapping parameters, and the adjusted text hidden state features. Specifically, the adjusted text features can be obtained by multiplying the second text output mapping parameters and the adjusted text hidden state features.

[0139] Understandable, Figure 8 In this process, for image features of different scales, the corresponding first image state space network can be used to obtain image hidden state features of different scales. At this time, the image hidden state features of different scales can be adjusted to features of the same dimension, and then feature fusion can be performed to obtain the image hidden state features of the image sample. Then, the text output mapping features are adjusted based on the image hidden state features according to the aforementioned processing method to obtain the adjusted text features and the adjusted text hidden state features.

[0140] In this embodiment, the obtained text output mapping features are used as the input signal of the second text state space network. Following the calculation principle of the state equation, and based on the image hidden state features, the adjusted text hidden state features of the text output mapping features in the state space are determined. Then, following the calculation principle of the output equation, and based on the image hidden state features, the adjusted text features of the text output mapping features in the state space are determined. Through these operations, prior knowledge of the image hidden state features previously fused with text features can be injected into the text output mapping features. This allows the prior knowledge to be compressed into the adjusted text hidden state features according to the state equation of the state space model. This is repeated, and the prior knowledge is further compressed into the adjusted text features according to the output equation of the state space model, achieving a second feature fusion and further improving the text-image alignment capability from the perspective of the global receptive field.

[0141] like Figure 4 and Figure 8 As shown in the embodiments of this application, the preset image detection model may further include a second image state space network. The second image state space network can be understood as a state space model that aggregates text-to-image features. Therefore, the second image state space network also has various matrices of a state space model (such as state transition matrices, input mapping matrices, and output mapping matrices), and features can also be processed according to state equations and output equations in the second image state space network. Furthermore, the second image state space network has a similar network structure to the first image state space network, the difference being that their input signals are different, and the parameter values ​​in the various matrices they contain are different. The second image state space network is used to implement a third feature fusion, which is a fusion from text to image. Figure 8 The third feature fusion is indicated by an arrow with a filled pattern between the second text state space network and the second image state space network.

[0142] Based on this, the adjusted image features of the image output mapping features in the state space are determined according to the adjusted text hidden state features. This can include: using a second image state space network to perform feature segmentation on the image output mapping features to obtain a first image output feature and a second image output feature; adjusting the first image output feature according to the adjusted text hidden state features to obtain the adjusted image output mapping features of the first image output feature in the state space; and fusing the adjusted image output mapping features with the second image output features to obtain the adjusted image features.

[0143] Figure 9 This illustration shows a data processing diagram in a second image state space network provided in an embodiment of this application. For example... Figure 9As shown, the network structure of the second image state space network integrates the network structure of cross-stage partial connectivity (CSP) and the state space model.

[0144] In this embodiment, the image output mapping features can be input into a second image state space network. The second image state space network can perform feature segmentation on the image output mapping features to obtain two branches, namely a first image output feature and a second image output feature. Then, feature processing is performed on one branch (the first image output feature) to obtain the adjusted image output mapping features of the first image output feature in the state space. The adjusted image output mapping features are then fused with the other branch (the second image output feature) to obtain the adjusted image features.

[0145] Similar to the first image state space network, the second image state space network also includes a parallel state space sub-network, which, for ease of distinction, is referred to as the parallel second state space sub-network. It can be understood that the network parameters of the parallel second state space sub-network contained in the second image state space network are different from the network parameters of the parallel first state space sub-network contained in the first image state space network.

[0146] During the feature processing of one of the branches (first image output features) mentioned above, a parallel second state space sub-network can be used to adjust the first image output features based on the adjusted text hidden state features, thereby obtaining the adjusted image output mapping features of the first image output features in the state space.

[0147] In this embodiment of the application, adjusting the first image output feature based on the adjusted text hiding state feature to obtain the adjusted image output mapping feature of the first image output feature in the state space may include: obtaining the second image state space parameters of the second image state space network and the second image state feature of the second image state space network at the current time; adjusting the second image state space parameters based on the adjusted text hiding state feature to obtain the adjusted second image state space parameters; and adjusting the first image output feature based on the adjusted second image state space parameters and the second image state feature to obtain the adjusted image output mapping feature.

[0148] The second image state space parameters of the second image state space network are the state space parameters in the parallel second state space sub-network. The second image state space parameters may include second image state transition parameters, second image input mapping parameters, and second image output mapping parameters. Specifically, the second image state transition parameters are the state transition matrix (denoted by A) in the network (either the second image state space network or the parallel second state space sub-network), the second image input mapping parameters are the input mapping matrix (denoted by B) in the network (either the second image state space network or the parallel second state space sub-network), and the second image output mapping parameters are the output mapping matrix (denoted by C) in the network (either the second image state space network or the parallel second state space sub-network).

[0149] The second image state features of the second image state space network at the current time can be understood as the hidden state that the network (the second image state space network or the parallel second state space sub-network) has obtained at the current time.

[0150] After obtaining the state transition parameters of the second image, the state space parameters of the second image can be adjusted according to the adjusted text hidden state features to obtain the adjusted state space parameters of the second image. Then, the output features of the first image can be processed according to the state equation and output equation of the state space model to obtain the adjusted image output mapping features.

[0151] Specifically, adjusting the state space parameters of the second image based on the adjusted text hidden state features to obtain the adjusted second image state space parameters may include: adjusting the state transition parameters of the second image based on the adjusted text hidden state features to obtain the adjusted second image state transition parameters; adjusting the input mapping parameters of the second image based on the adjusted text hidden state features to obtain the adjusted second image input mapping parameters; adjusting the output mapping parameters of the second image based on the adjusted text hidden state features to obtain the adjusted second image output mapping parameters; and using the adjusted second image state transition parameters, the adjusted second image input mapping parameters, and the adjusted second image output mapping parameters as the adjusted second image state space parameters.

[0152] Figure 10 A schematic diagram of data processing in a parallel second state space sub-network provided in an embodiment of this application is shown. Figure 10As shown, the adjusted text hidden state features can be incorporated into the second image state transition parameters, the second image input mapping parameters, and the second image output mapping parameters to adjust these parameters, thereby obtaining the adjusted second image state space parameters. The adjusted second image state space parameters include the adjusted second image state transition parameters, the adjusted second image input mapping parameters, and the adjusted second image output mapping parameters.

[0153] In this embodiment, there are several ways to adjust the second image state transition parameters based on the adjusted text hidden state features. For example, the feature values ​​in the adjusted text hidden state features can be used as weight parameters to weight the second image state transition parameters, thereby obtaining the adjusted second image state transition parameters. Another example is to treat the adjusted text hidden state features as one matrix and the second image state transition parameters as another matrix, and then perform matrix multiplication on the two matrices to obtain the adjusted second image state transition parameters, and so on.

[0154] Similarly, the method for adjusting the second image input mapping parameters based on the adjusted text hiding state parameters can refer to the above method, and the method for adjusting the second image output mapping parameters based on the adjusted text hiding state features can also refer to the above method. These will not be elaborated upon in the embodiments of this application.

[0155] In this embodiment, the parameters of the second image state space (including the second image state transition parameters, the second image input mapping parameters, and the second image output mapping parameters) are adjusted according to the adjusted text hidden state features. This process can use the adjusted text hidden state features as input conditions to achieve parallel adjustment of various parameters in the second image state space network, thereby further improving the feature fusion degree of subsequent text-image.

[0156] As mentioned above, after obtaining the adjusted second image state space parameters, the first image output features can be processed according to the state equation and output equation of the state space model to obtain the adjusted image output mapping features. In this embodiment, adjusting the first image output features based on the adjusted second image state space parameters and the second image state features to obtain the adjusted image output mapping features may include: adjusting the second image state features according to the adjusted second image state transition parameters to obtain the adjusted second image state features; adjusting the first image output features according to the adjusted second image input mapping parameters to obtain the adjusted first image output features; fusing the adjusted second image state features and the adjusted first image output features to obtain the adjusted image hidden state features; and adjusting the adjusted image hidden state features according to the adjusted second image output mapping parameters to obtain the adjusted image output mapping features of the first image output features in the state space.

[0157] According to the output processing method of the state equation, the state features of the second image can be adjusted according to the state transition parameters of the adjusted second image to obtain the adjusted state features of the second image, and the output features of the first image can be adjusted according to the input mapping parameters of the adjusted second image to obtain the adjusted output features of the first image. Then, the adjusted state features of the second image and the adjusted output features of the first image are fused to obtain the adjusted hidden state features of the image.

[0158] Following this, the data processing method of the output equation can be continued to perform matrix multiplication on the adjusted second image output mapping parameters and the adjusted image hidden state features. The result is the adjusted image output mapping features.

[0159] In step 104, the adjusted text hidden state features are introduced into the second image state space network and injected into the second image state space parameters of the second image state space network to adjust the second image state space parameters. This allows the adjusted second image state space parameters to incorporate the relevant features of the text description data and image samples. The relevant features of the text description data and image samples are then passed to the adjusted image output mapping features and the adjusted image hidden state features through the relevant calculation rules of the subsequent state space model, completing the third feature fusion. This further improves the text-image alignment capability from the perspective of the global receptive field, laying a solid foundation for the preset image detection model to complete subsequent prediction tasks. Furthermore, the state space model is also used as the calculation basis for fusion in the third feature fusion process. Because the state space model has linear complexity, its O(N+1) complexity is significantly lower than the unavoidable quadratic complexity in feature fusion in related technologies (such as Transformer), thus significantly reducing the computational load of the model and improving its prediction efficiency.

[0160] Furthermore, as mentioned earlier, the image features obtained after feature extraction from image samples can be multi-scale image features. Based on this, such as... Figure 8 As shown, for image output mapping features of different scales output by the first image state space network, the adjusted image hidden state features and adjusted image output mapping features at that scale can be determined in the state space based on the adjusted text hidden state features.

[0161] For any scale of image output mapping features, a second image state space network corresponding to that scale can be used to calculate the adjusted image hidden state features and adjusted image output mapping features in the state space. The specific processing method is the same as described above, where a first image state space network corresponding to any scale of image features is used to calculate the image hidden state features and image output mapping features in the state space. For example, taking any scale of image output mapping features as an example, a second image state space network corresponding to the initial image mapping features at that scale can be used for feature segmentation to obtain the first image output features and the second image output features; based on the adjusted text hidden state features, the adjusted image output mapping features of the first image output features in the state space are determined; then, the adjusted image output mapping features and the second image output features are fused to obtain the adjusted image features.

[0162] In this embodiment, the adjusted text hidden state features are introduced into the second image state space network corresponding to each of the multi-scale image output mapping features, and then injected into the second image state space parameters of the second image state space network. This adjusts the second image state space parameters corresponding to the image output mapping features at different scales, ensuring that the adjusted second image state space parameters for the image output mapping features at different scales incorporate the relevant features of the text description data and image samples. Furthermore, the relevant features of the text description data and image samples are passed to the adjusted image hidden state features and the adjusted image output mapping features through the relevant calculation rules of the subsequent state space model, completing the third feature fusion. The third feature fusion is a text-to-image feature fusion. Figure 8 By using a filled-pattern arrow to indicate the third feature fusion between the second text state space network and the second image state space network, multi-scale image output mapping features can be fused at the pixel level rather than the channel level, improving the fine-grainedness of the feature fusion process. Furthermore, by determining the adjusted image hidden state features and adjusted image output mapping features based on the multi-scale image output mapping features, the text-image alignment capability can also be improved from the perspective of the global receptive field, laying a solid foundation for the pre-defined image detection model to complete subsequent prediction tasks. Moreover, using a state space model as the computational basis for fusion in the third feature fusion process, the O(N+1) complexity of the state space model is significantly lower than the unavoidable quadratic complexity in feature fusion in related techniques (such as Transformer), thus significantly reducing the computational load of the model and improving its prediction efficiency.

[0163] It should be noted that, Figure 8 Multi-scale image features can include multiple image features at different scales. The dimensionality of feature maps can be transformed through feature upsampling and downsampling. For ease of explanation, let's take a multi-scale image feature consisting of three feature maps at different scales as an example. These three feature maps are named, in ascending order of feature dimension, the first-scale feature map, the second-scale feature map, and the third-scale feature map. The first-scale feature map is upsampled by a factor of 2 to obtain the second-scale feature map, and the second-scale feature map is upsampled by a factor of 2 to obtain the third-scale feature map. Correspondingly, the third-scale feature map is downsampled by a factor of 2 to obtain the second-scale feature map, and the second-scale feature map is downsampled by a factor of 2 to obtain the first-scale feature map.

[0164] During model training, different state space networks within the pre-defined image detection model can be used for feature processing on feature maps of different scales. For example, a second image state space network can be used for feature processing on a first-scale feature map; a first and second image state space networks can be used for feature processing on a second-scale feature map; and a first image state space network can be used for feature processing on a third-scale feature map. The processed feature maps of different scales generated during the above feature processing can be upsampled or downsampled based on the dimensional relationships between them, and then further processed according to... Figure 14 The data flow shown is processed to obtain the adjusted multi-scale image features.

[0165] 105. Based on the adjusted text features and adjusted image features, predict the location of the labeled object.

[0166] In this embodiment, after obtaining the adjusted text features and adjusted image features, the location prediction data of the labeled objects can be predicted based on the adjusted text features and adjusted image features. The location prediction data can be understood as the predicted bounding box corresponding to the labeled object in the image sample. Specifically, based on the adjusted text features and adjusted image features, the predicted bounding box corresponding to the adjusted text features can be determined from the image sample, and then the labeled object to which each predicted bounding box belongs can be determined based on the similarity between the adjusted text features and the text features, thereby obtaining the location prediction data of each labeled object.

[0167] It is understandable that when the number of labeled objects is greater than 1, the predicted bounding box corresponding to each labeled object in the image sample can be predicted based on the adjusted image features, thereby obtaining the position prediction data corresponding to each of the multiple labeled objects.

[0168] 106. Based on the adjusted text features, text features, location prediction data, and location annotation data, the preset image detection model is trained to obtain the image detection model.

[0169] The adjusted text features and location prediction data are obtained from the predictions of the preset image detection model based on image samples. The text features and location annotation data are the labeled data of the image samples. Based on this, the predicted data and the labeled data are compared. The prediction loss of the preset image detection model can be determined according to the comparison results. Then, the preset image detection model is trained according to the prediction loss to obtain the image detection model.

[0170] Specifically, the image detection model is trained based on the adjusted text features, text features, location prediction data, and location annotation data to obtain the image detection model. This process may include: comparing the adjusted text features with the text features and determining the text feature loss based on the comparison results; comparing the location prediction data with the location annotation data and determining the location prediction loss based on the comparison results; and converging the image detection model based on the text feature loss and the location prediction loss to obtain the image detection model.

[0171] The prediction loss of the pre-defined image detection model consists of two parts: text feature loss and location prediction loss. Text feature loss indicates the difference between the adjusted text features and the actual text features. It can be understood as the degree of distortion of the text features after multiple feature fusions by the pre-defined image detection model. A smaller text feature loss indicates that the adjusted text features more accurately reflect the semantics of the text features; conversely, a larger text feature loss indicates a greater deviation between the adjusted text features obtained after multiple feature fusions and the actual text features, thus failing to accurately reflect the semantics of the text features. Location prediction loss can be understood as the difference between the predicted bounding box and the labeled bounding box for the same labeled object.

[0172] In this embodiment of the application, the adjusted text features can be compared with the text features, and the text feature loss can be determined based on the comparison results.

[0173] In this embodiment, the predicted location data can be compared with the labeled location data, and the location prediction loss can be determined based on the comparison result. During the comparison, the predicted bounding box corresponding to the predicted location data can be compared with the labeled bounding box corresponding to the labeled location data. The intersection-union ratio (IUU) between the two can be calculated, and the location prediction loss can be determined based on the calculation result.

[0174] After obtaining the text feature loss and the location prediction loss, the text feature loss and the location prediction loss can be fused to obtain the prediction loss of the preset image detection model. The model parameters of the preset image detection model are then updated in reverse according to the prediction loss until the preset image detection model converges, and finally the image detection model is obtained.

[0175] It is understandable that since the preset image detection model includes multiple networks (first text state space network, first image state space network, second text state space network and second image state space network), and each network contains its own state space parameters, the state space parameters of each of the above networks can be adjusted simultaneously during the training of the preset image detection model until the entire preset image detection model converges to obtain the image detection model.

[0176] As described above, this embodiment of the application obtains image samples and corresponding text description data, where each image sample includes location annotation data corresponding to at least one labeled object. Then, using a preset image detection model, it determines the text hidden state features and text output mapping features of the text description data in the state space. Next, based on the text hidden state features, it determines the image hidden state features and image output mapping features of the image samples in the state space. Then, based on the image hidden state features, it determines the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space, and based on the adjusted text hidden state features, it determines the adjusted image features of the image output mapping features in the state space. Then, based on the adjusted text features and adjusted image features, it predicts the location prediction data of the labeled object. Finally, based on the adjusted text features, text features, location prediction data, and location annotation data, it trains the preset image detection model to obtain the image detection model. This scheme utilizes a pre-defined image detection model to map textual features of textual description data into the state space to obtain textual hidden state features, and to map image features of image samples into the state space to obtain image hidden state features. These textual and image hidden state features serve as intermediaries for feature fusion between different modalities (textual and image features), sequentially achieving three-stage feature fusion: from text to image, from image to text, and then from text to image. The resulting adjusted textual and image features not only improve the ability to capture the global receptive field during feature fusion but also enhance the fine-grainedness of feature fusion. Furthermore, since feature fusion in the state space has linear computational complexity, it significantly reduces the computational load of the model and improves its prediction efficiency. Subsequently, the pre-defined image detection model uses the adjusted textual and image features obtained from the three-stage feature fusion to predict the target object and its location. The model is then trained based on the prediction results, enabling it to possess excellent open-vocabulary detection capabilities.

[0177] This application also provides an image detection method, apparatus, electronic device, and computer-readable storage medium. The image detection model can be deployed on the image detection apparatus, and the image detection apparatus can be integrated into an electronic device, which can be a server, a terminal, or other similar device.

[0178] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud-preset databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN) acceleration services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these.

[0179] Figure 11 A schematic diagram illustrating an application scenario of the image detection method provided in this application embodiment is shown. For example... Figure 11 As shown, taking an image detection device integrated into an electronic device, with the electronic device serving as a server, as an example, the image detection model can be deployed on the server. The server can then acquire the image to be detected and the target description text corresponding to the target object in the image. Using the image detection model, the hidden text features and output mapping features of the target description text in the state space are determined. Based on the hidden text features, the hidden image features and output mapping features of the image to be detected in the state space are determined. Based on the hidden image features, the adjusted text features and adjusted hidden text features of the output mapping features in the state space are determined, and based on the adjusted hidden text features, the adjusted image features of the output mapping features in the state space are determined. Based on the adjusted text features and adjusted image features, the target location corresponding to the target object is detected from the image to be detected.

[0180] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0181] This embodiment will be described from the perspective of an image detection device, which can be integrated into an electronic device, such as a server.

[0182] An image detection method includes: acquiring an image to be detected and target description text corresponding to a target object in the image to be detected; using an image detection model to determine the text hidden state features and text output mapping features of the target text features in the state space; based on the text hidden state features, determining the image hidden state features and image output mapping features of the image to be detected in the state space; based on the image hidden state features, determining the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space, and based on the adjusted text hidden state features, determining the adjusted image features of the image output mapping features in the state space; and based on the adjusted text features and adjusted image features, detecting the target location corresponding to the target object in the image to be detected.

[0183] Figure 12 A schematic flowchart of the image detection method provided in an embodiment of this application is shown. Figure 12 As shown, the specific process of this image detection method is as follows:

[0184] 201. Obtain the image to be detected and the target description text corresponding to the target object in the image to be detected.

[0185] In this embodiment of the application, during the application stage of the image detection model, the image to be detected and the target description text corresponding to the target detection object in the image to be detected can be input into the image detection model so that the image detection model can perform image detection based on the input data and determine the target position of the target detection object in the image to be detected.

[0186] The image to be detected can be any image that requires target detection in any scenario. For example, the image to be detected can be a product image in an industrial AI quality inspection scenario, or a photo or video frame image in a face recognition scenario, and so on.

[0187] The target detection object of an image to be inspected can be understood as the recognition target of the image to be inspected, or it can be understood as the target object that needs to be identified in the image to be inspected. For example, in an industrial AI quality inspection scenario, for a product image, it is necessary to identify certain defects present in it; these defects are the target detection objects of the image to be inspected.

[0188] The target description text corresponding to the target detection object can be understood as data generated by textually describing the target detection object, that is, the textual expression of the target detection object. In this embodiment, the target description text can be used to instruct the image detection device to identify the detection target (i.e., the target detection object). For example, if the target description text is: "Please detect the metal crack in this image," then the target detection object of the image (the image to be detected) can be determined from the target description text as a metal crack. As another example, if the target description text is: "Please detect the animal in this image," then the target detection object of the image (the image to be detected) can be determined from the target description text as an animal.

[0189] It is understandable that the length, vocabulary type, or content of the target description text can be adjusted according to the actual situation. For example, for a target object to be identified in an image to be detected, text of different lengths can be used to describe it, resulting in target description text. A longer target description text may provide a more detailed description of the image to be detected, while a shorter target description text may provide a more concise description. For example, the target description text "Please detect the metal crack in this image" mentioned above can be simplified to "metal crack". Based on this simplified target description text, the target object for this image (the image to be detected) can still be identified as a metal crack.

[0190] 202. Using an image detection model, determine the hidden text features and output text mapping features of the target text in the state space.

[0191] After obtaining the image to be detected and the target description text corresponding to the target object in the image to be detected, the image detection model trained above can be used to perform image detection in order to identify the target location corresponding to the target object in the image to be detected.

[0192] In this embodiment, target text features can be obtained by extracting features from the target description text. Since image detection targets or identifies objects such as people and objects in the image to be detected, and these objects are generally represented by nouns, the target description text can first be segmented into word segments to obtain text vocabulary fragments. Then, target vocabulary fragments of noun type are selected from the text vocabulary fragments, and feature extraction is performed on these target vocabulary fragments to obtain target text features. Through this method, redundant information in the target description text can be removed, resulting in target text features for the objects in the image to be detected.

[0193] In this embodiment, a CLIP (Contrastive Language-Image Pre-Training) text encoder can be used to extract features from the target descriptive text, thereby obtaining the target text features. After determining the target text features of the target descriptive text, an image detection model can be used to convert them into feature representations in the state space. That is, the image detection model is used to determine the text hidden state features and text output mapping features corresponding to the target text features in the state space.

[0194] It can be understood that an image detection model is a network model that can be used to perform open vocabulary detection tasks. The way the image detection model processes the input image to be detected and the target description text during the model application phase can be similar to the way the preset image detection model processes the input image samples and text description data during the model training phase.

[0195] Figure 13 This illustration shows a data processing diagram of the image detection method provided in an embodiment of this application. Figure 14 This illustration shows another data processing diagram of the image detection method provided in an embodiment of this application. For example... Figure 13 and Figure 14 As shown, the image detection model also includes a first text state space network, a first image state space network, a second text state space network, and a second image state space network. It can be understood that since the image detection model is trained on top of a pre-defined image detection model, the first text state space network in the image detection model has the same network architecture and operating method as the first text state space network in the pre-defined image detection model, but the values ​​of the various network parameters may differ. Similarly, the first image state space network, the second text state space network, and the second image state space network in the image detection model are also similar, and will not be elaborated upon here.

[0196] During the model application phase, the first text input mapping parameters and the first text output mapping parameters of the first text state space network can be obtained, and the target text features of the target description text can be adjusted according to the first text input mapping parameters to obtain the text input mapping features; the first text state features of the first text state space network at the current time can be obtained, and the first text state features can be fused with the text input mapping features to obtain the text hidden state features; the text hidden state features can be adjusted according to the first text output mapping parameters to obtain the text output mapping features.

[0197] 203. Based on the text hidden state features, determine the image hidden state features and image output mapping features of the image to be detected in the state space.

[0198] In this embodiment, feature extraction can be performed on the image to be detected to obtain the image features. Specifically, the backbone network of the YOLO (You Only Look Once) series of models can be used to extract features from the image to be detected, and the obtained image features are multi-scale image features.

[0199] After determining the hidden text state features of the target text in the state space using the first text state space network, these hidden text state features can be used as an intermediary for the first feature fusion between different modal features (target text features and image features to be detected). Specifically, based on the hidden text state features, the hidden image state features and image output mapping features of the image features to be detected in the state space can be determined.

[0200] Specifically, determining the image hidden state features and image output mapping features of the image to be detected in the state space based on the text hidden state features may include: using a first image state space network to segment the image features to be detected to obtain first image features and second image features; determining the image hidden state features and first image output mapping features of the first image features in the state space based on the text hidden state features; and fusing the first image output mapping features and the second image features to obtain image output mapping features.

[0201] In this embodiment, the features of the image to be detected can be input into a first image state space network. The first image state space network can perform feature segmentation on the features of the image to be detected to obtain two branches, namely the first image features and the second image features. Then, feature processing is performed on one of the branches (the first image features) to obtain the image hidden state features and the first image output mapping features; then, the first image output mapping features and the other branch (the second image features) are fused to obtain the image output mapping features.

[0202] The first image state space network includes a parallel state space sub-network, which is referred to as the parallel first state space sub-network for easy distinction. During the feature processing of one of its branches (first image features), the parallel first state space sub-network can be used to determine the image hidden state features and the first image output mapping features of the first image features in the state space based on the text hidden state features.

[0203] In this embodiment of the application, determining the image hidden state features and the first image output mapping features of the first image features in the state space based on the text hidden state features may include: obtaining the first image state space parameters of the first image state space network and the first image state features of the first image state space network at the current time; adjusting the first image state space parameters according to the text hidden state features to obtain the adjusted first image state space parameters; and adjusting the first image features based on the adjusted first image state space parameters and the first image state features to obtain the image hidden state features and the first image output mapping features.

[0204] The first image state space parameters of the first image state space network are the state space parameters in the parallel first state space sub-network. The first image state space parameters may include first image state transition parameters, first image input mapping parameters, and first image output mapping parameters. Based on these, the first image state space parameters are adjusted according to text hidden state features to obtain adjusted first image state space parameters. This adjustment may include: adjusting the first image state transition parameters according to text hidden state features; adjusting the first image input mapping parameters according to text hidden state features; adjusting the first image output mapping parameters according to text hidden state features; and using the adjusted first image state transition parameters, adjusted first image input mapping parameters, and adjusted first image output mapping parameters as the adjusted first image state space parameters.

[0205] Specifically, adjusting the first image features based on the adjusted first image state space parameters and first image state features to obtain image hidden state features and first image output mapping features may include: adjusting the first image state features according to the adjusted first image state transition parameters to obtain adjusted first image state features; adjusting the first image features according to the adjusted first image input mapping parameters to obtain adjusted first image features; fusing the adjusted first image state features and adjusted first image features to obtain image hidden state features; and determining the first image output mapping features based on the adjusted first image output mapping parameters and image hidden state features.

[0206] In step 203, the text hidden state features are introduced into the first image state space network and injected into the first image state space parameters of the first image state space network to adjust the first image state space parameters. This adjusts the first image state space parameters to incorporate the relevant features of the text description data. Then, the relevant features of the text description data are passed to the image hidden state features and image output mapping features through the relevant calculation rules of the subsequent state space model to complete the first feature fusion. This first feature fusion is a feature fusion from text to image, which can improve the alignment ability of text and image from the perspective of the global receptive field, laying the foundation for the image detection model to complete subsequent prediction tasks. Furthermore, the state space model is used as the calculation basis for fusion in the first feature fusion process. Since the state space model has linear complexity, its O(N+1) complexity is significantly lower than the unavoidable quadratic complexity in feature fusion in related techniques (such as Transformer), thus significantly reducing the computational load of the model and improving its prediction efficiency.

[0207] Furthermore, as mentioned earlier, the features of the image to be detected obtained after feature extraction can be multi-scale features. Based on this, it can be... Figure 14 The data processing method shown determines the image hidden state features and image output mapping features of the multi-scale image features to be detected in the state space based on the text hidden state features. Specific steps can be found in the processing method described in the aforementioned model training phase; these will not be repeated in detail in this embodiment.

[0208] 204. Based on the image hidden state features, determine the adjusted text features and adjusted text hidden state features in the state space of the text output mapping features, and based on the adjusted text hidden state features, determine the adjusted image features in the state space of the image output mapping features.

[0209] In this embodiment of the application, the process of determining the adjusted text features and the adjusted text hidden state features in the state space based on the image hidden state features can be understood as follows: using the image hidden state features as an intermediary for the second feature fusion between different modal features (target text features and image features to be detected) to achieve the second feature fusion, which is the feature fusion from image to text.

[0210] Following this, the process of determining the adjusted image features in the state space based on the adjusted text hidden state features can be understood as: using the adjusted text hidden state features as an intermediary for the third feature fusion between different modal features (target text features and image features to be detected), thereby achieving the third feature fusion from text to image.

[0211] like Figure 13 and Figure 14 As shown, the image detection model can also include a second text state space network. The second text state space network can be understood as a state space model that aggregates image to text features. Therefore, the second text state space network possesses various matrices of a state space model (such as state transition matrices, input mapping matrices, and output mapping matrices), and features can also be processed according to state equations and output equations within the second text state space network. Furthermore, the second text state space network is used to implement a second feature fusion.

[0212] Based on this, according to the image hidden state features, the adjusted text features and adjusted text hidden state features in the state space of the text output mapping features can be determined, which may include: obtaining the second text state space parameters of the second text state space network; adjusting the second text state space parameters according to the image hidden state features to obtain the adjusted second text state space parameters; and adjusting the text output mapping features according to the adjusted second text state space parameters to obtain the adjusted text features and adjusted text hidden state features.

[0213] In this embodiment, the adjusted second text state space parameters include adjusted second text state transition parameters, adjusted second text input mapping parameters, and adjusted second text output mapping parameters. Based on these, the text output mapping features are adjusted according to the adjusted second text state space parameters to obtain adjusted text features and adjusted text hidden state features. This can include: adjusting the text output mapping features according to the adjusted second text input mapping parameters to obtain adjusted text output mapping features; obtaining the second text state features of the second text state space network at the current time, and adjusting the second text state features according to the adjusted second text state transition parameters to obtain adjusted second text state features; fusing the adjusted text output mapping features and the adjusted second text state features to obtain adjusted text hidden state features; and determining the adjusted text features based on the adjusted second text output mapping parameters and the adjusted text hidden state features.

[0214] In this embodiment, the obtained text output mapping features are used as the input signal of the second text state space network. Following the calculation principle of the state equation, and based on the image hidden state features, the adjusted text hidden state features of the text output mapping features in the state space are determined. Then, following the calculation principle of the output equation, and based on the image hidden state features, the adjusted text features of the text output mapping features in the state space are determined. Through these operations, prior knowledge of the image hidden state features previously fused with text features can be injected into the text output mapping features. This allows the prior knowledge to be compressed into the adjusted text hidden state features according to the state equation of the state space model. This is repeated, and the prior knowledge is further compressed into the adjusted text features according to the output equation of the state space model, achieving a second feature fusion and further improving the text-image alignment capability from the perspective of the global receptive field.

[0215] like Figure 13 and Figure 14 As shown, the image detection model can also include a second image state space network. The second image state space network can be understood as a state space model that aggregates text-to-image features. The second image state space network has a similar network structure to the first image state space network, but the input signals and parameter values ​​in the matrices differ. The second image state space network is used to implement the third feature fusion, which is the fusion from text to image.

[0216] Based on this, the adjusted image features of the image output mapping features in the state space are determined according to the adjusted text hidden state features. This can include: using a second image state space network to perform feature segmentation on the image output mapping features to obtain a first image output feature and a second image output feature; adjusting the first image output feature according to the adjusted text hidden state features to obtain the adjusted image output mapping features of the first image output feature in the state space; and fusing the adjusted image output mapping features with the second image output features to obtain the adjusted image features.

[0217] Specifically, adjusting the first image output features based on the adjusted text hidden state features to obtain the adjusted image output mapping features in the state space can include: obtaining the second image state space parameters of the second image state space network and the second image state features of the second image state space network at the current time; adjusting the second image state space parameters based on the adjusted text hidden state features to obtain the adjusted second image state space parameters; and adjusting the first image output features based on the adjusted second image state space parameters and the second image state features to obtain the adjusted image output mapping features.

[0218] The second image state space parameters may include second image state transition parameters, second image input mapping parameters, and second image output mapping parameters. Based on these, the second image state space parameters are adjusted according to the adjusted text hidden state features to obtain adjusted second image state space parameters. This adjustment may include: adjusting the second image state transition parameters according to the adjusted text hidden state features; adjusting the second image input mapping parameters according to the adjusted text hidden state features; adjusting the second image output mapping parameters according to the adjusted text hidden state features; and using the adjusted second image state transition parameters, adjusted second image input mapping parameters, and adjusted second image output mapping parameters as the adjusted second image state space parameters.

[0219] Furthermore, based on the adjusted state space parameters of the second image and the state features of the second image, the output features of the first image are adjusted to obtain the adjusted image output mapping features. This can include: adjusting the state features of the second image according to the adjusted state transition parameters of the second image to obtain the adjusted second image state features; adjusting the output features of the first image according to the adjusted input mapping parameters of the second image to obtain the adjusted first image output features; fusing the adjusted second image state features and the adjusted first image output features to obtain the adjusted image hidden state features; and adjusting the hidden state features of the adjusted image according to the adjusted second image output mapping parameters to obtain the adjusted image output mapping features of the first image output features in the state space.

[0220] In step 204, the adjusted text hidden state features are introduced into the second image state space network and injected into the second image state space parameters of the network. This adjusts the second image state space parameters, integrating the relevant features of the text description data and image samples. The relevant features of the target description text and the image to be detected are then passed to the adjusted image output mapping features and the adjusted image hidden state features through the relevant calculation rules of the subsequent state space model. This completes the third feature fusion, further improving the text-image alignment capability from the perspective of the global receptive field, laying a solid foundation for the image detection model to complete subsequent prediction tasks. Furthermore, the state space model is also used as the calculation basis for the fusion in the third feature fusion process. Because the state space model has linear complexity, its O(N+1) complexity is significantly lower than the unavoidable quadratic complexity in feature fusion in related techniques (such as Transformer), thus significantly reducing the computational load and improving the prediction efficiency of the model.

[0221] As mentioned earlier, the features of the image to be detected obtained after feature extraction can be multi-scale features. Based on this, such as... Figure 14 As shown, for image output mapping features of different scales output by the first image state space network, the adjusted image hidden state features and adjusted image output mapping features at that scale can be determined in the state space based on the adjusted text hidden state features. Specific steps can be referred to the processing method in the aforementioned model training stage, and will not be repeated in detail in this embodiment.

[0222] It should be noted that, Figure 14 The multi-scale image features to be detected can include multiple image features at different scales. The dimensionality of the feature maps can be transformed through feature upsampling and downsampling. For ease of explanation, let's take an example where the multi-scale image features to be detected include feature maps at three different scales. These three feature maps are named, in ascending order of feature dimension, the first-scale feature map, the second-scale feature map, and the third-scale feature map. The first-scale feature map is upsampled by a factor of 2 to obtain the second-scale feature map, and the second-scale feature map is upsampled by a factor of 2 to obtain the third-scale feature map. Correspondingly, the third-scale feature map is downsampled by a factor of 2 to obtain the second-scale feature map, and the second-scale feature map is downsampled by a factor of 2 to obtain the first-scale feature map.

[0223] In image detection, different state space networks within the image detection model can be used for feature processing on feature maps of different scales. For example, a second image state space network can be used for feature processing on a first-scale feature map; a first and second image state space networks can be used for feature processing on a second-scale feature map; and a first image state space network can be used for feature processing on a third-scale feature map. The processed feature maps of different scales generated during the above feature processing can be upsampled or downsampled based on the dimensional relationships between them, and then further processed according to... Figure 14 The data flow shown is processed to obtain the adjusted multi-scale image features.

[0224] 205. Based on the adjusted text features and adjusted image features, detect the target location corresponding to the target object in the image to be detected.

[0225] In this embodiment of the application, the target location corresponding to the target detection object can be represented in the image to be detected by a bounding box. Figure 15 This illustration shows a schematic diagram illustrating the target locations of some defects detected by an image detection model in industrial artificial intelligence quality inspection, as provided in an embodiment of this application. For example... Figure 15 As shown, the image detection model provided in this application embodiment can accurately detect the target location corresponding to each defect (the target detection object defined according to actual needs in industrial quality inspection).

[0226] In the process of detecting the target location corresponding to the target object from the image to be detected based on the adjusted text features and adjusted image features, feature alignment between the adjusted text features and adjusted image features is required. For example, when the target description text includes multiple target objects, multiple predicted bounding boxes may be obtained in the image to be detected. For each predicted bounding box, it is necessary to match the corresponding target object to achieve a one-to-one correspondence between the target object and the predicted bounding box.

[0227] Therefore, detecting the target location corresponding to the target object in the image to be detected based on the adjusted text features and the adjusted image features may include: identifying candidate locations corresponding to the target object in the image to be detected based on the adjusted image features; filtering object features corresponding to the candidate locations from the adjusted image features; comparing the object features with the adjusted text features; and identifying the target location corresponding to the target object from the candidate locations based on the comparison results.

[0228] Candidate locations can be understood as at least one predicted bounding box obtained in the image to be detected based on the adjusted image features. Each predicted bounding box in the image to be detected corresponds to a region feature. Therefore, after determining the predicted bounding boxes, the region features corresponding to each predicted bounding box can be extracted from the adjusted image features as the object features corresponding to that candidate location. At this point, the object features corresponding to each candidate location can be obtained. Then, each object feature can be compared with the adjusted text features, and based on the feature similarity between each object feature and the adjusted text features, it can be determined whether each candidate location is the target location corresponding to the target object. If the feature similarity between the object feature and the adjusted text features is greater than a preset similarity threshold, the candidate location corresponding to that object feature can be determined as the target location corresponding to the target object; conversely, if the feature similarity between the object feature and the adjusted text features is less than or equal to the preset similarity threshold, the candidate location corresponding to that object feature can be deleted.

[0229] As described above, this embodiment of the application acquires the image to be detected and the target description text corresponding to the target detection object in the image to be detected; uses an image detection model to determine the text hidden state features and text output mapping features of the target text features in the state space; based on the text hidden state features, determines the image hidden state features and image output mapping features of the image to be detected in the state space; based on the image hidden state features, determines the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space, and based on the adjusted text hidden state features, determines the adjusted image features of the image output mapping features in the state space; and based on the adjusted text features and adjusted image features, detects the target position corresponding to the target detection object from the image to be detected. This scheme utilizes an image detection model to map the target text features of the target description text into the state space to obtain text hidden state features, and to map the image features of the image to be detected into the state space to obtain image hidden state features. Thus, the text hidden state features and image hidden state features serve as intermediaries for feature fusion between different modal features (text features and image features), sequentially achieving three-stage feature fusion from text to image, from image to text, and then from text to image, resulting in adjusted text features and adjusted image features. Subsequently, the image detection model uses the adjusted text features and adjusted image features obtained from the three-stage feature fusion to predict the target location corresponding to the target object. This process not only improves the ability to capture the global receptive field during feature fusion but also enhances the fine-grainedness of feature fusion, improving the prediction accuracy of the image detection model in open vocabulary detection tasks. Furthermore, since feature fusion in the state space has linear complexity, it significantly reduces the computational load of the image detection model and improves its prediction efficiency.

[0230] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0231] In this embodiment, the image detection device will be specifically integrated into an electronic device, which will be a server, as an example for explanation.

[0232] Figure 16 Another flowchart illustrating the image detection method provided in this application is shown. Figure 16 As shown, an image detection method has the following specific process:

[0233] 301. The server obtains the image to be detected and the target description text corresponding to the target detection object in the image to be detected.

[0234] For example, the image to be detected and the target description text corresponding to the target object to be detected for the image to be detected can be pre-stored in a database. The server can directly call the database to obtain the image to be detected and the target description text corresponding to the target object to be detected for the image to be detected.

[0235] 302. The server uses an image detection model to determine the target text features in the state space, including the hidden text state features and the text output mapping features.

[0236] For example, an image detection model may include a first text state space network. The server can obtain the first text input mapping parameters and the first text output mapping parameters of the first text state space network, and adjust the target text features according to the first text input mapping parameters to obtain text input mapping features; obtain the first text state features of the first text state space network at the current time, and fuse the first text state features with the text input mapping features to obtain text hidden state features; adjust the text hidden state features according to the first text output mapping parameters to obtain text output mapping features.

[0237] 303. Based on the text hidden state features, the server determines the image hidden state features and image output mapping features of the image to be detected in the state space.

[0238] For example, the image detection model may also include a first image state space network. The server may use the first image state space network to perform feature segmentation on the features of the image to be detected, obtaining first image features and second image features; based on the text hidden state features, determine the image hidden state features and first image output mapping features of the first image features in the state space; and fuse the first image output mapping features and the second image features to obtain the image output mapping features.

[0239] For example, the server can obtain the first image state space parameters of the first image state space network and the first image state features of the first image state space network at the current time; adjust the first image state space parameters according to the text hiding state features to obtain the adjusted first image state space parameters; and adjust the first image features based on the adjusted first image state space parameters and the first image state features to obtain the image hiding state features and the first image output mapping features.

[0240] For example, the first image state space parameters may include first image state transition parameters, first image input mapping parameters, and first image output mapping parameters. The server can adjust the first image state transition parameters according to the text hidden state features to obtain adjusted first image state transition parameters; adjust the first image input mapping parameters according to the text hidden state features to obtain adjusted first image input mapping parameters; adjust the first image output mapping parameters according to the text hidden state features to obtain adjusted first image output mapping parameters; and use the adjusted first image state transition parameters, adjusted first image input mapping parameters, and adjusted first image output mapping parameters as the adjusted first image state space parameters.

[0241] For example, the server can adjust the state features of the first image according to the adjusted state transition parameters of the first image to obtain the adjusted state features of the first image; adjust the features of the first image according to the adjusted input mapping parameters of the first image to obtain the adjusted features of the first image; fuse the adjusted state features of the first image and the adjusted features of the first image to obtain the hidden state features of the image; and determine the output mapping features of the first image based on the adjusted output mapping parameters of the first image and the hidden state features of the image.

[0242] 304. Based on the image hidden state features, the server determines the adjusted text features and adjusted text hidden state features in the state space of the text output mapping features.

[0243] For example, the image detection model may also include a second text state space network. The server can obtain the second text state space parameters of the second text state space network; adjust the second text state space parameters according to the image hidden state features to obtain the adjusted second text state space parameters; adjust the text output mapping features according to the adjusted second text state space parameters to obtain the adjusted text features and the adjusted text hidden state features.

[0244] For example, the adjusted second text state space parameters may include adjusted second text state transition parameters, adjusted second text input mapping parameters, and adjusted second text output mapping parameters. The server can adjust the text output mapping features according to the adjusted second text input mapping parameters to obtain adjusted text output mapping features; obtain the second text state features of the second text state space network at the current time, and adjust the second text state features according to the adjusted second text state transition parameters to obtain adjusted second text state features; fuse the adjusted text output mapping features with the adjusted second text state features to obtain adjusted text hidden state features; and determine the adjusted text features based on the adjusted second text output mapping parameters and the adjusted text hidden state features.

[0245] 305. Based on the adjusted text hidden state features, the server determines the adjusted image features in the state space of the image output mapping features.

[0246] For example, the image detection model may also include a second image state space network. The server may use the second image state space network to perform feature segmentation on the image output mapping features to obtain first image output features and second image output features; based on the adjusted text hidden state features, the first image output features are adjusted to obtain the adjusted image output mapping features of the first image output features in the state space; the adjusted image output mapping features are fused with the second image output features to obtain the adjusted image features.

[0247] For example, the server can obtain the second image state space parameters of the second image state space network and the second image state features of the second image state space network at the current time; adjust the second image state space parameters according to the adjusted text hiding state features to obtain the adjusted second image state space parameters; and adjust the first image output features based on the adjusted second image state space parameters and the second image state features to obtain the adjusted image output mapping features.

[0248] For example, the second image state space parameters may include second image state transition parameters, second image input mapping parameters, and second image output mapping parameters. The server can adjust the second image state transition parameters according to the adjusted text hidden state features to obtain adjusted second image state transition parameters; adjust the second image input mapping parameters according to the adjusted text hidden state features to obtain adjusted second image input mapping parameters; adjust the second image output mapping parameters according to the adjusted text hidden state features to obtain adjusted second image output mapping parameters; and use the adjusted second image state transition parameters, adjusted second image input mapping parameters, and adjusted second image output mapping parameters as the adjusted second image state space parameters.

[0249] For example, the server can adjust the state features of the second image according to the adjusted state transition parameters of the second image to obtain the adjusted state features of the second image; adjust the output features of the first image according to the adjusted input mapping parameters of the second image to obtain the adjusted output features of the first image; fuse the adjusted state features of the second image and the adjusted output features of the first image to obtain the adjusted hidden state features of the image; and adjust the adjusted hidden state features of the image according to the adjusted output mapping parameters of the second image to obtain the adjusted output mapping features of the first image in the state space.

[0250] 306. The server detects the target location corresponding to the target object from the image to be detected based on the adjusted text features and adjusted image features.

[0251] For example, the server can identify candidate locations corresponding to the target object in the image to be detected based on the adjusted image features; filter out the object features corresponding to the candidate locations from the adjusted image features; compare the object features with the adjusted text features; and identify the target location corresponding to the target object from the candidate locations based on the comparison results.

[0252] As described above, this embodiment of the application obtains the image to be detected and the target description text corresponding to the target detection object in the image to be detected through a server. Then, the server uses an image detection model to determine the text hidden state features and text output mapping features of the target text features in the state space. The server then determines the image hidden state features and image output mapping features of the image to be detected in the state space based on the text hidden state features. After that, the server determines the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space based on the image hidden state features. Then, the server determines the adjusted image features of the image output mapping features in the state space based on the adjusted text hidden state features. Finally, the server detects the target position corresponding to the target detection object from the image to be detected based on the adjusted text features and adjusted image features. This scheme utilizes an image detection model to map the target text features of the target description text into the state space to obtain text hidden state features, and to map the image features of the image to be detected into the state space to obtain image hidden state features. Thus, the text hidden state features and image hidden state features serve as intermediaries for feature fusion between different modal features (text features and image features), sequentially achieving three-stage feature fusion from text to image, from image to text, and then from text to image, resulting in adjusted text features and adjusted image features. Subsequently, the image detection model uses the adjusted text features and adjusted image features obtained from the three-stage feature fusion to predict the target location corresponding to the target object. This process not only improves the ability to capture the global receptive field during feature fusion but also enhances the fine-grainedness of feature fusion, improving the prediction accuracy of the image detection model in open vocabulary detection tasks. Furthermore, since feature fusion in the state space has linear complexity, it significantly reduces the computational load of the image detection model and improves its prediction efficiency.

[0253] To better implement the above methods, this application also provides an image detection device, which can be integrated into a network device, such as a server or terminal. The terminal may include a tablet computer, a laptop computer, and / or a personal computer.

[0254] Figure 17A schematic diagram of the structure of the image detection apparatus provided in an embodiment of this application is shown. Figure 17 As shown, the image detection device may include a first acquisition unit 401, a first text feature determination unit 402, a first image feature determination unit 403, a first adjustment unit 404, and a detection unit 405, as follows:

[0255] (1) First acquisition unit 401;

[0256] The first acquisition unit 401 is used to acquire the image to be detected and the target description text corresponding to the target detection object in the image to be detected.

[0257] (2) First text feature determination unit 402;

[0258] The first text feature determination unit 402 is used to determine the text hidden state features and text output mapping features of the target text in the state space by using an image detection model.

[0259] For example, an image detection model may include a first text state space network. The first text feature determination unit 402 can specifically be used to obtain the first text input mapping parameters and the first text output mapping parameters of the first text state space network, and adjust the target text features according to the first text input mapping parameters to obtain text input mapping features; obtain the first text state features of the first text state space network at the current time, and fuse the first text state features with the text input mapping features to obtain text hidden state features; and adjust the text hidden state features according to the first text output mapping parameters to obtain text output mapping features.

[0260] (3) First image feature determination unit 403;

[0261] The first image feature determination unit 403 is used to determine the image hidden state features and image output mapping features of the image to be detected in the state space based on the text hidden state features.

[0262] For example, the image detection model also includes a first image state space network. The first image feature determination unit 403 can be specifically used to perform feature segmentation on the image features to be detected using the first image state space network to obtain first image features and second image features; determine the image hidden state features and first image output mapping features in the state space based on the text hidden state features; and fuse the first image output mapping features and the second image features to obtain the image output mapping features.

[0263] For example, the first image feature determination unit 403 can be specifically used to obtain the first image state space parameters of the first image state space network and the first image state features of the first image state space network at the current time; adjust the first image state space parameters according to the text hiding state features to obtain the adjusted first image state space parameters; and adjust the first image features based on the adjusted first image state space parameters and the first image state features to obtain the image hiding state features and the first image output mapping features.

[0264] For example, the first image state space parameters include first image state transition parameters, first image input mapping parameters, and first image output mapping parameters. The first image feature determination unit 403 can specifically be used to adjust the first image state transition parameters according to text hidden state features to obtain adjusted first image state transition parameters; adjust the first image input mapping parameters according to text hidden state features to obtain adjusted first image input mapping parameters; adjust the first image output mapping parameters according to text hidden state features to obtain adjusted first image output mapping parameters; and use the adjusted first image state transition parameters, adjusted first image input mapping parameters, and adjusted first image output mapping parameters as adjusted first image state space parameters.

[0265] For example, the first image feature determination unit 403 can be specifically used to adjust the first image state features according to the adjusted first image state transition parameters to obtain the adjusted first image state features; adjust the first image features according to the adjusted first image input mapping parameters to obtain the adjusted first image features; fuse the adjusted first image state features and the adjusted first image features to obtain the image hidden state features; and determine the first image output mapping features based on the adjusted first image output mapping parameters and the image hidden state features.

[0266] (4) First adjustment unit 404;

[0267] The first adjustment unit 404 is used to determine the adjusted text features and adjusted text hidden state features in the state space based on the image hidden state features, and to determine the adjusted image features in the state space based on the adjusted text hidden state features.

[0268] For example, the image detection model may also include a second text state space network. The first adjustment unit 404 is specifically used to obtain the second text state space parameters of the second text state space network; adjust the second text state space parameters according to the image hidden state features to obtain the adjusted second text state space parameters; and adjust the text output mapping features according to the adjusted second text state space parameters to obtain the adjusted text features and the adjusted text hidden state features.

[0269] For example, the adjusted second text state space parameters may include adjusted second text state transition parameters, adjusted second text input mapping parameters, and adjusted second text output mapping parameters. The first adjustment unit 404 can specifically be used to adjust the text output mapping features according to the adjusted second text input mapping parameters to obtain adjusted text output mapping features; obtain the second text state features of the second text state space network at the current time, and adjust the second text state features according to the adjusted second text state transition parameters to obtain adjusted second text state features; fuse the adjusted text output mapping features with the adjusted second text state features to obtain adjusted text hidden state features; and determine the adjusted text features based on the adjusted second text output mapping parameters and the adjusted text hidden state features.

[0270] For example, the image detection model may also include a second image state space network. The first adjustment unit 404 can specifically be used to perform feature segmentation on the image output mapping features using the second image state space network to obtain first image output features and second image output features; adjust the first image output features according to the adjusted text hidden state features to obtain adjusted image output mapping features of the first image output features in the state space; and fuse the adjusted image output mapping features with the second image output features to obtain adjusted image features.

[0271] For example, the first adjustment unit 404 can be used to obtain the second image state space parameters of the second image state space network and the second image state features of the second image state space network at the current time; adjust the second image state space parameters according to the adjusted text hiding state features to obtain the adjusted second image state space parameters; and adjust the first image output features based on the adjusted second image state space parameters and the second image state features to obtain the adjusted image output mapping features.

[0272] For example, the second image state space parameters may include second image state transition parameters, second image input mapping parameters, and second image output mapping parameters. The first adjustment unit 404 can specifically be used to adjust the second image state transition parameters according to the adjusted text hidden state features to obtain adjusted second image state transition parameters; adjust the second image input mapping parameters according to the adjusted text hidden state features to obtain adjusted second image input mapping parameters; adjust the second image output mapping parameters according to the adjusted text hidden state features to obtain adjusted second image output mapping parameters; and use the adjusted second image state transition parameters, adjusted second image input mapping parameters, and adjusted second image output mapping parameters as the adjusted second image state space parameters.

[0273] For example, the first adjustment unit 404 can be specifically used to adjust the state features of the second image according to the adjusted state transition parameters of the second image to obtain the adjusted state features of the second image; adjust the output features of the first image according to the adjusted input mapping parameters of the second image to obtain the adjusted output features of the first image; fuse the adjusted state features of the second image and the adjusted output features of the first image to obtain the adjusted hidden state features of the image; and adjust the hidden state features of the image according to the adjusted output mapping parameters of the second image to obtain the adjusted output mapping features of the first image in the state space.

[0274] (5) Detection unit 405;

[0275] The detection unit 405 is used to detect the target location corresponding to the target object in the image to be detected based on the adjusted text features and the adjusted image features.

[0276] For example, the detection unit 405 can be used to identify the candidate position corresponding to the target detection object from the image to be detected based on the adjusted image features; filter the object features corresponding to the candidate positions from the adjusted image features; compare the object features with the adjusted text features; and identify the target position corresponding to the target detection object from the candidate positions based on the comparison results.

[0277] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0278] As can be seen from the above, in this embodiment of the application, the first acquisition unit 401 acquires the image to be detected and the target description text corresponding to the target detection object in the image to be detected; then, the first text feature determination unit 402 uses an image detection model to determine the text hidden state features and text output mapping features of the target text in the state space; then, the first image feature determination unit 403 determines the image hidden state features and image output mapping features of the image to be detected in the state space based on the text hidden state features; the first adjustment unit 404 then determines the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space based on the image hidden state features, and determines the adjusted image features of the image output mapping features in the state space based on the adjusted text hidden state features; finally, the detection unit 405 detects the target position corresponding to the target detection object from the image to be detected based on the adjusted text features and adjusted image features. This scheme utilizes an image detection model to map the target text features of the target description text into the state space to obtain text hidden state features, and to map the image features of the image to be detected into the state space to obtain image hidden state features. Thus, the text hidden state features and image hidden state features serve as intermediaries for feature fusion between different modal features (text features and image features), sequentially achieving three-stage feature fusion from text to image, from image to text, and then from text to image, resulting in adjusted text features and adjusted image features. Subsequently, the image detection model uses the adjusted text features and adjusted image features obtained from the three-stage feature fusion to predict the target location corresponding to the target object. This process not only improves the ability to capture the global receptive field during feature fusion but also enhances the fine-grainedness of feature fusion, improving the prediction accuracy of the image detection model in open vocabulary detection tasks. Furthermore, since feature fusion in the state space has linear complexity, it significantly reduces the computational load of the image detection model and improves its prediction efficiency.

[0279] To better implement the above methods, this application also provides a model training device that can be integrated into a network device, such as a server or terminal. The terminal may include a tablet computer, a laptop computer, and / or a personal computer.

[0280] Figure 18 A schematic diagram of the model training apparatus provided in an embodiment of this application is shown. Figure 18 As shown, the model training device may include a second acquisition unit 501, a second text feature determination unit 502, a second image feature determination unit 503, a second adjustment unit 504, a prediction unit 505, and a training unit 506, as follows:

[0281] (1) Second acquisition unit 501;

[0282] The second acquisition unit 501 is used to acquire image samples and text description data corresponding to the image samples. The image samples include location annotation data corresponding to at least one labeled object.

[0283] (2) Second text feature determination unit 502;

[0284] The second text feature determination unit 502 is used to determine the text hidden state features and text output mapping features of the text description data in the state space by using a preset image detection model.

[0285] For example, the second text feature determination unit 502 can be specifically used to obtain the first text input mapping parameters and the first text output mapping parameters of the first text state space network, and adjust the text features according to the first text input mapping parameters to obtain the text input mapping features; obtain the first text state features of the first text state space network at the current time, and fuse the first text state features with the text input mapping features to obtain the text hidden state features; and adjust the text hidden state features according to the first text output mapping parameters to obtain the text output mapping features.

[0286] (3) Second image feature determination unit 503;

[0287] The second image feature determination unit 503 is used to determine the image hidden state features and image output mapping features of the image sample in the state space based on the text hidden state features.

[0288] For example, the second image feature determination unit 503 can be specifically used to perform feature segmentation on image features using a first image state space network to obtain first image features and second image features; determine the image hidden state features and first image output mapping features of the first image features in the state space based on the text hidden state features; and fuse the first image output mapping features and the second image features to obtain image output mapping features.

[0289] For example, the second image feature determination unit 503 can be specifically used to obtain the first image state space parameters of the first image state space network and the first image state features of the first image state space network at the current time; adjust the first image state space parameters according to the text hiding state features to obtain the adjusted first image state space parameters; and adjust the first image features based on the adjusted first image state space parameters and the first image state features to obtain the image hiding state features and the first image output mapping features.

[0290] For example, the second image feature determination unit 503 can be specifically used to adjust the state transition parameters of the first image according to the text hidden state features to obtain the adjusted state transition parameters of the first image; adjust the input mapping parameters of the first image according to the text hidden state features to obtain the adjusted input mapping parameters of the first image; adjust the output mapping parameters of the first image according to the text hidden state features to obtain the adjusted output mapping parameters of the first image; and use the adjusted state transition parameters, the adjusted input mapping parameters, and the adjusted output mapping parameters of the first image as the state space parameters of the adjusted first image.

[0291] For example, the second image feature determination unit 503 can be specifically used to adjust the first image state features according to the adjusted first image state transition parameters to obtain the adjusted first image state features; adjust the first image features according to the adjusted first image input mapping parameters to obtain the adjusted first image features; fuse the adjusted first image state features and the adjusted first image features to obtain the image hidden state features; and determine the first image output mapping features based on the adjusted first image output mapping parameters and the image hidden state features.

[0292] (4) Second adjustment unit 504;

[0293] The second adjustment unit 504 is used to determine the adjusted text features and adjusted text hidden state features in the state space based on the image hidden state features, and to determine the adjusted image features in the state space based on the adjusted text hidden state features.

[0294] For example, the second adjustment unit 504 can be used to obtain the second text state space parameters of the second text state space network; adjust the second text state space parameters according to the image hidden state features to obtain the adjusted second text state space parameters; and adjust the text output mapping features according to the adjusted second text state space parameters to obtain the adjusted text features and the adjusted text hidden state features.

[0295] For example, the second adjustment unit 504 can be specifically used to adjust the text output mapping features according to the adjusted second text input mapping parameters to obtain the adjusted text output mapping features; obtain the second text state features of the second text state space network at the current time, and adjust the second text state features according to the adjusted second text state transition parameters to obtain the adjusted second text state features; fuse the adjusted text output mapping features with the adjusted second text state features to obtain the adjusted text hidden state features; and determine the adjusted text features based on the adjusted second text output mapping parameters and the adjusted text hidden state features.

[0296] For example, the second adjustment unit 504 can be used to perform feature segmentation on the image output mapping features using a second image state space network to obtain a first image output feature and a second image output feature; adjust the first image output feature according to the adjusted text hidden state feature to obtain the adjusted image output mapping feature of the first image output feature in the state space; and fuse the adjusted image output mapping feature with the second image output feature to obtain the adjusted image feature.

[0297] For example, the second adjustment unit 504 can be used to obtain the second image state space parameters of the second image state space network and the second image state features of the second image state space network at the current time; adjust the second image state space parameters according to the adjusted text hiding state features to obtain the adjusted second image state space parameters; and adjust the first image output features based on the adjusted second image state space parameters and the second image state features to obtain the adjusted image output mapping features.

[0298] For example, the second adjustment unit 504 can be specifically used to adjust the state transition parameters of the second image according to the adjusted text hidden state features to obtain the adjusted state transition parameters of the second image; adjust the input mapping parameters of the second image according to the adjusted text hidden state features to obtain the adjusted input mapping parameters of the second image; adjust the output mapping parameters of the second image according to the adjusted text hidden state features to obtain the adjusted output mapping parameters of the second image; and use the adjusted state transition parameters, the adjusted input mapping parameters, and the adjusted output mapping parameters of the second image as the state space parameters of the adjusted second image.

[0299] For example, the second adjustment unit 504 can be specifically used to adjust the state features of the second image according to the adjusted state transition parameters of the second image to obtain the adjusted state features of the second image; adjust the output features of the first image according to the adjusted input mapping parameters of the second image to obtain the adjusted output features of the first image; fuse the adjusted state features of the second image and the adjusted output features of the first image to obtain the adjusted hidden state features of the image; and adjust the hidden state features of the image according to the adjusted output mapping parameters of the second image to obtain the adjusted output mapping features of the first image in the state space.

[0300] (5) Prediction unit 505;

[0301] Prediction unit 505 is used to predict the location prediction data of the labeled object based on the adjusted text features and adjusted image features.

[0302] (6) Training Unit 506;

[0303] Training unit 506 is used to train a preset image detection model based on adjusted text features, text features, location prediction data, and location annotation data to obtain an image detection model.

[0304] For example, training unit 506 can be used to compare the adjusted text features with the text features and determine the text feature loss based on the comparison results; compare the location prediction data with the location annotation data and determine the location prediction loss based on the comparison results; and converge the preset image detection model based on the text feature loss and the location prediction loss to obtain the image detection model.

[0305] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0306] As can be seen from the above, in this embodiment of the application, the second acquisition unit 501 acquires image samples and corresponding text description data, wherein the image sample includes location annotation data corresponding to at least one labeled object; then, the second text feature determination unit 502 uses a preset image detection model to determine the text hidden state features and text output mapping features of the text description data in the state space; then, the second image feature determination unit 503 determines the image hidden state features and image output mapping features of the image sample in the state space based on the text hidden state features; the second adjustment unit 504 then determines the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space based on the image hidden state features, and determines the adjusted image features of the image output mapping features in the state space based on the adjusted text hidden state features; then, the prediction unit 505 predicts the location prediction data of the labeled object based on the adjusted text features and adjusted image features; finally, the training unit 506 trains the preset image detection model based on the adjusted text features, text features, location prediction data, and location annotation data to obtain the image detection model. This scheme utilizes a pre-defined image detection model to map text features of text description data into the state space to obtain text hidden state features, and to map image features of image samples into the state space to obtain image hidden state features. These text and image hidden state features serve as intermediaries for feature fusion between different modalities (text and image features), sequentially achieving three-stage feature fusion: from text to image, from image to text, and then from text to image. The resulting adjusted text and image features not only improve the ability to capture the global receptive field during feature fusion but also enhance the fine-grainedness of feature fusion. Furthermore, since feature fusion in the state space has linear computational complexity, it significantly reduces the computational load of the model and improves its prediction efficiency. Subsequently, the pre-defined image detection model predicts the location of labeled objects based on the adjusted text and image features obtained from the three-stage feature fusion. The model is then trained based on the prediction results, enabling it to possess excellent open-vocabulary detection capabilities.

[0307] This application also provides an electronic device, such as... Figure 19 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0308] The electronic device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a power supply 603, and an input unit 604. Those skilled in the art will understand that... Figure 19The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0309] The processor 601 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes various functions and processes data by running or executing software programs and / or modules stored in the memory 602, and by calling data stored in the memory 602. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 601.

[0310] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0311] The electronic device also includes a power supply 603 that supplies power to the various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 603 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0312] The electronic device may also include an input unit 604, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0313] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 601 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 602 according to the following instructions, and the processor 601 runs the applications stored in the memory 602 to realize various functions, as follows:

[0314] The process involves acquiring the image to be detected and the target description text corresponding to the target object in the image; using an image detection model, determining the text hidden state features and text output mapping features of the target text in the state space; based on the text hidden state features, determining the image hidden state features and image output mapping features of the image to be detected in the state space; based on the image hidden state features, determining the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space, and based on the adjusted text hidden state features, determining the adjusted image features of the image output mapping features in the state space; and based on the adjusted text features and adjusted image features, detecting the target location corresponding to the target object in the image to be detected.

[0315] And / or,

[0316] The process involves acquiring image samples and their corresponding text description data, where each image sample includes location annotation data corresponding to at least one labeled object. A pre-defined image detection model is used to determine the text hidden state features and text output mapping features of the text description data in the state space. Based on the text hidden state features, the image hidden state features and image output mapping features of the image samples in the state space are determined. Based on the image hidden state features, the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space are determined, and based on the adjusted text hidden state features, the adjusted image features of the image output mapping features in the state space are determined. Based on the adjusted text features and adjusted image features, the location prediction data of the labeled object is predicted. The pre-defined image detection model is trained based on the adjusted text features, text features, location prediction data, and location annotation data to obtain the image detection model.

[0317] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0318] As described above, this embodiment of the application acquires the image to be detected and the target description text corresponding to the target detection object in the image to be detected; uses an image detection model to determine the text hidden state features and text output mapping features of the target text features in the state space; based on the text hidden state features, determines the image hidden state features and image output mapping features of the image to be detected in the state space; based on the image hidden state features, determines the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space, and based on the adjusted text hidden state features, determines the adjusted image features of the image output mapping features in the state space; and based on the adjusted text features and adjusted image features, detects the target position corresponding to the target detection object from the image to be detected. This scheme utilizes an image detection model to map the target text features of the target description text into the state space to obtain text hidden state features, and to map the image features of the image to be detected into the state space to obtain image hidden state features. Thus, the text hidden state features and image hidden state features serve as intermediaries for feature fusion between different modal features (text features and image features), sequentially achieving three-stage feature fusion from text to image, from image to text, and then from text to image, resulting in adjusted text features and adjusted image features. Subsequently, the image detection model uses the adjusted text features and adjusted image features obtained from the three-stage feature fusion to predict the target location corresponding to the target object. This process not only improves the ability to capture the global receptive field during feature fusion but also enhances the fine-grainedness of feature fusion, improving the prediction accuracy of the image detection model in open vocabulary detection tasks. Furthermore, since feature fusion in the state space has linear complexity, it significantly reduces the computational load of the image detection model and improves its prediction efficiency.

[0319] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0320] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the model training methods or image detection methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0321] The process involves acquiring the image to be detected and the target description text corresponding to the target object in the image; using an image detection model, determining the text hidden state features and text output mapping features of the target text in the state space; based on the text hidden state features, determining the image hidden state features and image output mapping features of the image to be detected in the state space; based on the image hidden state features, determining the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space, and based on the adjusted text hidden state features, determining the adjusted image features of the image output mapping features in the state space; and based on the adjusted text features and adjusted image features, detecting the target location corresponding to the target object in the image to be detected.

[0322] And / or,

[0323] The process involves acquiring image samples and their corresponding text description data, where each image sample includes location annotation data corresponding to at least one labeled object. A pre-defined image detection model is used to determine the text hidden state features and text output mapping features of the text description data in the state space. Based on the text hidden state features, the image hidden state features and image output mapping features of the image samples in the state space are determined. Based on the image hidden state features, the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space are determined, and based on the adjusted text hidden state features, the adjusted image features of the image output mapping features in the state space are determined. Based on the adjusted text features and adjusted image features, the location prediction data of the labeled object is predicted. The pre-defined image detection model is trained based on the adjusted text features, text features, location prediction data, and location annotation data to obtain the image detection model.

[0324] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0325] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0326] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the model training methods and image detection methods provided in the embodiments of this application, the beneficial effects that any of the model training methods and image detection methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0327] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in various optional implementations of the aforementioned model training method and image detection method.

[0328] The above provides a detailed description of the image detection method, model training method, and related equipment provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An image detection method, characterized in that, include: Acquire the image to be detected, and the target description text corresponding to the target detection object in the image to be detected; An image detection model is used to determine the target text features in the state space, including the hidden text state features and the text output mapping features. Based on the text hidden state features, determine the image hidden state features and image output mapping features of the image to be detected in the state space; Based on the image hidden state features, the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space are determined, and based on the adjusted text hidden state features, the adjusted image features of the image output mapping features in the state space are determined. Based on the adjusted text features and the adjusted image features, the target location corresponding to the target detection object is detected from the image to be detected.

2. The image detection method as described in claim 1, characterized in that, The image detection model includes a first text state space network. The step of employing an image detection model to determine the target text features in the state space, including the text hidden state features and text output mapping features, includes: Obtain the first text input mapping parameters and the first text output mapping parameters of the first text state space network, and adjust the target text features according to the first text input mapping parameters to obtain the text input mapping features; The first text state feature of the first text state space network at the current time is obtained, and the first text state feature is fused with the text input mapping feature to obtain the text hidden state feature; The text hidden state features are adjusted according to the first text output mapping parameters to obtain the text output mapping features.

3. The image detection method as described in claim 1, characterized in that, The image detection model also includes a first image state space network. The step of determining the image hidden state features and image output mapping features of the image to be detected in the state space based on the text hidden state features includes: The first image state space network is used to segment the features of the image to be detected to obtain the first image features and the second image features. Based on the text hiding state features, determine the image hiding state features and the first image output mapping features of the first image features in the state space; The first image output mapping feature and the second image feature are fused together to obtain the image output mapping feature.

4. The image detection method as described in claim 3, characterized in that, The step of determining the image hidden state features and the first image output mapping features of the first image features in the state space based on the text hidden state features includes: Obtain the first image state space parameters of the first image state space network and the first image state features of the first image state space network at the current time. Based on the text hiding state features, the state space parameters of the first image are adjusted to obtain the adjusted state space parameters of the first image. Based on the adjusted first image state space parameters and the first image state features, the first image features are adjusted to obtain the image hidden state features and the first image output mapping features.

5. The image detection method as described in claim 4, characterized in that, The first image state space parameters include first image state transition parameters, first image input mapping parameters, and first image output mapping parameters. The step of adjusting the first image state space parameters according to the text hidden state features to obtain the adjusted first image state space parameters includes: The first image state transition parameters are adjusted based on the text hidden state features to obtain the adjusted first image state transition parameters. The first image input mapping parameters are adjusted based on the text hidden state features to obtain the adjusted first image input mapping parameters; The first image output mapping parameters are adjusted based on the text hidden state features to obtain the adjusted first image output mapping parameters. The adjusted first image state transition parameters, the adjusted first image input mapping parameters, and the adjusted first image output mapping parameters are used as the adjusted first image state space parameters.

6. The image detection method as described in claim 5, characterized in that, The step of adjusting the first image features based on the adjusted first image state space parameters and the first image state features to obtain the image hidden state features and the first image output mapping features includes: The first image state features are adjusted according to the adjusted first image state transition parameters to obtain the adjusted first image state features; The first image features are adjusted according to the adjusted first image input mapping parameters to obtain the adjusted first image features; The adjusted first image state features and the adjusted first image features are fused together to obtain the image hidden state features; Based on the adjusted first image output mapping parameters and the image hidden state features, the first image output mapping features are determined.

7. The image detection method as described in claim 1, characterized in that, The image detection model also includes a second text state space network. The step of determining the adjusted text features and adjusted text hidden state features in the state space based on the image hidden state features includes: Obtain the second text state space parameters of the second text state space network; The second text state space parameters are adjusted based on the image hidden state features to obtain the adjusted second text state space parameters. The text output mapping features are adjusted according to the adjusted second text state space parameters to obtain the adjusted text features and the adjusted text hidden state features.

8. The image detection method as described in claim 7, characterized in that, The adjusted second text state space parameters include adjusted second text state transition parameters, adjusted second text input mapping parameters, and adjusted second text output mapping parameters. The step of adjusting the text output mapping features according to the adjusted second text state space parameters to obtain the adjusted text features and the adjusted text hidden state features includes: The text output mapping features are adjusted according to the adjusted second text input mapping parameters to obtain the adjusted text output mapping features; The second text state features of the second text state space network at the current time are obtained, and the second text state features are adjusted according to the adjusted second text state transition parameters to obtain the adjusted second text state features. The adjusted text output mapping feature is fused with the adjusted second text state feature to obtain the adjusted text hidden state feature; The adjusted text features are determined based on the adjusted second text output mapping parameters and the adjusted text hidden state features.

9. The image detection method as described in claim 1, characterized in that, The image detection model also includes a second image state space network. The step of determining the adjusted image features in the state space based on the adjusted text hidden state features includes: The second image state space network is used to perform feature segmentation on the image output mapping features to obtain the first image output features and the second image output features. Based on the adjusted text hidden state features, the first image output features are adjusted to obtain the adjusted image output mapping features of the first image output features in the state space. The adjusted image output mapping features are fused with the second image output features to obtain the adjusted image features.

10. The image detection method as described in claim 9, characterized in that, The step of adjusting the first image output features based on the adjusted text hidden state features to obtain the adjusted image output mapping features of the first image output features in the state space includes: Obtain the second image state space parameters of the second image state space network and the second image state features of the second image state space network at the current time. Based on the adjusted text hidden state features, the state space parameters of the second image are adjusted to obtain the adjusted state space parameters of the second image. Based on the adjusted second image state space parameters and the second image state features, the first image output features are adjusted to obtain the adjusted image output mapping features.

11. The image detection method as described in claim 10, characterized in that, The second image state space parameters include second image state transition parameters, second image input mapping parameters, and second image output mapping parameters. The step of adjusting the second image state space parameters based on the adjusted text hidden state features to obtain the adjusted second image state space parameters includes: The second image state transition parameters are adjusted based on the adjusted text hidden state features to obtain the adjusted second image state transition parameters. The second image input mapping parameters are adjusted based on the adjusted text hidden state features to obtain the adjusted second image input mapping parameters; The second image output mapping parameters are adjusted based on the adjusted text hidden state features to obtain the adjusted second image output mapping parameters. The adjusted second image state transition parameters, the adjusted second image input mapping parameters, and the adjusted second image output mapping parameters are used as the adjusted second image state space parameters.

12. The image detection method as described in claim 11, characterized in that, The step of adjusting the first image output features based on the adjusted second image state space parameters and the second image state features to obtain the adjusted image output mapping features includes: The second image state features are adjusted according to the adjusted second image state transition parameters to obtain the adjusted second image state features; The first image output features are adjusted according to the adjusted second image input mapping parameters to obtain the adjusted first image output features; The adjusted second image state features and the adjusted first image output features are fused together to obtain the adjusted image hidden state features. The adjusted image hidden state features are adjusted according to the adjusted second image output mapping parameters to obtain the adjusted image output mapping features of the first image output features in the state space.

13. The image detection method as described in claim 1, characterized in that, The step of detecting the target location corresponding to the target object from the image to be detected based on the adjusted text features and the adjusted image features includes: Based on the adjusted image features, candidate locations corresponding to the target detection object are identified from the image to be detected; The object features corresponding to the candidate positions are selected from the adjusted image features; The object features are compared with the adjusted text features, and the target location corresponding to the target detection object is identified from the candidate locations based on the comparison results.

14. A model training method, characterized in that, include: Obtain image samples and corresponding text description data for the image samples, wherein the image samples include location annotation data corresponding to at least one labeled object; A preset image detection model is used to determine the text hidden state features and text output mapping features of the text description data in the state space; Based on the text hidden state features, determine the image hidden state features and image output mapping features of the image sample in the state space; Based on the image hidden state features, the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space are determined, and based on the adjusted text hidden state features, the adjusted image features of the image output mapping features in the state space are determined. Based on the adjusted text features and the adjusted image features, predict the location prediction data of the labeled object; Based on the adjusted text features, the text features, the location prediction data, and the location annotation data, the preset image detection model is trained to obtain the image detection model.

15. The model training method as described in claim 1, characterized in that, The step of training the preset image detection model based on the adjusted text features, the text features, the location prediction data, and the location annotation data to obtain the image detection model includes: The adjusted text features are compared with the original text features, and the text feature loss is determined based on the comparison results; The predicted location data is compared with the labeled location data, and the location prediction loss is determined based on the comparison results. Based on the text feature loss and the location prediction loss, the preset image detection model is converged to obtain the image detection model.

16. An image detection device, characterized in that, include: The first acquisition unit is used to acquire the image to be detected and the target description text corresponding to the target detection object in the image to be detected. The first text feature determination unit is used to use an image detection model to determine the text hidden state features and text output mapping features of the target text in the state space of the target description text. The first image feature determination unit is used to determine the image hidden state features and image output mapping features of the image to be detected in the state space based on the text hidden state features. The first adjustment unit is used to determine the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space based on the image hidden state features, and to determine the adjusted image features of the image output mapping features in the state space based on the adjusted text hidden state features. The detection unit is used to detect the target location corresponding to the target object from the image to be detected based on the adjusted text features and the adjusted image features.

17. A model training device, characterized in that, include: The second acquisition unit is used to acquire image samples and text description data corresponding to the image samples, wherein the image samples include location annotation data corresponding to at least one labeled object; The second text feature determination unit is used to determine the text hidden state features and text output mapping features of the text description data in the state space using a preset image detection model. The second image feature determination unit is used to determine the image hidden state features and image output mapping features of the image sample in the state space based on the text hidden state features. The second adjustment unit is used to determine the adjusted text features and adjusted text hidden state features of the text output mapping features in the state space based on the image hidden state features, and to determine the adjusted image features of the image output mapping features in the state space based on the adjusted text hidden state features. The prediction unit is used to predict the location prediction data of the labeled object based on the adjusted text features and the adjusted image features; The training unit is used to train the preset image detection model based on the adjusted text features, the text features, the location prediction data, and the location annotation data to obtain the image detection model.

18. An electronic device, characterized in that, It includes a processor and a memory, the memory storing an application program, and the processor running the application program in the memory to perform the steps of the image detection method according to any one of claims 1 to 13 and the model training method according to any one of claims 14 to 15.

19. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instructions implement the steps of the image detection method according to any one of claims 1 to 13 and the model training method according to any one of claims 14 to 15.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform steps in the image detection method of any one of claims 1 to 13 and the model training method of any one of claims 14 to 15.

Images