Image detection method, computer and readable storage medium

Abstract

Embodiments of the present application disclose an image detection method, a computer and a readable storage medium. The method comprises: obtaining character fusion features of a first predicted character associated with a target image; obtaining a target attention hidden state for predicting a second predicted character based on the character fusion features, inputting the target attention hidden state into k candidate box detection networks, and sequentially identifying target candidate boxes in the target image based on the target attention hidden state in the k candidate box detection networks; k is a positive integer; obtaining target candidate image features corresponding to the target candidate boxes respectively identified by the k candidate box detection networks from the target image, and generating candidate aggregated image features corresponding to the k target candidate image features; obtaining a global image feature of the target image, adding the global image feature in the candidate aggregated image features to obtain an attention image feature, and predicting a second predicted character corresponding to the attention image feature. By using the present application, the accuracy of image detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image detection method, a computer, and a readable storage medium. Background Art

[0002] Image description generation and localization based on weak supervision are gaining increasing attention. This task involves automatically generating a description sentence for a given image to describe the image's content, while also predicting the corresponding positions of the nouns in the description sentence within the image. Due to the lack of supervised information about nouns and their corresponding positions, weakly supervised image description generation and localization are difficult. Typically, local regions of the image are identified, utilizing discriminative local features to avoid noise interference and improve classification performance. However, this approach makes it impossible to locate the complete target in the image, resulting in lower image detection accuracy. Summary of the Invention

[0003] The embodiments of the present application provide an image detection method, a computer, and a readable storage medium, which can improve the accuracy of image detection.

[0004] On the one hand, an embodiment of the present application provides an image detection method, the method comprising:

[0005] Obtaining a character fusion feature of a first predicted character associated with the target image;

[0006] A target attention hidden state for predicting the second predicted character is obtained based on the character fusion feature, and the target attention hidden state is input into k candidate box detection networks. In the k candidate box detection networks, target candidate boxes in the target image are identified based on the target attention hidden state in turn; k is a positive integer;

[0007] Obtain target candidate image features corresponding to the target candidate frames respectively identified by the k candidate frame detection networks from the target image, and generate candidate aggregate image features corresponding to the k target candidate image features;

[0008] A global image feature of the target image is obtained, the global image feature is added to the candidate aggregate image feature to obtain an attention image feature, and a second predicted character corresponding to the attention image feature is predicted.

[0009] On the one hand, an embodiment of the present application provides an image detection method, the method comprising:

[0010] Obtain an image training sample and obtain an image description sentence corresponding to the image training sample; the image description sentence includes a first character label and a second character label;

[0011] Predicting a first sample character of the image training sample based on the initial character prediction model, and obtaining a sample character fusion feature of the first sample character;

[0012] In an initial attention memory network included in the initial character prediction model, a sample attention latent state for predicting a second sample character is obtained based on the sample character fusion feature, the sample attention latent state is input into k initial candidate box detection networks, and in the k initial candidate box detection networks, target sample candidate boxes in the image training sample are sequentially identified based on the sample attention latent state, where k is a positive integer;

[0013] Obtain target sample candidate image features corresponding to target sample candidate frames respectively identified by k initial candidate frame detection networks from image training samples, and generate sample candidate aggregate image features corresponding to the k target sample candidate image features;

[0014] Obtaining a sample global image feature of an image training sample, adding the sample global image feature to the sample candidate aggregate image feature to obtain a sample attention image feature, and predicting a first sample prediction result corresponding to the sample attention image feature;

[0015] Obtaining a sample language latent state of the sample attention image feature, and predicting a second sample prediction result corresponding to the image training sample based on the sample attention latent state and the sample language latent state;

[0016] According to the first sample prediction result, the second sample prediction result and the second character label, the parameters of the initial character prediction model are adjusted to obtain a target character prediction model.

[0017] In one aspect, an embodiment of the present application provides an image detection device, comprising:

[0018] An input acquisition module, configured to acquire a character fusion feature of a first predicted character associated with a target image;

[0019] An attention detection module is used to obtain a target attention hidden state for predicting a second predicted character based on the character fusion feature;

[0020] Feature transfer module, used to input the target attention hidden state into the k candidate box detection network;

[0021] The candidate box recognition module is used to identify the target candidate box in the target image based on the target attention hidden state in the k candidate box detection networks; k is a positive integer;

[0022] A feature recognition module is used to obtain target candidate image features corresponding to the target candidate boxes respectively identified by the k candidate box detection networks from the target image;

[0023] A candidate aggregation module is used to generate candidate aggregated image features corresponding to k target candidate image features;

[0024] The feature fusion module is used to obtain the global image features of the target image and add the global image features to the candidate aggregated image features to obtain the attention image features;

[0025] The character prediction module is used to predict a second predicted character corresponding to the attention image feature.

[0026] The candidate frame recognition module includes:

[0027] An initial recognition unit is configured to recognize (N-i+1) initial candidate boxes and initial candidate features of each initial candidate box in the target image based on the target attention hidden state in the i-th candidate box detection network; N is a positive integer, and i is a positive integer less than or equal to k; the (N-i+1) initial candidate boxes refer to the initial candidate boxes among the N initial candidate boxes, excluding the target candidate boxes respectively recognized by the (i-1) candidate box detection networks;

[0028] The probability determination unit is used to convert the feature dimensions of the (N-i+1) initial candidate features to obtain the candidate probabilities corresponding to the (N-i+1) initial candidate boxes;

[0029] The candidate box selection unit is used to determine the target candidate box from the (N-i+1) initial candidate boxes based on the candidate probabilities corresponding to the (N-i+1) initial candidate boxes; the target candidate boxes corresponding to the k candidate box detection networks are different from each other.

[0030] The attention detection module includes:

[0031] The attention fusion unit is used to obtain the historical attention latent state used to predict the first predicted character, and perform feature fusion on the historical attention latent state and the character fusion feature based on the attention memory network to obtain the target attention latent state used to predict the second predicted character.

[0032] The candidate aggregation module includes:

[0033] The weight acquisition unit is used to fuse the k target candidate image features with the target attention hidden state to obtain k initial candidate weights;

[0034] A weight determination unit is used to normalize the k initial candidate weights to obtain target candidate weights corresponding to the k target candidate image features;

[0035] The feature weighting unit is used to perform weighted summation on the k target candidate image features based on the target candidate weights corresponding to the k target candidate image features to obtain the candidate aggregate image feature.

[0036] The character prediction module comprises:

[0037] The first prediction unit is configured to perform feature prediction processing on the attention image feature and the target attention hidden state in the language memory network to obtain a first prediction result and a target language hidden state.

[0038] The hidden state fusion unit is configured to perform feature fusion on the target attention hidden state and the target language hidden state to obtain a hidden state fusion feature.

[0039] The second prediction unit is configured to perform prediction on the hidden state fusion feature to obtain a second prediction result.

[0040] The result integration unit is configured to perform result integration processing on the first prediction result and the second prediction result to obtain a second prediction character.

[0041] The hidden state fusion unit comprises:

[0042] The first conversion subunit is configured to perform dimension conversion on the target attention hidden state by using a first fully connected layer to generate a first converted hidden state of a target dimension.

[0043] The second conversion subunit is configured to perform dimension conversion on the target language hidden state by using a second fully connected layer to generate a second converted hidden state of a target dimension.

[0044] The state superposition subunit is configured to perform state superposition processing on the first converted hidden state and the second converted hidden state to obtain the hidden state fusion feature.

[0045] The character prediction module comprises:

[0046] The preliminary prediction unit is configured to obtain a historical language hidden state corresponding to the first prediction character in the language memory network, perform feature fusion on the historical language hidden state and the attention image feature, and predict M candidate characters and a character matching probability corresponding to each candidate character; M is a positive integer.

[0047] The character determination unit is configured to determine a candidate character with the largest character matching probability in the M candidate characters as the second prediction character.

[0048] The input acquisition module comprises:

[0049] The feature acquisition unit is configured to obtain a candidate box global feature corresponding to the target image.

[0050] The embedding acquisition unit is configured to perform feature extraction on the first prediction character associated with the target image to obtain a character embedding feature of the first prediction character.

[0051] The feature determination unit is used to obtain the character fusion feature of the first predicted character based on the global feature of the candidate box and the character embedding feature.

[0052] The feature acquisition unit includes:

[0053] The local recognition subunit is used to obtain N initial candidate boxes corresponding to the target image and identify the candidate local features corresponding to each initial candidate box from the target image; N is a positive integer;

[0054] The local fusion subunit is used to fuse N candidate local features to obtain the global features of the candidate box corresponding to the target image.

[0055] The feature determination unit is specifically configured to:

[0056] Perform feature splicing on the global features of the candidate box and the character embedding features to obtain the character fusion features of the first predicted character; or,

[0057] The historical language latent state of the first predicted character is obtained, and feature splicing is performed on the historical language latent state, the global features of the candidate box, and the character embedding features to obtain the character fusion features of the first predicted character.

[0058] In one aspect, an embodiment of the present application provides an image detection device, comprising:

[0059] A sample acquisition module is used to acquire an image training sample and an image description sentence corresponding to the image training sample; the image description sentence includes a first character label and a second character label;

[0060] A sample prediction module is used to predict a first sample character of the image training sample based on an initial character prediction model, and obtain a sample character fusion feature of the first sample character;

[0061] The sample candidate recognition module is used to obtain a sample attention hidden state for predicting the second sample character based on the sample character fusion feature in the initial attention memory network included in the initial character prediction model, input the sample attention hidden state into k initial candidate box detection networks, and sequentially identify the target sample candidate box in the image training sample based on the sample attention hidden state in the k initial candidate box detection networks; k is a positive integer;

[0062] The sample aggregation module is used to obtain the target sample candidate image features corresponding to the target sample candidate frames respectively identified by the k initial candidate frame detection networks from the image training samples, and generate sample candidate aggregated image features corresponding to the k target sample candidate image features;

[0063] A first sample prediction module is configured to obtain a sample global image feature of an image training sample, add the sample global image feature to the sample candidate aggregate image feature to obtain a sample attention image feature, and predict a first sample prediction result corresponding to the sample attention image feature;

[0064] A second sample prediction module is used to obtain the sample language latent state of the sample attention image feature, and predict the second sample prediction result corresponding to the image training sample based on the sample attention latent state and the sample language latent state;

[0065] The model training module is used to adjust the parameters of the initial character prediction model according to the first sample prediction result, the second sample prediction result and the second character label to obtain the target character prediction model.

[0066] Among them, the model training module includes:

[0067] a loss generating unit, configured to generate a first loss function based on the first sample prediction result and the second character label, generate a second loss function based on the second sample prediction result and the second character label, and generate a third loss function based on the first sample prediction result and the second sample prediction result;

[0068] A parameter adjustment unit is used to adjust the parameters of the initial character prediction model through the first loss function and the second loss function, and to adjust the parameters of the initial attention memory network in the initial character prediction model through the third loss function to obtain a target character prediction model.

[0069] On the one hand, an embodiment of the present application provides a computer device, including a processor, a memory, and an input and output interface;

[0070] The processor is connected to the memory and the input and output interface respectively, wherein the input and output interface is used to receive and output data, the memory is used to store the computer program, and the processor is used to call the computer program so that the computer device including the processor executes the image detection method in one aspect of the embodiment of the present application.

[0071] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the image detection method in one aspect of the embodiment of the present application.

[0072] In one aspect, an embodiment of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional embodiments of the present application. In other words, when the computer instructions are executed by the processor, the methods provided in various optional embodiments of the present application are implemented.

[0073] Implementing the embodiments of this application will have the following beneficial effects:

[0074] In an embodiment of the present application, a character fusion feature of a first predicted character associated with a target image is obtained; a target attention hidden state for predicting a second predicted character is obtained based on the character fusion feature, and the target attention hidden state is input into k candidate frame detection networks. In the k candidate frame detection networks, target candidate frames in the target image are identified based on the target attention hidden state in turn; k is a positive integer; target candidate image features corresponding to the target candidate frames respectively identified by the k candidate frame detection networks are obtained from the target image, and candidate aggregate image features corresponding to the k target candidate image features are generated; global image features of the target image are obtained, and global image features are added to the candidate aggregate image features to obtain attention image features, and the second predicted character corresponding to the attention image features is predicted. Through the above process, k candidate frame detection networks are used to identify multiple target candidate frames with similar semantics from the target image, that is, multiple target candidate frames with similar semantics but not exactly the same positions, and feature fusion is performed on the multiple target candidate frames to obtain relatively complete features, so as to obtain more complete position information, so that the second predicted character can be predicted based on the target candidate frame after feature fusion, thereby improving the accuracy of image positioning and improving the accuracy of image character prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0076] Figure 1 This is a network interaction architecture diagram of an image detection provided by an embodiment of the present application;

[0077] Figure 2 This is a model architecture diagram of a target character prediction model provided in an embodiment of the present application;

[0078] Figure 3 This is a schematic diagram of an image detection scenario provided by an embodiment of the present application;

[0079] Figure 4 This is a flow chart of an image detection method provided by an embodiment of the present application;

[0080] Figure 5 The embodiment of the present application provides a character prediction model architecture diagram;

[0081] Figure 6 This is a candidate box extraction architecture diagram provided by an embodiment of the present application;

[0082] Figure 7 This is a schematic diagram of a distributed attention network provided by an embodiment of the present application;

[0083] Figure 8 This is a flow chart of a model training method provided in an embodiment of the present application;

[0084] Figure 9 Schematic diagram of an image detection device provided in an embodiment of the present application;

[0085] Figure 10 Schematic diagram of an image detection device provided in an embodiment of the present application;

[0086] Figure 11 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0088] Among them, weakly supervised grounded image caption refers to describing and locating images, where image description refers to generating a description sentence for a given image; image description and localization refers to not only generating a description sentence for the image, but also predicting the position of the target corresponding to the phrase contained in the description sentence in the image; weakly supervised grounded image caption refers to training annotation with only a given image and the corresponding description sentence, while the position of the phrase and the corresponding target in the description sentence is not provided.

[0089] In the examples of this application, see Figure 1 , Figure 1This is a network interaction architecture diagram of an image detection provided by an embodiment of the present application. Figure 1 As shown, the computer device 101 can obtain the target image from a terminal device (such as terminal device 102a, terminal device 102b or terminal device 102c, etc.), or can obtain the target image from the computer device 101, etc. For example, the computer device 101 responds to an image detection request for the target image sent by any terminal device, and obtains the target image from the terminal device that sends the image detection request; or, the computer device 101 performs image detection processing on an image stored locally (i.e., stored in the computer device 101), and obtains the target image from the computer device 101, etc. In other words, the present application can be applied to any scenario where image detection processing is required for a target image, and the computer device 101 can obtain the target image from the storage location of the target image. The computer device 101 can detect the target image, predict d predicted characters associated with the target image in sequence, and form the image description information of the target image with the d predicted characters, where d is a positive integer and the value of d is determined based on the target image prediction, that is, the number of predicted characters included in the image description information predicted by different images is different, that is, the value of d corresponding to different images may be different. For example, if the target image is subjected to image detection processing through the present application and 5 predicted characters are predicted in sequence, then d is 5, and these 5 predicted characters constitute the image description information of the target image; for example, if the target image is subjected to image detection processing through the present application and 8 predicted characters are predicted in sequence, then d is 8, and these 8 predicted characters constitute the image description information of the target image, etc.

[0090] The computer device 101 can perform image detection on the target image in the target character prediction model, predict d predicted characters associated with the target image in sequence, and compose the image description information of the target image with the d predicted characters. Figure 2 , Figure 2 This is a model architecture diagram of a target character prediction model provided by an embodiment of the present application. Figure 2 As shown, the target character prediction model may include an attention memory network, a distributed attention network, and a language memory network, wherein the distributed attention network includes k candidate box detection networks. The computer device may predict the j-th predicted character associated with the target image based on the target character prediction model. In the target character prediction model, based on the character fusion features of the j-th predicted character, the (j+1)-th predicted character associated with the target image is predicted, where j is a positive integer and is less than d.

[0091] Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision making. For example, in the detection of the target image in the present application, the detection process of the target image is intelligently processed, so that the present application can realize intelligent and automatic execution of image detection based on artificial intelligence technology. Among them, artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc. The present application relates to computer vision technology and machine learning technology in the field of artificial intelligence.

[0092] Computer vision (Computer Vision, CV) is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics, so that computer processing becomes images more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, intelligent transportation, etc. It also includes common face recognition, fingerprint recognition and other biometric identification technologies. For example, in the present application, image recognition and image semantic understanding of the target image, such as recognition of the target candidate frame in the target image, and prediction of the second predicted character based on the target candidate frame, etc.

[0093] With the research and progress of artificial intelligence technology, artificial intelligence technology is researched and applied in many fields, such as common smart home, smart wearable device, virtual assistant, smart speaker, intelligent marketing, unmanned driving, autonomous driving, unmanned aerial vehicle, robot, intelligent medical treatment, intelligent customer service, Internet of Vehicles, autonomous driving, intelligent transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0094] The scheme provided by the embodiments of the present application relates to computer vision and machine learning technologies of artificial intelligence, which is specifically explained by the following embodiments:

[0095] Specifically, please refer to Figure 3 , Figure 3 is a schematic diagram of an image detection scene provided by an embodiment of the present application. As shown in Figure 3 , the computer device can obtain a target image 301, obtain character fusion features of a first predicted character associated with the target image 301, obtain a target attention hidden state for predicting a second predicted character based on the character fusion features, and input the target attention hidden state into k candidate box detection networks, such as candidate box detection network 3021, candidate box detection network 3022, and candidate box detection network 302k. In the k candidate box detection networks, target candidate boxes in the target image 301 are recognized based on the target attention hidden state in turn, such as a target candidate box recognized by the candidate box detection network 3021, a target candidate box recognized by the candidate box detection network 3022, and a target candidate box recognized by the candidate box detection network 302k. Target candidate image features corresponding to the target candidate boxes recognized by the k candidate box detection networks are obtained from the target image 301, and candidate aggregated image features corresponding to the k target candidate image features are generated. A global image feature of the target image 301 is obtained, the global image feature is added to the candidate aggregated image features to obtain an attention image feature, and the second predicted character corresponding to the attention image feature is predicted. Through the k candidate box detection networks, the target image is recognized to obtain k target candidate boxes that are not completely the same, and feature aggregation is performed on target candidate image features corresponding to the k target candidate boxes, which can expand the features obtained based on the candidate boxes to obtain more complete and more global features, so as to make the image positioning more accurate, the semantic information used for predicting the second predicted character can be enriched based on the complete and global features, and the accuracy of image positioning and image character prediction is improved.

[0096] It is understood that the computer devices mentioned in the embodiments of the present application include but are not limited to terminal devices or servers. In other words, the computer device can be a server or a terminal device, or it can be a system composed of a server and a terminal device. Among them, the terminal device mentioned above can be an electronic device, including but not limited to mobile phones, tablet computers, desktop computers, laptop computers, PDAs, vehicle-mounted devices, augmented reality / virtual reality (AR / VR) devices, helmet displays, smart TVs, wearable devices, smart speakers, digital cameras, cameras and other mobile Internet devices (mobile internet devices, MIDs) with network access capabilities, or terminal devices in scenarios such as trains, ships, and flights. Among them, the server mentioned above can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0097] Optionally, the data involved in the embodiments of the present application can be stored in a computer device, or the data can be stored based on cloud storage technology or a blockchain network, without limitation. For example, the computer device can store the data generated during the image detection process in the computer device's own storage space, or can store the data generated during the image detection process in a blockchain network, or can store the data generated during the image detection process in a cloud storage, without limitation.

[0098] Further, see Figure 4 , Figure 4 This is a flow chart of an image detection method provided by an embodiment of the present application. Figure 4 As shown, the image detection process includes the following steps:

[0099] Step S401: Acquire a character fusion feature of a first predicted character associated with a target image.

[0100] In an embodiment of the present application, a computer device can obtain the global features of the candidate box corresponding to the target image, perform feature extraction on the first predicted character associated with the target image, and obtain the character embedding features of the first predicted character. For example, the computer device can perform embedding vector conversion (embedding) on ​​the first predicted character associated with the target image to obtain the character embedding features of the first predicted character; or, the computer device performs character feature extraction on the first predicted character to obtain the character embedding features of the first predicted character. According to the global features of the candidate box and the character embedding features, the character fusion features of the first predicted character are obtained. Wherein, when obtaining the global features of the candidate box corresponding to the target image, the computer device can obtain N initial candidate boxes corresponding to the target image, and identify the candidate local features corresponding to each initial candidate box from the target image; N is a positive integer. Specifically, the computer device can obtain the candidate area corresponding to each initial candidate box in the target image, and determine the image features of the candidate area corresponding to each initial candidate box as the candidate local features corresponding to the initial candidate box. Perform feature fusion on the N candidate local features to obtain the global features of the candidate box corresponding to the target image. Specifically, the computer device can obtain the initial weights corresponding to the N initial candidate boxes, perform weighted summation on the N candidate local features based on the initial weights corresponding to the N initial candidate boxes, and obtain the global features of the candidate box corresponding to the target image. Furthermore, when the computer device obtains the character fusion features of the first predicted character based on the global features of the candidate box and the character embedding features, it can perform feature splicing on the global features of the candidate box and the character embedding features to obtain the character fusion features of the first predicted character; or, obtain the historical language latent state of the first predicted character, perform feature splicing on the historical language latent state, the global features of the candidate box and the character embedding features to obtain the character fusion features of the first predicted character. For example, a method for generating the character fusion features can be shown in formula ①:

[0101]

[0102] In formula ①, It is used to represent the input features at time t, that is, the character fusion features of the first predicted character. In other words, the first predicted character refers to the character predicted at time (t-1), and the input features at time t are determined based on the first predicted character. It refers to the language latent state generated at time (t-1), that is, the historical language latent state of the first predicted character, where the subscript is used to indicate the time, such as (t-1) refers to time (t-1), and the superscript is used to indicate the type of latent state, such as superscript 1 indicates the attention latent state and superscript 2 indicates the language latent state. refers to the candidate global features, e t-1It refers to the embedding features of the predicted character predicted at time (t-1), that is, the character embedding features of the first predicted character. In other words, at time t, the computer device can obtain the first predicted character predicted at time (t-1), obtain the language latent state generated at time (t-1), and record it as the historical language latent state of the first predicted character; obtain the character embedding features of the first predicted character, perform feature concatenation on the historical language latent state, the candidate box global features, and the character embedding features, to obtain the character fusion features of the first predicted character, and use the character fusion features as the input features at time t. Wherein, t is a positive integer. Optionally, if t is 1, it means that the currently predicted character is the first character, and there is no predicted character at time (t-1), that is, the first predicted character is empty. The default language latent state can be determined as the historical language latent state of the first predicted character, the default embedding feature can be determined as the character embedding feature of the first predicted character, and the character fusion features of the first predicted character can be obtained based on the candidate box global features and the character embedding features. Alternatively, the empty feature can be determined as the character fusion feature of the first predicted character, or the basic character feature can be determined as the character fusion feature of the first predicted character, etc. The basic character feature refers to the default character feature when no predicted character exists. In other words, if t is 1, it means that the computer device currently predicts the first predicted character associated with the target image.

[0103] Among them, the first predicted character refers to the predicted character predicted at time (t-1), and the second predicted character refers to the predicted character predicted at time t. For example, assuming that the image description information of the predicted target image includes d predicted characters, the jth predicted character can be recorded as the first predicted character, and the (j+1)th predicted character can be recorded as the second predicted character, where j is a positive integer less than d.

[0104] Step S402: obtaining a target attention hidden state for predicting a second predicted character based on the character fusion feature, and inputting the target attention hidden state into k candidate box detection networks.

[0105] In an embodiment of the present application, a computer device can obtain a historical attention latent state for predicting a first predicted character, perform feature fusion on the historical attention latent state and the character fusion feature based on an attention memory network, and obtain a target attention latent state for predicting a second predicted character. Specifically, the computer device can obtain the attention latent state generated at time (t-1), perform feature fusion on the attention latent state and the character fusion feature at time (t-1) based on an attention memory network, and obtain the attention latent state at time t, wherein the attention latent state at time (t-1) is used to predict the first predicted character and can be recorded as the historical attention latent state, and the attention latent state at time t is used to predict the second predicted character and can be recorded as the target attention latent state. For example, see Figure 5 , Figure 5 This is a character prediction model architecture diagram provided by the embodiment of the present application. Figure 5 As shown, the computer device can obtain the historical attention latent state generated by the attention memory network at time (t-1), process the historical attention latent state and the character fusion features, and generate the attention latent state at time t, i.e., the target attention latent state. Optionally, if t is 1, it means that the current prediction is the first character, and the historical attention latent state can be considered empty. Optionally, the attention memory network can be an Attention Long Short-Term Memory (Attention LSTM) network.

[0106] Furthermore, the computer device can input the target attention hidden state into k candidate box detection networks, wherein the network where the k candidate box detection networks are located can be recorded as a distributed attention network, that is, the distributed attention network includes k candidate box detection networks, and the distributed attention network is equivalent to a virtual network. Figure 5 As shown, the computer device inputs the target attention hidden state into k candidate box detection networks 5011. Optionally, the k candidate box detection networks 5011 can be considered to belong to the distributed attention network 501.

[0107] Step S403: In the k candidate box detection networks, target candidate boxes in the target image are identified based on the target attention hidden state in turn.

[0108] In an embodiment of the present application, a computer device can identify target candidate frames in a target image based on target attention hidden states in sequence in k candidate frame detection networks. Optionally, the computer device can obtain an initial candidate frame from a candidate pool in the i-th candidate frame detection network, and identify the target candidate frame in the target image in the obtained initial candidate frame. When the i-th candidate frame detection network identifies the target candidate frame, the target candidate frame identified by the i-th candidate frame detection network in the candidate pool can be deleted so that other candidate frame detection networks will not recognize the target candidate frame identified by the i-th candidate frame detection network again, so as to obtain k semantically similar but not identical (i.e., different positions) target candidate frames, thereby improving the integrity of target candidate frame recognition. Among them, due to the characteristics of the candidate box detection network, continuous identification will be carried out in a candidate box detection network to obtain a selected candidate box, making the similarity of semantic information between the selected candidate box and other candidate boxes unclear. That is, except for the selected candidate box, the greater the probability of the predicted candidate box, the greater the correlation with the character to be predicted. Therefore, k candidate box detection networks are used to select respectively to obtain k semantically similar or even identical target candidate boxes, thereby improving the integrity of candidate box prediction and the integrity of semantic information retention.

[0109] In the i-th candidate box detection network, the (N-i+1) initial candidate boxes and the initial candidate features of each initial candidate box in the target image are identified based on the target attention hidden state; N is a positive integer, and i is a positive integer less than or equal to k; the (N-i+1) initial candidate boxes refer to the initial candidate boxes among the N initial candidate boxes, excluding the target candidate boxes identified by the (i-1) candidate box detection networks, wherein the (i-1) candidate box detection network here refers to the candidate box detection network that identified the target candidate box before the i-th candidate box detection network. For example, when i is 1, the (i-1) candidate box detection network is empty. When i is greater than 1, the (i-1) candidate box detection network can be considered to be the 1st candidate box detection network to the (i-1)th candidate box detection network. Optionally, the i in the i-th candidate box detection network is used to represent the order in which the candidate box detection network predicts the target candidate boxes. Furthermore, the (N-i+1) initial candidate features are transformed into feature dimensions to obtain candidate probabilities corresponding to the (N-i+1) initial candidate frames. Based on the candidate probabilities corresponding to the (N-i+1) initial candidate frames, the target candidate frame is determined from the (N-i+1) initial candidate frames. The target candidate frames corresponding to the k candidate frame detection networks are different from each other.

[0110] For example, it is assumed that the initial candidate features of each initial candidate box are b-dimensional features, b being a positive integer, that is, in the i-th candidate box detection network, the computer device can obtain an initial feature map, which is an (N-i+1)*b-dimensional feature, wherein the initial feature map includes initial candidate features corresponding to the (N-i+1) initial candidate boxes. Alternatively, the initial feature map can also be an N*b-dimensional feature, at this time, the initial candidate feature corresponding to the target candidate box that has been identified in the initial feature map is a default value, such as 0, to indicate that the candidate box has been identified and cannot be selected continuously. The computer device can perform feature dimension conversion on the (N-i+1) initial candidate features to obtain candidate probabilities corresponding to the (N-i+1) initial candidate boxes, respectively. For example, the initial feature map can be processed based on a fully connected layer to realize feature dimension conversion of the (N-i+1) initial candidate features to obtain conversion features, and determine candidate probabilities corresponding to the (N-i+1) initial candidate boxes, respectively, according to the conversion features; wherein the dimension of the conversion features is determined according to the dimension of the initial features, for example, if the dimension of the initial features is (N-i+1)*b, then the dimension of the conversion features is (N-i+1)*1; if the dimension of the initial features is N*b, then the dimension of the conversion features is N*1. Alternatively, the (N-i+1) initial candidate features can be converted in feature dimension respectively, that is, the b-dimensional features are converted into one-dimensional features to obtain one-dimensional features corresponding to each initial candidate feature, respectively, and then the one-dimensional features corresponding to each initial candidate feature are normalized to obtain candidate probabilities corresponding to the (N-i+1) initial candidate boxes, respectively, wherein the method used for feature dimension conversion of the (N-i+1) initial candidate features is the same, so as to ensure that the probability acquisition methods of the initial candidate boxes are the same, and improve the determination accuracy of the candidate probabilities. Further, the initial candidate box with the largest candidate probability can be determined as the target candidate box corresponding to the i-th candidate box detection network.

[0111] Alternatively, the computer device can use a faster Region Convolutional Neural Network (faster RCNN) method to obtain the target candidate box corresponding to each candidate box detection network, in other words, the candidate box detection network can be a faster RCNN. In this method, please refer to Figure 6 , Figure 6 is a candidate box extraction architecture diagram provided by an embodiment of the present application. As Figure 6As shown, the computer device can extract an image feature map 602 of the target image in the convolutional layer 601, generate an initial candidate box 603 based on a target attention hidden state, fuse the initial candidate box 603 with the image feature map 602 to obtain an initial feature map 604, obtain a box category of each initial candidate box based on the initial feature map 604, and perform boundary regression processing on the initial feature map 604 using the box category to obtain a candidate probability of each initial candidate box.

[0112] Optionally, referring to Figure 7 , Figure 7 is a distributed attention network diagram provided by an embodiment of the present application. As shown in Figure 7 , it is assumed that the computer device has predicted the predicted character "A" and the predicted character "soccer", at this time, the predicted character "soccer" is determined as a first predicted character, the character fusion feature of the first predicted character is obtained, the character fusion feature is input into the attention memory network, the target attention hidden state for predicting a second predicted character is obtained based on the attention memory network, and the target attention hidden state is input into the distributed attention network 701, specifically, the target attention hidden state is input into k candidate box detection networks. In the k candidate box detection networks, the target candidate box in the target image is identified based on the target attention hidden state in turn, for example, the target candidate box 7011 in the target image is identified in the first candidate box detection network, and the target candidate box 7011 is marked as an identified state; in the second candidate box detection network, the target candidate box 7011 is in the identified state, the target candidate box 7012 in the target image is identified, and the target candidate box 7012 is marked as an identified state; in the kth candidate box detection network, the target candidate box 7011 and the target candidate box 7012 are in the identified state, and the target candidate box 701k in the target image is identified. Among them, Figure 7 , the solid line box represents the target candidate box identified by the corresponding candidate box detection network, the dotted line box is used to represent the identified candidate box of the corresponding candidate box detection network when identifying the target candidate box, the identified candidate box refers to the candidate box in the identified state, and the long dashed line box is used to represent the candidate box that can be detected by the corresponding candidate box detection network when identifying the target candidate box, that is, the unidentified candidate box, which refers to the candidate box that is not identified as the target candidate box.

[0113] Among them, in the present application, the k candidate box detection networks predict the same predicted character (i.e., the second predicted character), and the prediction is based on the features of the same predicted character (i.e., the first predicted character), so that the semantic information focused on by each candidate box detection network is the same, that is, it corresponds to the same target object (i.e., the object corresponding to the second predicted character), so that the k candidate box detection networks can extract target candidate boxes with similar or even identical semantics and different positions. When the k target candidate boxes are feature-fused, the semantic range of the candidate boxes can be expanded, that is, a more complete candidate box is obtained, richer semantic information and more accurate prediction features are obtained, thereby improving the accuracy of image positioning and character prediction when predicting the second predicted character based on the k target candidate boxes.

[0114] Step S404: acquiring target candidate image features corresponding to the target candidate frames respectively identified by the k candidate frame detection networks from the target image, and generating candidate aggregate image features corresponding to the k target candidate image features.

[0115] In an embodiment of the present application, a computer device can obtain from a target image the target candidate regions corresponding to the target candidate frames respectively identified by k candidate frame detection networks, obtain regional image features of the k target candidate regions from the target image, and determine the regional image features as the target candidate image features of the corresponding target candidate frames. That is, one target candidate frame corresponds to one target candidate region, and one target candidate region corresponds to one regional image feature. Based on the correspondence, the regional image features corresponding to each target candidate frame are determined, which are the target candidate image features corresponding to the target candidate frame. Furthermore, the computer device can perform feature fusion on the k target candidate image features to obtain candidate aggregate image features, wherein the feature fusion can be a weighted processing, that is, the k target candidate image features can be weighted processing to obtain candidate aggregate image features. Specifically, the computer device can perform feature fusion on the k target candidate image features and the target attention hidden state respectively to obtain k initial candidate weights. Optionally, the computer device can directly convert the k target candidate image features and the target attention hidden state into features of the same dimension, and perform feature superposition of the converted k target candidate image features with the converted target attention hidden state to obtain k initial candidate weights; or, the computer device can obtain a first weight parameter corresponding to the target candidate frame and a second weight parameter corresponding to the target attention hidden state, and perform weighted processing on the k target candidate image features based on the first weight parameter to obtain k weighted target candidate image features, and perform weighted processing on the target attention hidden state based on the second weight parameter to obtain a semantic weight parameter, and add the semantic weight parameter to each weighted target candidate image feature to obtain k initial candidate weights, etc., without limitation here. The k initial candidate weights are normalized to obtain the target candidate weights corresponding to the k target candidate image features; based on the target candidate weights corresponding to the k target candidate image features, the k target candidate image features are weighted and summed to obtain a candidate aggregate image feature.

[0116] An optional method for obtaining the initial candidate weight can be shown in formula ②:

[0117]

[0118] In formula ②, a is used to represent the candidate weight, a i,t It is used to represent the initial candidate weight corresponding to the i-th target candidate image feature, and t refers to the time t, which can be considered as the time when the second predicted character is predicted. i represents the i-th target candidate image feature, It is used to represent the attention hidden state at time t, that is, the target attention hidden state used to predict the second predicted character. Specifically, the computer device can obtain the first weight parameter W corresponding to the target candidate boxva , and the second weight parameter W corresponding to the target attention hidden state ha , wherein, in the subscripts of the two weight parameters, a is used to indicate that the parameter is used to generate candidate weights, v is used to indicate that the first weight parameter is a weight parameter for the target candidate frame, and h is used to indicate that the second weight parameter is a weight parameter for the hidden state. The computer device performs weighted processing on the i-th target candidate image feature based on the first weight parameter to obtain the i-th weighted target candidate image feature; performs weighted processing on the target attention hidden state based on the second weight parameter to obtain a semantic weight parameter; adds the semantic weight parameter to the i-th weighted target candidate image feature to obtain a weight fusion parameter; performs activation processing on the weight fusion parameter to obtain an activation weight parameter, and performs dimension conversion on the activation weight parameter to obtain the initial candidate weight corresponding to the i-th target candidate image feature. Optionally, an activation function may be used to activate the weight fusion parameter, and the activation function includes but is not limited to the hyperbolic tangent function tanh(), the linear rectifier function (Rectified Linear Unit, ReLU) and the S-type function (Sigmoid). Further optionally, the computer device may use a third weight parameter to perform dimension conversion on the activation weight parameter to obtain the initial candidate weight corresponding to the i-th target candidate image feature. The third weight parameter may be ω as in formula ② a The parameters shown, The superscript T in refers to the transpose, that is, is the transpose of the third weight parameter; or, The third weight parameter can also be directly expressed.

[0119] Furthermore, the computer device can perform normalization processing on the k initial candidate weights to obtain target candidate weights corresponding to the k target candidate image features. The normalization processing can be to obtain the weight ratios of the k initial candidate weights in the sum of the k initial candidate weights, that is, the computer device can determine the weight ratios of the k initial candidate weights in the sum of the k initial candidate weights as the target candidate weights corresponding to the k target candidate image features; or the computer device can perform feature splicing on the k initial candidate weights to obtain a weight vector, and use a normalization function to normalize the weight vector to obtain the target candidate weights corresponding to the k target candidate image features, etc., which are not limited here.

[0120] An optional generation process of the target candidate weight can be shown in formula ③:

[0121] α t =softmax(a t ) ③

[0122] Among them, a t Refers to k initial candidate weights, that is, k initial candidate weights are feature spliced ​​to obtain the weight vector a t , the weight vector is normalized using a normalization function to obtain the target candidate weights corresponding to the k target candidate image features, wherein the normalization function can be but is not limited to a normalized exponential function (softmax) or sample-by-sample mean reduction. t It can be considered as a k-dimensional vector, which can be recorded as a target weight vector, used to represent the target candidate weights corresponding to the k target candidate image features.

[0123] Furthermore, the generation process of candidate aggregated image features can be shown in formula ④:

[0124]

[0125] In formula ④, α i,t Refers to the target candidate weight corresponding to the i-th target candidate image feature, that is, the target weight vector α t The i-th target candidate weight in is based on the target candidate weights corresponding to the k target candidate image features, and the k target candidate image features are weighted summed to obtain the candidate aggregate image feature

[0126] Step S405 , obtaining the global image features of the target image, and adding the global image features to the candidate aggregated image features to obtain the attention image features.

[0127] In the embodiment of the present application, the computer device can obtain the global image features of the target image, specifically, it can obtain the image feature map of the target image and determine the image feature map as the global image features of the target image. Figure 5 As shown in , global image features are added to the candidate aggregate image features to obtain attention image features, where ⊕ is used to represent feature addition. The acquisition process of the attention image features can be shown in formula ⑤:

[0128]

[0129] In formula ⑤, in the candidate aggregate image feature Add the global image feature Fc to get the attention image feature

[0130] Optionally, the computer device may also generate an object positioning frame corresponding to the second predicted character based on the k target candidate frames, such as Figure 7As shown in the object positioning frame 702 in the figure. The computer device can parse the candidate aggregate image features to obtain the object positioning frame. Alternatively, the computer device can directly perform position fusion on the k target candidate frames to obtain the object positioning frame. In this way, the computer device can also directly obtain the features corresponding to the object positioning frame in the target image as the candidate aggregate image features corresponding to the k target candidate image features. The position fusion method includes but is not limited to the maximum bounding box determination method and the Gaussian distribution fitting method, etc., which are not limited here. For example, the computer device can obtain the candidate frame position information corresponding to the k target candidate frames in the target image, perform position expansion based on the candidate frame position information, and obtain the object positioning frame so that the object positioning frame can include k target candidate frames and will not be too large. Specifically, the maximum vertical coordinate, minimum vertical coordinate, maximum horizontal coordinate and minimum horizontal coordinate in the candidate frame position information corresponding to the k target candidate frames can be obtained, and the candidate frame composed of the maximum vertical coordinate, minimum vertical coordinate, maximum horizontal coordinate and minimum horizontal coordinate is determined as the object positioning frame. This method is the maximum bounding box determination method. Alternatively, the computer device can obtain the candidate frame position information corresponding to k target candidate frames in the target image, obtain the fitting position information corresponding to the k candidate frame position information by fitting the Gaussian distribution, and determine the object positioning frame based on the fitting position information, where the fitting position information includes the coordinates of the four vertices of the object positioning frame, etc.

[0131] Step S406: predicting a second predicted character corresponding to the attention image feature.

[0132] In an embodiment of the present application, the computer device can predict the second predicted character corresponding to the attention image feature in the language memory network; or, the computer device can predict the first prediction result corresponding to the attention image feature in the language memory network, predict the second prediction result based on the attention image feature in the semantic consistency network, and integrate the first prediction result and the second prediction result to obtain the second predicted character.

[0133] In one embodiment, the computer device can obtain the historical language latent state corresponding to the first predicted character in the language memory network, perform feature fusion on the historical language latent state and the attention image feature, and predict M candidate characters and the character matching probability corresponding to each candidate character; M is a positive integer. Among the M candidate characters, the candidate character with the largest character matching probability is determined as the second predicted character. In another embodiment, the language memory network can be a language long short-term memory network (Language Long Short-Term Memory, Language LSTM). Figure 7 As shown, in the language memory network, the attention image features are predicted to obtain the second predicted character "player".

[0134] In one approach, a computer device can perform feature prediction processing on the attention image features and the target attention latent state in a language memory network to obtain a first prediction result and a target language latent state. The first prediction result can include M candidate characters and a first matching probability corresponding to each candidate character. Feature fusion is performed on the target attention latent state and the target language latent state to obtain a latent state fusion feature. The latent state fusion feature is then predicted to obtain a second prediction result. The second prediction result includes M candidate characters and a second matching probability corresponding to each candidate character. The first and second prediction results are then integrated to obtain a second predicted character. Specifically, the first matching probability and the second matching probability corresponding to each candidate character can be summed to obtain the character matching probability of the candidate character; or, the first network weight corresponding to the language memory network and the second network weight corresponding to the semantic consistency network can be obtained, and based on the first network weight and the second network weight, the first matching probability and the second matching probability corresponding to each candidate character can be weighted summed to obtain the character matching probability of the candidate character. For example, if the first network weight is 0.6 and the second network weight is 0.4, the first matching probability of candidate character 1 is 0.5 and the second matching probability is 0.3, then the character matching probability of candidate character 1 is (0.6*0.5+0.4*0.3=0.42). The computer device can determine the candidate character with the largest character matching probability as the second predicted character.

[0135] Among them, when performing feature fusion on the target attention latent state and the target language latent state to obtain the latent state fusion feature, the computer device can use the first fully connected layer to perform dimension conversion on the target attention latent state to generate a first converted latent state of the target dimension; use the second fully connected layer to perform dimension conversion on the target language latent state to generate a second converted latent state of the target dimension; perform state superposition processing on the first converted latent state and the second converted latent state to obtain the latent state fusion feature.

[0136] like Figure 5As shown, in the language memory network, the computer device performs feature prediction processing on the attention image features and the target attention latent state to obtain a first prediction result and a target language latent state. In the semantic consistency network 502, the first fully connected layer 5021 performs dimension conversion on the target attention latent state to generate a first converted latent state of the target dimension. The second fully connected layer 5022 performs dimension conversion on the target language latent state to generate a second converted latent state of the target dimension. The first converted latent state and the second converted latent state are subjected to state superposition processing to obtain a latent state fusion feature 5023, where ⊕ is used to represent state superposition processing. The latent state fusion feature 5023 is predicted by the activation layer in the semantic consistency network 502 to obtain a second prediction result. The first prediction result and the second prediction result can be integrated to obtain a second predicted character.

[0137] Among them, the target character prediction model used in this application is based on the semantic consistency network for parameter adjustment, that is, the target character prediction model can predict the second sample prediction result corresponding to the image training sample through the sample attention latent state and the sample language latent state generated by the image training sample, and adjust the parameters of the attention memory network in the target character prediction model based on the second sample prediction result, thereby reducing the semantic difference caused by the prediction of the second predicted character by the first predicted character, and improving the semantic accuracy of the attention latent state. In other words, the trained target character prediction model can improve the semantic accuracy of the attention latent state. Therefore, the second predicted character directly predicted based on the language memory network also has this characteristic, that is, it can improve the prediction accuracy. Optionally, the prediction result can be optimized through the semantic consistency network on the basis of the language memory network to further improve the accuracy of character prediction.

[0138] In an embodiment of the present application, a character fusion feature of a first predicted character associated with a target image is obtained; a target attention hidden state for predicting a second predicted character is obtained based on the character fusion feature, and the target attention hidden state is input into k candidate frame detection networks. In the k candidate frame detection networks, target candidate frames in the target image are identified based on the target attention hidden state in turn; k is a positive integer; target candidate image features corresponding to the target candidate frames respectively identified by the k candidate frame detection networks are obtained from the target image, and candidate aggregate image features corresponding to the k target candidate image features are generated; global image features of the target image are obtained, and global image features are added to the candidate aggregate image features to obtain attention image features, and the second predicted character corresponding to the attention image features is predicted. Through the above process, k candidate frame detection networks are used to identify multiple target candidate frames with similar semantics from the target image, that is, multiple target candidate frames with similar semantics but not exactly the same positions, and feature fusion is performed on the multiple target candidate frames to obtain relatively complete features, so as to obtain more complete position information, so that the second predicted character can be predicted based on the target candidate frame after feature fusion, thereby improving the accuracy of image positioning and improving the accuracy of image character prediction.

[0139] Further, see Figure 8 , Figure 8 This is a flow chart of a model training method provided in an embodiment of the present application. Figure 8 As shown in Figure 2, the model training process includes the following steps:

[0140] Step S801: Obtain an image training sample and obtain an image description sentence corresponding to the image training sample.

[0141] In an embodiment of the present application, a computer device can obtain an image training sample and an image description sentence corresponding to the image training sample, wherein one image training sample can correspond to p image description sentences, where p is a positive integer, that is, a sample training pair consisting of an image training sample and p image description sentences can be obtained. In an embodiment of the present application, an image training sample and an image description sentence corresponding to the image training sample are used as an example for description. The image description sentence may include a first character label and a second character label, such as Figure 7As shown, if the image is used as an image training sample, the image description sentence corresponding to the image training sample is "A soccer player in white is kicking the ball.", including the character label "A", the character label "soccer", the character label "player", the character label "in", the character label "white", the character label "is", the character label "kicking", the character label "the", the character label "ball" and the character label ".", wherein the character label located in front of the two adjacent character labels can be recorded as the first character label, and the character label located behind the two adjacent character labels can be recorded as the second character label. For example, in the character label "soccer" and the character label "player", the character label "soccer" can be recorded as the first character label, and the character label "player" can be recorded as the second character label.

[0142] Optionally, the computer device may obtain a training sample set and divide the training sample set into image training samples and image test samples. Optionally, the training sample set may also be divided into image training samples, image verification samples and image test samples.

[0143] Step S802 : predicting a first sample character of the image training sample based on the initial character prediction module, and obtaining a sample character fusion feature of the first sample character.

[0144] In the embodiment of this application, this process can be seen in Figure 4 The specific description shown in step S401 will not be repeated here. Optionally, the first sample character of the image training sample is predicted here to indicate that the present application predicts the second sample character on the basis of the first sample character, that is, the description information of the image is predicted character by character. Optionally, the computer device can directly obtain the data associated with the first character label based on the initial character prediction module, obtain the label character fusion feature of the first character label, and use it to predict the second sample character, which is not limited here. Here, the first sample character is predicted, and the model is adjusted based on the first sample character and the first character label so that the first sample character predicted by the model is the first character label.

[0145] Step S803: In the initial attention memory network included in the initial character prediction model, a sample attention hidden state for predicting the second sample character is obtained based on the sample character fusion feature, and the sample attention hidden state is input into k initial candidate box detection networks.

[0146] In the embodiment of this application, this process can be seen in Figure 4 The specific description of step S402 in FIG. 1 is omitted here.

[0147] Step S804: In the k initial candidate box detection networks, target sample candidate boxes in the image training samples are identified based on the sample attention hidden state in sequence.

[0148] In the embodiment of this application, this process can be seen in Figure 4 The specific description of step S403 in FIG. 4 is omitted here.

[0149] Step S805 , obtaining target sample candidate image features corresponding to the target sample candidate frames respectively identified by the k initial candidate frame detection networks from the image training samples, and generating sample candidate aggregate image features corresponding to the k target sample candidate image features.

[0150] In the embodiment of this application, this process can be seen in Figure 4 The specific description of step S404 in will not be repeated here.

[0151] Step S806 , obtaining the sample global image features of the image training sample, adding the sample global image features to the sample candidate aggregated image features, and obtaining the sample attention image features.

[0152] In the embodiment of this application, this process can be seen in Figure 4 The specific description of step S405 in will not be repeated here.

[0153] Step S807 , predicting a first sample prediction result corresponding to the sample attention image feature.

[0154] In an embodiment of the present application, in a language memory network, feature prediction processing is performed on the sample attention image features to obtain a first sample prediction result and a sample language latent state. The first sample prediction result includes M candidate characters and a first sample matching probability corresponding to each candidate character.

[0155] Step S808: Obtain the sample language latent state of the sample attention image feature, and predict a second sample prediction result corresponding to the image training sample based on the sample attention latent state and the sample language latent state.

[0156] In this embodiment of the present application, a computer device obtains a sample language latent state of a sample attention image feature, that is, obtains the sample language latent state generated in the language memory network. Feature fusion is performed on the sample attention latent state and the sample language latent state to obtain a sample latent state fusion feature. Feature fusion is then performed on the sample latent state fusion feature to obtain a second sample prediction result.

[0157] Step S809 , adjusting parameters of the initial character prediction model according to the first sample prediction result, the second sample prediction result and the second character label to obtain a target character prediction model.

[0158] In an embodiment of the present application, a first loss function is generated based on the first sample prediction result and the second character label, a second loss function is generated based on the second sample prediction result and the second character label, and a third loss function is generated based on the first sample prediction result and the second sample prediction result; the parameters of the initial character prediction model are adjusted through the first loss function and the second loss function, and the parameters of the initial attention memory network in the initial character prediction model are adjusted through the third loss function to obtain the target character prediction model.

[0159] Optionally, the computer device may test and evaluate the target character prediction model based on the image test sample to obtain the model accuracy of the target character prediction model. Optionally, the computer device may also optimize and adjust the target character prediction model based on the model accuracy and the test process to obtain an optimized target character prediction model. Optionally, if an image verification sample is available, the target character prediction model may be verified before testing and evaluating the target character prediction model to fine-tune the target character prediction model, such as the number of convolutional layers or the size of the convolution kernel in the target character prediction model.

[0160] In an embodiment of the present application, since the input at the attention memory network is the first predicted character, and the second predicted character needs to be predicted in the language memory network, the distributed attention network may have semantic confusion when determining the target candidate box, and cannot accurately select the target candidate box for the second predicted character. During model training, the semantic information of the language latent state is transferred to the semantic information of the attention latent state in the semantic consistency network to adjust the parameters of the attention memory network. Since the language latent state represents more semantic information of the currently predicted character, the semantic difference between the language latent state and the attention latent state is reduced, so that the trained target character prediction model can more accurately predict the second predicted character based on the first predicted character, and also improves the accuracy of image positioning (i.e., target candidate box determination).

[0161] in, Figure 4 Computer equipment used for image detection and Figure 8 The computer device used for model training can be the same computer device or different computer devices. In other words, Figure 4 The method shown is the same as Figure 8 The methods shown can be implemented in the same computer device or in different computer devices.

[0162] Further, see Figure 9 , Figure 9 Schematic diagram of an image detection device provided in an embodiment of the present application. The image detection device can be a computer program (including program code, etc.) running on a computer device. For example, the image detection device can be an application software; the device can be used to execute the corresponding steps of the method provided in an embodiment of the present application. Figure 9 As shown, the image detection device 900 can be used to Figure 4 The computer device in the corresponding embodiment, specifically, the device may include: an input acquisition module 11, an attention detection module 12, a feature transfer module 13, a candidate frame recognition module 14, a feature recognition module 15, a candidate aggregation module 16, a feature fusion module 17 and a character prediction module 18.

[0163] An input acquisition module 11 is configured to acquire a character fusion feature of a first predicted character associated with a target image;

[0164] An attention detection module 12 is configured to obtain a target attention latent state for predicting a second predicted character based on the character fusion feature;

[0165] Feature transfer module 13, used to input the target attention hidden state into the k candidate box detection network;

[0166] The candidate box recognition module 14 is used to sequentially recognize target candidate boxes in the target image based on the target attention hidden state in k candidate box detection networks; k is a positive integer;

[0167] A feature recognition module 15 is used to obtain target candidate image features corresponding to the target candidate frames respectively identified by the k candidate frame detection networks from the target image;

[0168] A candidate aggregation module 16 is configured to generate candidate aggregated image features corresponding to k target candidate image features;

[0169] A feature fusion module 17 is used to obtain the global image features of the target image, add the global image features to the candidate aggregated image features, and obtain the attention image features;

[0170] The character prediction module 18 is configured to predict a second predicted character corresponding to the attention image feature.

[0171] The candidate frame recognition module 14 includes:

[0172] The initial recognition unit 141 is configured to recognize (N-i+1) initial candidate frames and initial candidate features of each initial candidate frame in the target image based on the target attention hidden state in the i-th candidate frame detection network; N is a positive integer, and i is a positive integer less than or equal to k; the (N-i+1) initial candidate frames refer to the initial candidate frames among the N initial candidate frames, excluding the target candidate frames respectively recognized by the (i-1) candidate frame detection networks;

[0173] The probability determination unit 142 is used to perform feature dimension conversion on the (N-i+1) initial candidate features to obtain candidate probabilities corresponding to the (N-i+1) initial candidate boxes.

[0174] The candidate box selection unit 143 is used to determine the target candidate box from the (N-i+1) initial candidate boxes based on the candidate probabilities corresponding to the (N-i+1) initial candidate boxes; the target candidate boxes corresponding to the k candidate box detection networks are different from each other.

[0175] The attention detection module 12 includes:

[0176] The attention fusion unit 121 is used to obtain the historical attention latent state used to predict the first predicted character, and perform feature fusion on the historical attention latent state and the character fusion feature based on the attention memory network to obtain the target attention latent state used to predict the second predicted character.

[0177] The candidate aggregation module 16 includes:

[0178] The weight acquisition unit 161 is used to perform feature fusion on the k target candidate image features and the target attention hidden state respectively to obtain k initial candidate weights;

[0179] The weight determination unit 162 is used to normalize the k initial candidate weights to obtain target candidate weights corresponding to the k target candidate image features;

[0180] The feature weighting unit 163 is configured to perform weighted summation on the k target candidate image features based on the target candidate weights corresponding to the k target candidate image features, to obtain a candidate aggregated image feature.

[0181] The character prediction module 18 includes:

[0182] A first prediction unit 181 is configured to perform feature prediction processing on the attention image features and the target attention latent state in the language memory network to obtain a first prediction result and a target language latent state;

[0183] A latent state fusion unit 182 is used to fuse the target attention latent state and the target language latent state to obtain a latent state fusion feature;

[0184] The second prediction unit 183 is used to predict the hidden state fusion feature to obtain a second prediction result;

[0185] The result integration unit 184 is configured to perform result integration processing on the first prediction result and the second prediction result to obtain a second predicted character.

[0186] The Cain state fusion unit 182 includes:

[0187] A first conversion subunit 1821 is configured to perform dimension conversion on the target attention hidden state using a first fully connected layer to generate a first converted hidden state of the target dimension;

[0188] The second conversion subunit 1822 is configured to perform dimension conversion on the target language hidden state using a second fully connected layer to generate a second converted hidden state of the target dimension;

[0189] The state superposition subunit 1823 is used to perform state superposition processing on the first conversion hidden state and the second conversion hidden state to obtain a hidden state fusion feature.

[0190] The character prediction module 18 includes:

[0191] The preliminary prediction unit 185 is configured to obtain, in the language memory network, a historical language latent state corresponding to the first predicted character, perform feature fusion on the historical language latent state and the attention image features, and predict M candidate characters and a character matching probability corresponding to each candidate character; M is a positive integer;

[0192] The character determination unit 186 is configured to determine the candidate character with the greatest character matching probability among the M candidate characters as the second predicted character.

[0193] The input acquisition module 11 includes:

[0194] A feature acquisition unit 111 is used to acquire global features of a candidate box corresponding to a target image;

[0195] The embedding acquisition unit 112 is configured to extract features of the first predicted character associated with the target image to obtain a character embedding feature of the first predicted character;

[0196] The feature determination unit 113 is configured to obtain a character fusion feature of the first predicted character based on the global feature of the candidate box and the character embedding feature.

[0197] The feature acquisition unit 111 includes:

[0198] The local recognition subunit 1111 is used to obtain N initial candidate frames corresponding to the target image and identify candidate local features corresponding to each initial candidate frame from the target image; N is a positive integer;

[0199] The local fusion subunit 1112 is used to perform feature fusion on N candidate local features to obtain the candidate frame global features corresponding to the target image.

[0200] The feature determination unit 113 is specifically configured to:

[0201] Perform feature splicing on the global features of the candidate box and the character embedding features to obtain the character fusion features of the first predicted character; or,

[0202] The historical language latent state of the first predicted character is obtained, and feature splicing is performed on the historical language latent state, the global features of the candidate box, and the character embedding features to obtain the character fusion features of the first predicted character.

[0203] The embodiment of the present application provides an image detection device, which can be run in a computer device, obtain a character fusion feature of a first predicted character associated with a target image; obtain a target attention hidden state for predicting a second predicted character based on the character fusion feature, input the target attention hidden state into k candidate frame detection networks, and in the k candidate frame detection networks, sequentially identify target candidate frames in the target image based on the target attention hidden state; k is a positive integer; obtain target candidate image features corresponding to the target candidate frames respectively identified by the k candidate frame detection networks from the target image, generate candidate aggregate image features corresponding to the k target candidate image features; obtain global image features of the target image, add global image features to the candidate aggregate image features, obtain attention image features, and predict the second predicted character corresponding to the attention image features. Through the above process, k candidate frame detection networks are used to identify multiple target candidate frames with similar semantics from the target image, that is, multiple target candidate frames with similar semantics but different positions, perform feature fusion on the multiple target candidate frames, obtain relatively complete features, and obtain more complete position information, so that the second predicted character can be predicted based on the target candidate frame after feature fusion, thereby improving the accuracy of image positioning and improving the accuracy of image character prediction.

[0204] Further, see Figure 10 , Figure 10 Schematic diagram of an image detection device provided in an embodiment of the present application. The image detection device can be a computer program (including program code, etc.) running on a computer device. For example, the image detection device can be an application software; the device can be used to execute the corresponding steps of the method provided in an embodiment of the present application. Figure 10 As shown, the image detection device 1000 can be used for Figure 8 The computer device in the corresponding embodiment, specifically, the apparatus can comprise: a sample acquisition module 21, a sample prediction module 22, a sample candidate identification module 23, a sample aggregation module 24, a first sample prediction module 25, a second sample prediction module 26, and a model training module 27.

[0205] The sample acquisition module 21 is configured to acquire an image training sample and an image description sentence corresponding to the image training sample; the image description sentence comprises a first character label and a second character label.

[0206] The sample prediction module 22 is configured to predict a first sample character of the image training sample based on an initial character prediction model and acquire a sample character fusion feature of the first sample character.

[0207] The sample candidate identification module 23 is configured to acquire a sample attention hidden state for predicting a second sample character based on the sample character fusion feature in an initial attention memory network included in the initial character prediction model, input the sample attention hidden state into k initial candidate box detection networks, and identify target sample candidate boxes in the image training sample based on the sample attention hidden state in the k initial candidate box detection networks in turn; k is a positive integer.

[0208] The sample aggregation module 24 is configured to acquire target sample candidate image features corresponding to the target sample candidate boxes identified by the k initial candidate box detection networks from the image training sample, and generate a sample candidate aggregation image feature corresponding to the k target sample candidate image features.

[0209] The first sample prediction module 25 is configured to acquire a sample global image feature of the image training sample, add the sample global image feature in the sample candidate aggregation image feature to obtain a sample attention image feature, and predict a first sample prediction result corresponding to the sample attention image feature.

[0210] The second sample prediction module 26 is configured to acquire a sample language hidden state of the sample attention image feature, and predict a second sample prediction result corresponding to the image training sample according to the sample attention hidden state and the sample language hidden state.

[0211] The model training module 27 is configured to adjust parameters of the initial character prediction model according to the first sample prediction result, the second sample prediction result, and the second character label to obtain a target character prediction model.

[0212] The model training module 27 comprises:

[0213] A loss generating unit 271 is configured to generate a first loss function based on the first sample prediction result and the second character label, generate a second loss function based on the second sample prediction result and the second character label, and generate a third loss function based on the first sample prediction result and the second sample prediction result;

[0214] The parameter adjustment unit 272 is used to adjust the parameters of the initial character prediction model through the first loss function and the second loss function, and to adjust the parameters of the initial attention memory network in the initial character prediction model through the third loss function to obtain the target character prediction model.

[0215] An embodiment of the present application provides an image detection device, which, in a semantically consistent network, transfers the semantic information of the language latent state to the semantic information of the attention latent state to adjust the parameters of the attention memory network. Since the language latent state represents more semantic information of the currently predicted character, the semantic difference between the language latent state and the attention latent state is reduced, so that the trained target character prediction model can more accurately predict the second predicted character based on the first predicted character, while also improving the accuracy of image positioning (i.e., determination of the target candidate box).

[0216] See also Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 11 As shown, the computer device in the embodiment of the present application may include: one or more processors 1101, a memory 1102, and an input / output interface 1103. The processor 1101, the memory 1102, and the input / output interface 1103 are connected via a bus 1104. The memory 1102 is used to store a computer program, which includes program instructions. The input / output interface 1103 is used to receive and output data, such as for data exchange between the computer device and a terminal device, or for data exchange between various networks in a model; the processor 1101 is used to execute the program instructions stored in the memory 1102.

[0217] When the processor 1101 is located in a computer device that performs image detection, it may perform the following operations:

[0218] Obtaining a character fusion feature of a first predicted character associated with the target image;

[0219] A target attention hidden state for predicting the second predicted character is obtained based on the character fusion feature, and the target attention hidden state is input into k candidate box detection networks. In the k candidate box detection networks, target candidate boxes in the target image are identified based on the target attention hidden state in turn; k is a positive integer;

[0220] Obtain target candidate image features corresponding to the target candidate frames respectively identified by the k candidate frame detection networks from the target image, and generate candidate aggregate image features corresponding to the k target candidate image features;

[0221] A global image feature of the target image is obtained, the global image feature is added to the candidate aggregate image feature to obtain an attention image feature, and a second predicted character corresponding to the attention image feature is predicted.

[0222] When the processor 1101 is located in a computer device for model training, it can perform the following operations:

[0223] Obtain an image training sample and obtain an image description sentence corresponding to the image training sample; the image description sentence includes a first character label and a second character label;

[0224] Predicting a first sample character of the image training sample based on the initial character prediction model, and obtaining a sample character fusion feature of the first sample character;

[0225] In an initial attention memory network included in the initial character prediction model, a sample attention latent state for predicting a second sample character is obtained based on the sample character fusion feature, the sample attention latent state is input into k initial candidate box detection networks, and in the k initial candidate box detection networks, target sample candidate boxes in the image training sample are sequentially identified based on the sample attention latent state, where k is a positive integer;

[0226] Obtain target sample candidate image features corresponding to target sample candidate frames respectively identified by k initial candidate frame detection networks from image training samples, and generate sample candidate aggregate image features corresponding to the k target sample candidate image features;

[0227] Obtaining a sample global image feature of an image training sample, adding the sample global image feature to the sample candidate aggregate image feature to obtain a sample attention image feature, and predicting a first sample prediction result corresponding to the sample attention image feature;

[0228] Obtaining a sample language latent state of the sample attention image feature, and predicting a second sample prediction result corresponding to the image training sample based on the sample attention latent state and the sample language latent state;

[0229] According to the first sample prediction result, the second sample prediction result and the second character label, the parameters of the initial character prediction model are adjusted to obtain a target character prediction model.

[0230] In some feasible implementations, the processor 1101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0231] The memory 1102 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1101 and the input / output interface 1103. A portion of the memory 1102 may also include a non-volatile random access memory. For example, the memory 1102 may also store device type information.

[0232] In a specific implementation, the computer device can execute the following operations through its built-in functional modules: Figure 4 or Figure 8 For details on the implementation methods provided in each step, please refer to the Figure 4 or Figure 8 The implementation methods provided in each step are not repeated here.

[0233] The embodiment of the present application provides a computer device, including: a processor, an input and output interface, and a memory, wherein the processor obtains a computer program in the memory and executes the computer program. Figure 4The various steps of the method shown in are used to perform image detection operations. The embodiment of the present application realizes obtaining the character fusion feature of the first predicted character associated with the target image; obtaining the target attention hidden state for predicting the second predicted character based on the character fusion feature, inputting the target attention hidden state into k candidate frame detection networks, and in the k candidate frame detection networks, sequentially identifying the target candidate frames in the target image based on the target attention hidden state; k is a positive integer; obtaining the target candidate image features corresponding to the target candidate frames respectively identified by the k candidate frame detection networks from the target image, generating candidate aggregate image features corresponding to the k target candidate image features; obtaining the global image features of the target image, adding the global image features to the candidate aggregate image features, obtaining the attention image features, and predicting the second predicted character corresponding to the attention image features. Through the above process, k candidate frame detection networks are used to identify multiple target candidate frames with similar semantics from the target image, that is, multiple target candidate frames with similar semantics but not exactly the same positions, and performing feature fusion on the multiple target candidate frames to obtain relatively complete features, so as to obtain more complete position information, so that the second predicted character can be predicted based on the target candidate frame after feature fusion, thereby improving the accuracy of image positioning and improving the accuracy of image character prediction.

[0234] The present invention also provides a computer-readable storage medium storing a computer program suitable for being loaded and executed by the processor. Figure 4 or Figure 8 The image detection methods provided in each step are detailed in this Figure 4 or Figure 8 The implementation methods provided in each step will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the computer program can be deployed to be executed on one computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0235] The computer-readable storage medium can be the image detection device provided by any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or the memory of the computer device. The computer-readable storage medium can also be the external storage device of the computer device, such as the plug-in hard disk equipped on the computer device, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card) etc. Further, the computer-readable storage medium can also include both the internal storage unit of the computer device and the external storage device. The computer-readable storage medium is used to store other programs and data required for the computer program and the computer device. The computer-readable storage medium can also be used to temporarily store the data that have been output or will be output.

[0236] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 4 or Figure 8 The method provided in the various optional methods realizes the use of k candidate box detection networks to identify multiple target candidate boxes with similar semantics from the target image, that is, multiple target candidate boxes with similar semantics but not exactly the same positions, and performs feature fusion on the multiple target candidate boxes to obtain more complete features to obtain more complete position information, so that the second predicted character can be predicted based on the target candidate box after feature fusion, thereby improving the accuracy of image positioning and the accuracy of image character prediction.

[0237] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0238] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in this description according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0239] The methods and related devices provided by the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided by the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable image detection device to generate a machine, so that the instructions executed by the processor of the computer or other programmable image detection device generate instructions for implementing the process in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable image detection device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the function specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable image detection device, so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 The flow or flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0240] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0241] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0242] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image detection method, characterized in that: The method comprises: Acquire global features of a candidate box corresponding to a target image, perform feature extraction on a first predicted character associated with the target image, and obtain character embedding features of the first predicted character; Obtaining a character fusion feature of the first predicted character according to the candidate box global feature and the character embedding feature; Obtaining a historical attention latent state for predicting the first predicted character, performing feature fusion on the historical attention latent state and character fusion features based on an attention memory network to obtain a target attention latent state for predicting the second predicted character, inputting the target attention latent state into k candidate box detection networks, and sequentially identifying target candidate boxes in the target image based on the target attention latent state in the k candidate box detection networks; k is a positive integer; and the first predicted character is the predicted character obtained at the previous moment of predicting the second predicted character; Obtaining target candidate image features corresponding to the target candidate frames respectively identified by the k candidate frame detection networks from the target image, and generating candidate aggregate image features corresponding to the k target candidate image features; A global image feature of the target image is obtained, the global image feature is added to the candidate aggregate image feature to obtain an attention image feature, and the second predicted character corresponding to the attention image feature is predicted.

2. The method according to claim 1, wherein In the k candidate box detection networks, identifying the target candidate boxes in the target image based on the target attention hidden state in sequence includes: In the i-th candidate box detection network, (N-i+1) initial candidate boxes and initial candidate features of each initial candidate box in the target image are identified based on the target attention hidden state; N is a positive integer, and i is a positive integer less than or equal to k; the (N-i+1) initial candidate boxes refer to the initial candidate boxes among the N initial candidate boxes, excluding the target candidate boxes respectively identified by the (i-1) candidate box detection networks; Perform feature dimension conversion on the (N-i+1) initial candidate features to obtain the candidate probabilities corresponding to the (N-i+1) initial candidate frames; Based on the candidate probabilities corresponding to the (N-i+1) initial candidate frames, a target candidate frame is determined from the (N-i+1) initial candidate frames; the target candidate frames corresponding to the k candidate frame detection networks are different from each other.

3. The method according to claim 1, wherein Generating candidate aggregated image features corresponding to k target candidate image features includes: Perform feature fusion on the k target candidate image features and the target attention hidden state respectively to obtain k initial candidate weights; Normalizing the k initial candidate weights to obtain target candidate weights corresponding to the k target candidate image features; Based on the target candidate weights respectively corresponding to the k target candidate image features, the k target candidate image features are weightedly summed to obtain a candidate aggregate image feature.

4. The method according to claim 1, wherein The predicting the second predicted character corresponding to the attention image feature includes: In the language memory network, feature prediction processing is performed on the attention image features and the target attention latent state to obtain a first prediction result and a target language latent state; Performing feature fusion on the target attention latent state and the target language latent state to obtain a latent state fusion feature; Predicting the hidden state fusion feature to obtain a second prediction result; The first prediction result and the second prediction result are integrated to obtain the second predicted character.

5. The method according to claim 4, wherein The step of fusing the target attention latent state and the target language latent state to obtain a latent state fusion feature includes: Using a first fully connected layer to perform dimension conversion on the target attention hidden state to generate a first converted hidden state of the target dimension; Performing dimension conversion on the target language hidden state using a second fully connected layer to generate a second converted hidden state of the target dimension; Performing state superposition processing on the first conversion latent state and the second conversion latent state to obtain a latent state fusion feature.

6. The method according to claim 1, wherein The predicting the second predicted character corresponding to the attention image feature includes: In the language memory network, a historical language latent state corresponding to the first predicted character is obtained, and feature fusion is performed on the historical language latent state and the attention image features to predict M candidate characters and a character matching probability corresponding to each candidate character; M is a positive integer; The candidate character with the greatest character matching probability among the M candidate characters is determined as the second predicted character.

7. The method according to claim 1, wherein The obtaining of the global features of the candidate frame corresponding to the target image includes: Obtain N initial candidate frames corresponding to the target image, and identify candidate local features corresponding to each initial candidate frame from the target image; N is a positive integer; Feature fusion is performed on N candidate local features to obtain the candidate frame global features corresponding to the target image.

8. The method according to claim 1, wherein The obtaining, based on the candidate box global feature and the character embedding feature, a character fusion feature of the first predicted character includes: Perform feature splicing on the candidate box global feature and the character embedding feature to obtain a character fusion feature of the first predicted character; or A historical language latent state of the first predicted character is obtained, and feature splicing is performed on the historical language latent state, the candidate box global feature, and the character embedding feature to obtain a character fusion feature of the first predicted character.

9. An image detection method, characterized in that: The method comprises: Obtain an image training sample and obtain an image description sentence corresponding to the image training sample; the image description sentence includes a first character label and a second character label; Predicting a first sample character of the image training sample based on an initial character prediction model, obtaining a global feature of a candidate box corresponding to the image training sample, performing feature extraction on the first sample character associated with the image training sample, and obtaining a character embedding feature of the first sample character; Obtaining a sample character fusion feature of the first sample character according to the candidate box global feature corresponding to the image training sample and the character embedding feature of the first sample character; In an initial attention memory network included in the initial character prediction model, a sample attention latent state for predicting a second sample character is obtained based on the sample character fusion feature, the sample attention latent state is input into k initial candidate box detection networks, and in the k initial candidate box detection networks, target sample candidate boxes in the image training sample are identified in sequence based on the sample attention latent state; k is a positive integer; Obtaining target sample candidate image features corresponding to the target sample candidate frames respectively identified by the k initial candidate frame detection networks from the image training samples, and generating sample candidate aggregate image features corresponding to the k target sample candidate image features; Acquire a sample global image feature of the image training sample, add the sample global image feature to the sample candidate aggregate image feature to obtain a sample attention image feature, and predict a first sample prediction result corresponding to the sample attention image feature; Obtaining a sample language latent state of the sample attention image feature, and predicting a second sample prediction result corresponding to the image training sample based on the sample attention latent state and the sample language latent state; According to the first sample prediction result, the second sample prediction result and the second character label, the parameters of the initial character prediction model are adjusted to obtain a target character prediction model.

10. The method according to claim 9, wherein The step of adjusting parameters of the initial character prediction model according to the first sample prediction result, the second sample prediction result, and the second character label to obtain a target character prediction model includes: Generate a first loss function based on the first sample prediction result and the second character label, generate a second loss function based on the second sample prediction result and the second character label, and generate a third loss function based on the first sample prediction result and the second sample prediction result; The parameters of the initial character prediction model are adjusted using the first loss function and the second loss function, and the parameters of the initial attention memory network in the initial character prediction model are adjusted using the third loss function to obtain a target character prediction model.

11. A computer device, characterized in that: Includes processor, memory, input and output interfaces; The processor is connected to the memory and the input / output interface, respectively, wherein the input / output interface is used to receive and output data, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method described in any one of claims 1 to 8, or executes the method described in any one of claims 9 to 10.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 8, or executes the method according to any one of claims 9 to 10.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented, or the method according to any one of claims 9 to 10 is executed.

Images