Visual inspection method and device, equipment, storage medium and program product
By segmenting and adjusting the pictures, extracting local and global features, and combining text features for fusion, the problem of poor detection of edge objects in the prior art is solved, and the model's perception of complex details is improved.
Patent Information
- Application Number
- CN202411237521.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing multimodal visual detection algorithms have poor detection effects on image edge objects, making it difficult to effectively perceive complex details in images.
By segmenting and sizing the detected images, multiple local images and global images are generated, and input them into the picture encoder to extract features. Combining text features using an alignment decoder with cross attention mechanism, we determine the text tag object in the picture. location information in.
It improves the model's perception of complex details of images and improves the accuracy of image edge object detection.
Smart Images

Figure CN119091445A_ABST
Abstract
Description
Background Art
[0002] With the development of multimodal large models in the field of visual detection, in order to overcome the limitations of traditional visual models in understanding and interpreting complex scenes, a deeper and more comprehensive understanding of the environment and content is achieved by integrating multiple types of data modalities, such as images, text, etc. However, the multimodal visual detection algorithm based on related technologies has poor object detection effect on the edge of the picture, which reduces the model's ability to perceive complex details in the graphics.
[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0004] The purpose of the present disclosure is to provide a visual detection method, a visual detection device, an electronic device, a computer-readable storage medium and a computer program product, which at least to some extent overcome the problem that the visual detection algorithm in the related art has poor detection effect on objects with unknown edges in the picture.
[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0006] According to one aspect of the present disclosure, a visual detection method is provided, including: segmenting a picture to be detected to obtain a plurality of local images, wherein the plurality of local images have the same size; resizing the picture to be detected to obtain a global image with the same size as the local images; inputting the plurality of local images and the global image into a picture encoder to obtain picture features, wherein the picture features include a plurality of local image features corresponding to the plurality of local images and a global image feature corresponding to the global image; inputting a text label into a text encoder to output text features; inputting the picture features and the text features into an alignment decoder for fusion, and determining the position information of the object corresponding to the text label in the picture to be detected.
[0007] In one embodiment of the present disclosure, the alignment decoder is constructed based on a cross-attention mechanism; wherein, the image features and the text features are input into the alignment decoder for fusion, and the position information of the object corresponding to the text label in the image to be detected is determined, including: using the cross-attention mechanism to fuse the image features and the text features, and determining the position information of the object corresponding to the text label in the image to be detected.
[0008] In one embodiment of the present disclosure, the cross-attention mechanism is used to fuse the image features and the text features to determine the position information of the object corresponding to the text label in the image to be detected, including: determining the attention weight of the image feature according to the image features and the text features; determining the position information of the object corresponding to the text label in the image to be detected according to the attention weight of the image feature and the text label.
[0009] In one embodiment of the present disclosure, the step of inputting a text tag into a text encoder and outputting a text feature includes: performing an extended tag description on the text tag through a prompting project to obtain an extended text tag; and inputting the extended text tag into the text encoder and outputting the text feature.
[0010] In one embodiment of the present disclosure, the image to be detected is segmented to obtain multiple local images, including: dividing the width W of the image to be detected into x parts in equal proportion, dividing the height H of the image to be detected into y parts in equal proportion, and obtaining n local images, each of which has a width of W / x and a height of H / y, wherein x≥1, y≥1, n=x×y, and n≥2.
[0011] In one embodiment of the present disclosure, the dimension of the global image is (W / x, H / y, C), where C is the number of channels, and the inputting the multiple local images and the global image into a picture encoder to obtain picture features includes: splicing the multiple local images and the global image to obtain an image matrix, the dimension of the image matrix is (W / x, H / y, C(x×y+1)); inputting the image matrix into the picture encoder to obtain the picture features, wherein the picture encoder is constructed based on at least one of a convolutional neural network, an autoencoder, a recurrent neural network, and a long short-term memory network.
[0012] According to another aspect of the present disclosure, a visual detection device is provided, including: a first image processing module, used to segment a to-be-detected image to obtain a plurality of local images, wherein the plurality of local images have the same size; a second image processing module, used to resize the to-be-detected image to obtain a global image with the same size as the local images; an image feature extraction module, used to input the plurality of local images and the global image into an image encoder to obtain image features, wherein the image features include a plurality of local image features corresponding to the plurality of local images and a global image feature corresponding to the global image; a text feature extraction module, used to input a text label into a text encoder to output a text feature; and a feature fusion module, used to input the image features and the text features into an alignment decoder for fusion, so as to achieve visual detection of the to-be-detected image.
[0013] According to another aspect of the present disclosure, an electronic device is provided, including: a processor; and a memory for storing executable instructions of the processor; the processor is configured to perform the above-mentioned visual detection method by executing the executable instructions.
[0014] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned visual inspection method is implemented.
[0015] According to another aspect of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned visual inspection method is implemented.
[0016] The visual detection method provided by the embodiment of the present disclosure divides the image to be detected to obtain multiple local images, and the multiple local images have the same size; adjusts the size of the image to be detected to obtain a global image with the same size as the local image; inputs the multiple local images and the global image into an image encoder to obtain image features, and the image features include multiple local image features corresponding to the multiple local images and global image features corresponding to the global image; inputs text labels into a text encoder to output text features; inputs the image features and text features into an alignment decoder for fusion, and determines the position information of the object corresponding to the text label in the image to be detected. The present disclosure can better extract the detail features of high-resolution images by segmenting image blocks and adjusting the image size, and improve the model's ability to perceive complex details in images.
[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0019] Figure 1 A schematic diagram showing an exemplary application system architecture of a visual inspection method in an embodiment of the present disclosure.
[0020] Figure 2 A schematic flow chart of a visual detection method in an embodiment of the present disclosure is shown.
[0021] Figure 3 A schematic diagram showing an example of a visual detection method in an embodiment of the present disclosure.
[0022] Figure 4 A schematic flow chart of another visual detection method in an embodiment of the present disclosure is shown.
[0023] Figure 5 A schematic diagram showing another example of a visual detection method in an embodiment of the present disclosure.
[0024] Figure 6 A schematic diagram of a visual detection device in an embodiment of the present disclosure is shown.
[0025] Figure 7 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0026] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0027] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0028] Figure 1 FIG. 2 shows an exemplary application system architecture diagram to which the visual detection method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .
[0029] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.
[0030] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.
[0031] The terminal device 101 may be any electronic device, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a wearable device, an augmented reality device, a virtual reality device, an image acquisition device, etc. The image acquisition device may be a camera, etc.
[0032] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.
[0033] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received data such as the image to be processed, and feed back the processing results to the terminal device 101.
[0034] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.
[0035] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.
[0036] With the rapid development of 5G technology and artificial intelligence, video has become ubiquitous in work and life. In current video application scenarios, especially video surveillance scenarios, the location information of surveillance cameras is missing. It is impossible to obtain the specific information contained in the pictures collected by surveillance cameras. The current practice is to watch surveillance videos by people and manually mark the location information of cameras and the environment and content of the images taken by cameras one by one.
[0037] The 5G era has spawned a large number of video-oriented applications, such as smart and safe cities, the Internet, driverless cars, video surveillance, and other image- and video-based content. The application scenarios are very broad, and image- and video-related applications will become one of the main sources of incremental traffic in the 5G and post-5G eras.
[0038] In the related art, multimodal visual detection algorithms usually directly input images into the detection network, output detection results, and determine the position and category of the detection target by extracting image features. In the multimodal visual detection method, the prompt text is encoded by a text encoder to obtain text features, and the text features are deeply fused with the image features obtained by the image encoder to obtain the position information of the object corresponding to the text for positioning. However, when encoding through the image encoder, the edge image information of the image is easily lost, and the detection effect of objects in the center of the image is better, while the detection effect of objects at the edge of the image is poor.
[0039] In order to at least solve the above technical problems, the visual detection method provided by the embodiment of the present disclosure divides the image to be detected to obtain multiple local images, and the multiple local images have the same size; adjusts the size of the image to be detected to obtain a global image with the same size as the local image; inputs the multiple local images and the global image into the image encoder to obtain image features, and the image features include multiple local image features corresponding to the multiple local images and global image features corresponding to the global image; inputs the text label into the text encoder to output the text feature; inputs the image feature and the text feature into the alignment decoder for fusion, and determines the position information of the object corresponding to the text label in the image to be detected. The present disclosure can better extract the detail features of high-resolution images by segmenting the image blocks and adjusting the image size, and improve the model's ability to perceive complex details in the image.
[0040] Under the above system architecture, an embodiment of the present disclosure provides a visual inspection method, which can be executed by any electronic device with computing and processing capabilities.
[0041] In some embodiments, the visual detection method provided in the embodiments of the present disclosure can be executed by a terminal device of the above-mentioned system architecture; in other embodiments, it can also be executed by a server of the above-mentioned system architecture; in other embodiments, it can also be implemented by the terminal device and server in the above-mentioned system architecture through interaction.
[0042] The visual inspection method in this example implementation will be described in more detail below with reference to the accompanying drawings and examples.
[0043] like Figure 2 As shown, according to a visual inspection method of an embodiment of the present disclosure, the method comprises the following steps:
[0044] S202 , segmenting the image to be detected to obtain multiple local images, wherein the sizes of the multiple local images are the same.
[0045] In one embodiment, the image to be detected may be an image acquired by an image acquisition device, or may be a video frame determined by a video shot by the image acquisition device. The image acquisition device may store the acquired image to be detected in a database, so that the server extracts the corresponding image to be detected from the database.
[0046] The image to be detected may include one or more objects to be detected, and the objects to be detected may be people, animals, objects, etc., which is not limited in the present disclosure.
[0047] In the present disclosure, the purpose of visual inspection of the image to be inspected is to identify the object to be inspected in the image to be inspected, and the identified information includes but is not limited to the type of the object to be inspected, the position of the object to be inspected in the image to be inspected, etc.
[0048] In S202, the image to be detected can be segmented in the following manner: the width W of the image to be detected is divided into x parts in equal proportion, and the height H of the image to be detected is divided into y parts in equal proportion, to obtain n local images, each of which has a width of W / x and a height of H / y, wherein x≥1, y≥1, n=x×y, and n≥2.
[0049] For example, if the width W of the image to be detected is 800 and the height H is 450, if the width of the image to be detected is divided into 2 parts (x=2) and the height H is divided into 2 parts (y=2), then 4 (n=2×2=4) partial images can be obtained, and the size of each partial image is: width 400 and height 225. The 4 partial images are numbered respectively, and recorded as Pi, i={0, 1, ...3}.
[0050] For example, for the above-mentioned image to be detected, if the width of the image to be detected is divided into 4 parts (x=4) in equal proportion, and the height H is divided into 3 parts (y=3) in equal proportion, then n=12 partial images can be obtained, and the size of each partial image is: width is 200, height is 150. The partial images are numbered respectively, and recorded as Pi, i={0, 1, ...11}.
[0051] In one embodiment, a fully convolutional network (FCN), U-Net (a fully convolutional neural network composed of an encoder and a decoder), DeepLab (a semi-supervised model that combines a deep convolutional neural network and a probabilistic graph model), etc. can be used to achieve the cutting of the image to be detected.
[0052] It should be noted that the cutting method and number of cuts of the image to be detected can be determined according to actual needs. For example, when the width direction of the image to be detected includes more objects to be detected, the number of cuts in the width direction can be appropriately increased; when the height direction of the image to be detected includes angle objects to be detected, the number of cuts in the height direction can be appropriately increased.
[0053] S204, resizing the image to be detected to obtain a global image with the same size as the local image.
[0054] In one embodiment, a neural network may be used to adjust the size of the image to be detected, that is, the width of the image to be detected is reduced to 1 / x of the original, and the height is reduced to 1 / y of the original, to obtain a global image.
[0055] The dimension of the global image can be expressed as (W / x, H / y, C), where W / x represents the number of pixels in the horizontal direction, H / y represents the number of pixels in the vertical direction, and C represents the number of channels. For example, the number of channels for a black and white image is C=1, and the number of channels for an RGB color image is C=3. For an RGB color image, the value ranges of each channel are: R is between 0 and 255, G is between 0 and 255, and B is between 0 and 255.
[0056] In one embodiment, the global images may be numbered and recorded as Pn.
[0057] S206. Input the multiple local images and the global image into a picture encoder to obtain picture features, where the picture features include multiple local image features corresponding to the multiple local images and global image features corresponding to the global image.
[0058] In one embodiment, the image encoder is used to process the input image, extract image features and classify or recognize the image. The image encoder can be constructed based on at least one of a convolutional neural network (CNN), an autoencoder (Autoencoder), a recurrent neural network (RNN), and a long short-term memory network (LSTM).
[0059] Convolutional neural networks are deep learning models mainly used for image processing. They extract image features through multiple convolutional layers and pooling layers, and perform classification or regression through fully connected layers. They have powerful feature extraction capabilities.
[0060] An autoencoder is an unsupervised learning model consisting of an encoder and a decoder. The encoder compresses the input data into a low-dimensional code, and the decoder restores the low-dimensional code to reconstructed data. The autoencoder can be used for tasks such as data dimensionality reduction and feature extraction.
[0061] Recurrent neural networks are neural network models suitable for processing sequence data. By cyclically linking the output of the hidden layer as the input of the next time, they can process the information of variable-length input sequences, have memory functions, and capture the time dependencies in the sequence.
[0062] The Long Short-Term Memory Network is an improved RNN model that solves the problem of small or exploding gradients in RNN. It introduces memory units and gating mechanisms to better handle long-term dependencies.
[0063] It should be noted that the image feature extraction task can be performed by combining multiple CNNs, autoencoders, RNNs, and LSTMs.
[0064] In one embodiment, each local image Pi can determine a corresponding local image feature Vi, i = {0, 1, ... n-1}. The global image Pn corresponds to at least one global image feature Vn. The image feature Vi, i = {0, 1, ... n} can be determined according to the actual situation, and the present disclosure does not make specific limitations on this.
[0065] In one embodiment, in the above S206, multiple local images and the global image are input into a picture encoder to obtain picture features, including: splicing multiple local images and the global image to obtain an image matrix, the dimension of the image matrix is (W / x, H / y, C(x×y+1)); inputting the image matrix into the picture encoder to obtain picture features.
[0066] S208: Input the text label into the text encoder and output the text feature.
[0067] In one embodiment, the text label may be the object type or object name corresponding to the object to be detected, and is used to mark the object to be detected. The text label is used to describe information such as the name or type of the object to be detected that may be included in the corresponding scene in the image to be detected.
[0068] The above text encoder can convert the input text label into a high-dimensional vector representation to capture the semantic information in the text label. The text encoder converts the input text through a series of hidden layers, extracts semantic information at different levels, and obtains text features for subsequent processing and analysis.
[0069] In one embodiment, the text encoder can be built based on the Transformer model.
[0070] In S208, the text label may be extended by a prompting project to obtain an extended text label, and the extended text label may be input into a text encoder to output a text feature.
[0071] Hint engineering, also known as contextual hints, is a method of guiding the behavior of a large language model (LLM) toward a specific result by updating the model's weights or parameters. It is the process of effectively communicating with artificial intelligence (AI) about the desired result. The text label can be expanded into a descriptive sentence, "The pixels of CLASS_NAME", for example, "The pixels of sofa", "The pixels of desk", etc.
[0072] The text feature can be represented by T, T = {T 1 , T2,…Tk}, Tj is the j-th text feature.
[0073] It should be noted that the number k of text features may be the same as or different from the number of image features, and the present disclosure does not make any specific limitation on this.
[0074] S210: Input the image features and text features into an alignment decoder for fusion, and determine the position information of the object corresponding to the text label in the image to be detected.
[0075] In one embodiment, the alignment encoder can be constructed based on the cross-attention mechanism. In the Transformer architecture, the cross-attention mechanism refers to the cross-attention layer between the encoder and the decoder, in which the decoder adjusts the attention of the encoder's output to obtain encoder information related to the current decoding position. In the encoder-decoder architecture of the Transformer architecture, the encoder is responsible for encoding the input sequence into a series of feature vectors (for example, the text features obtained by the above-mentioned text encoder, and the picture features obtained by the picture encoder), and the decoder is responsible for gradually generating the output sequence (that is, the position information of the object corresponding to the text feature in the picture to be detected) according to the above-mentioned feature vectors (text features and picture features).
[0076] In one embodiment, the above S210 inputs the image features and the text features into the alignment decoder for fusion, and determines the position information of the object corresponding to the text features in the image to be detected, including: using a cross-attention mechanism to fuse the image features and the text features, and determining the position information of the object corresponding to the text label in the image to be detected.
[0077] The calculation process of the cross-attention mechanism includes:
[0078] Encoder input, which can be expressed as enc_inputs, of size (batch_size, seq_len_enc, hidden_dim);
[0079] Decoder input, which can be expressed as dec_inputs of size (batch_size, seq_len_dec, hidden_dim);
[0080] Each position of the decoder generates a query vector query, such as the above image features, which is used to calculate the attention weights at all positions of the encoder;
[0081] All positions of the encoder generate a set of key vectors keys and value vectors values. The key vector keys can be the above text vectors, and the value vector values can be the above text labels.
[0082] The query vector query and the key vector keys can be used to perform a dot product operation, and the attention weight can be determined by the softmax function;
[0083] Multiply the attention weight by the value vector and sum the results to get the encoder's adjusted output, which is the position information of the object corresponding to the text feature in the image to be detected.
[0084] In some embodiments, the above process can be understood as: the workflow of the model generalization system in intelligent video surveillance can be divided into data collection (text labels, images to be processed), data analysis (generating text features and image features), and model reasoning (i.e. fusing image features and text features to determine the location information of the object corresponding to the text label in the image to be detected).
[0085] It should be noted that the alignment encoder may also be implemented in other ways, which is not specifically limited in the present disclosure.
[0086] like Figure 3 As shown, in one embodiment, the image to be detected is as follows Figure 3 For the indoor picture, four local image patches are obtained by cutting, the picture to be detected is resized to obtain the global image, the global image and the four local images are input into the picture encoder to obtain image features, which include global image features corresponding to the global image and four local image features corresponding to the four local images; the text labels (such as sofa, desk, etc.) are label-expanded through the prompt engineering, and the expanded text labels are input into the text encoder to obtain text features, and the text features and picture features are input into the alignment decoder for fusion to determine the position information of the object corresponding to the text label in the picture to be detected.
[0087] It should be noted that the order of the image feature extraction step and the text feature extraction step is not specifically limited.
[0088] In the embodiments of the present disclosure, the image to be detected is segmented to obtain multiple local images, and the multiple local images have the same size; the image to be detected is resized to obtain a global image with the same size as the local image; the multiple local images and the global image are input into an image encoder to obtain image features, and the image features include multiple local image features corresponding to the multiple local images and global image features corresponding to the global image; the text label is input into a text encoder to output text features; the image features and the text features are input into an alignment decoder for fusion to determine the position information of the object corresponding to the text label in the image to be detected. The present disclosure can better extract the detail features of high-resolution images by segmenting image blocks and adjusting the image size, and improve the model's ability to perceive complex details in images.
[0089] Figure 4 Another visual inspection method flow chart in the embodiment of the present disclosure is shown. Figure 2On the basis of the embodiment, S210 is further refined into S212-S214 to limit the position information of the object corresponding to the text feature obtained by fusing the image feature and the text feature through the cross attention mechanism in the image to be detected. Figure 4 As shown, in one embodiment, the method includes S202 to S208, S212 to S214. Wherein:
[0090] S212, determining the attention weight of the image feature according to the image feature and the text feature;
[0091] S214: Determine the position information of the object corresponding to the text label in the image to be detected according to the attention weight of the image feature and the text label.
[0092] It should be noted that the specific implementation method of S202 to S208 in the embodiment of the present disclosure is the same as the implementation method of S202 to S208 in the aforementioned embodiment, and will not be repeated here.
[0093] In one embodiment, the above-mentioned determining the attention weight of the image feature based on the image feature and the text feature includes: determining a first function value that maps the image feature to a projection head of the same dimension as the text feature; determining a second function value that maps the text feature to a projection head of the same dimension as the image feature; and obtaining the weight of the image feature based on the first function value and the second function value.
[0094] like Figure 5 As shown, in one embodiment, the weight of the i-th image feature is i Determined using formula 1:
[0095] Weigth i =Softmax(g v (T) w (V i )) (Formula 1)
[0096] Among them, the function g(·) represents the projection head that maps the input to the same dimension, and the function g v (·) represents the projection head that maps text features to the same dimension as the image features. The function g w (·) represents the projection head that maps the image features to the same dimension as the text features; Softmax(·) is the normalized word count function, which is a generalization of the logical function and can present the results of multi-classification in the form of probability; T represents all text features.
[0097] In S214, the output of the alignment decoder may be determined by Formula 2:
[0098]
[0099] Where Decoder represents the alignment decoder; T 1 ,...,T k Represent the text label, x and y represent the width and height of the image to be detected, respectively; V i Represents the i image features.
[0100] In the disclosed embodiment, adaptive weights are used to reallocate weights to image features, which are then fused with text features to improve the accuracy of visual detection.
[0101] In order to deepen the understanding of the visual inspection method disclosed in the present invention, it is described below with reference to specific examples.
[0102] Example 1:
[0103] Step 1: Cut the image Img into multiple local images evenly.
[0104] The image to be processed has a width of 800 and a height of 450. The width is divided into two parts in equal proportion, and the height is divided into two parts in equal proportion, resulting in four partial images, each of which has a size of 400 in width and 225 in height.
[0105] Step 2: Resize the image to be processed to the local image size to obtain the global image (400, 225, 3).
[0106] Step 3: Concatenate the local image and the global image to obtain an image matrix with a dimension of (400, 225, 15), input it into the image encoder, and obtain image features, which include local image features and global image features.
[0107] Step 4: Extend the text label description through the prompt project.
[0108] The pixels of sofa.
[0109] The pixels of desk. ...
[0111] Step 5: Input the extended tag description into the text encoder to obtain text features.
[0112] Step 6: Input the text features and image features into the alignment decoder for fusion. The alignment decoder consists of two cross-attention mechanisms, in which the importance of image features is assigned through adaptive weights.
[0113] The attention weight of the i-th image feature is calculated by formula 3:
[0114] Weigth i =Softmax(gv (T) w (V i )) (Formula 3)
[0115] Among them, i = {1,2,…5}.
[0116] The output of the alignment decoder is determined by Equation 4:
[0117]
[0118] Decoder represents the alignment decoder, T refers to all text features, T 1 ,...,T k Represents each text label feature, V i Represents the i-th image feature. Function g v and g w is a projection head that maps the input to the same dimension.
[0119] Example 2:
[0120] Step 1: Cut the image Img into multiple local images evenly.
[0121] The image to be processed has a width of 800 and a height of 450. The width is divided into 4 parts in equal proportion, and the height is divided into 3 parts in equal proportion, resulting in 12 partial images. The size of each partial image is: 200 in width and 150 in height.
[0122] Step 2: Resize the image to be processed to the local image size to obtain the global image (200, 150, 3).
[0123] Step 3: Concatenate the local image and the global image to obtain an image matrix with a dimension of (200, 150, 36), input it into the image encoder, and obtain image features, which include local image features and global image features.
[0124] Step 4: Extend the text label description through the prompt project.
[0125] The pixels of sofa.
[0126] The pixels of desk. ...
[0128] Step 5: Input the extended tag description into the text encoder to obtain text features.
[0129] Step 6: Input the text features and image features into the alignment decoder for fusion. The alignment decoder consists of two cross-attention mechanisms, in which the importance of image features is assigned through adaptive weights.
[0130] The attention weight of the i-th image feature is calculated by formula 5:
[0131] Weigth i =Softmax(g v (T) w (V i )) (Formula 5)
[0132] Among them, i = {1,2,…12}.
[0133] The output of the alignment decoder is determined by Equation 4:
[0134]
[0135] Decoder represents the alignment decoder, T refers to all text features, T 1 ,...,T k Represents each text label feature, V i Represents the i-th image feature. Function g v and g w is a projection head that maps the input to the same dimension.
[0136] Based on the same inventive concept, the present disclosure also provides a visual inspection device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.
[0137] Figure 6 A schematic diagram of a visual detection device in an embodiment of the present disclosure is shown. Figure 6 As shown, the device comprises:
[0138] A first image processing module 610 is used to segment the image to be detected to obtain multiple local images, where the sizes of the multiple local images are the same;
[0139] A second image processing module 620 is used to resize the image to be detected to obtain a global image with the same size as the local image;
[0140] The picture feature extraction module 630 is used to input the multiple local images and the global image into the picture encoder to obtain picture features, where the picture features include multiple local image features corresponding to the multiple local images and global image features corresponding to the global image;
[0141] A text feature extraction module 640, used to input text labels into a text encoder and output text features;
[0142] The feature fusion module 650 is used to input the image features and the text features into the alignment decoder for fusion, so as to realize the visual detection of the image to be detected.
[0143] It should be noted that the first image processing module 610, the second image processing module 620, the image feature extraction module 630, the text feature extraction module 640 and the feature fusion module 650 correspond to S202 to S210 in the method embodiment, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above method embodiment. It should be noted that the above modules as part of the device can be executed in a computer system such as a set of computer executable instructions.
[0144] In one embodiment, the alignment decoder is constructed based on a cross-attention mechanism; wherein the feature fusion module 650 is used to fuse the image features and the text features using the cross-attention mechanism to determine the position information of the object corresponding to the text label in the image to be detected.
[0145] In one embodiment, the feature fusion module 650 is used to determine the attention weight of the image feature according to the image feature and the text feature; and determine the position information of the object corresponding to the text label in the image to be detected according to the attention weight of the image feature and the text label.
[0146] In one embodiment, the text feature extraction module 640 is used to perform extended label description on the text label through a prompting process to obtain an extended text label; input the extended text label into a text encoder to output text features.
[0147] In one embodiment, the first image processing module 610 is used to divide the width W of the image to be detected into x parts in equal proportion, and divide the height H of the image to be detected into y parts in equal proportion, to obtain n local images, each of which has a width of W / x and a height of H / y, wherein x≥1, y≥1, n=x×y, and n≥2.
[0148] In one embodiment, the dimension of the global image is (W / x, H / y, C), where C is the number of channels; the image feature extraction module 630 is used to splice multiple local images and the global image to obtain an image matrix, and the dimension of the image matrix is (W / x, H / y, C(x×y+1)); the image matrix is input into the image encoder to obtain image features, wherein the image encoder is constructed based on at least one of a convolutional neural network, an autoencoder, a recurrent neural network, and a long short-term memory network.
[0149] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".
[0150] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present disclosure is described. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0151] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710).
[0152] Among them, the storage unit stores a program code, and the program code can be executed by the processing unit 710, so that the processing unit 710 executes the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 710 can execute the following steps of the above method embodiment: segmenting the image to be detected to obtain multiple local images, and the multiple local images have the same size; resizing the image to be detected to obtain a global image with the same size as the local image; inputting the multiple local images and the global image into the image encoder to obtain image features, and the image features include multiple local image features corresponding to the multiple local images and global image features corresponding to the global image; inputting the text label into the text encoder to output the text feature; inputting the image feature and the text feature into the alignment decoder for fusion, and determining the position information of the object corresponding to the text label in the image to be detected.
[0153] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .
[0154] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0155] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0156] The electronic device 700 may also communicate with one or more external devices 740 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0157] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0158] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer program product, which includes: a computer program, which implements the above-mentioned visual inspection method when executed by a processor.
[0159] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. The computer-readable storage medium stores a program product capable of implementing the above-mentioned method of the present disclosure. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.
[0160] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0161] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0162] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0163] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).
[0164] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0165] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0166] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0167] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.
Claims
1. A visual inspection method, characterized in that: include: Segment the image to be detected to obtain multiple local images, wherein the multiple local images have the same size; Resizing the image to be detected to obtain a global image with the same size as the local image; Inputting the multiple local images and the global image into a picture encoder to obtain picture features, wherein the picture features include multiple local image features corresponding to the multiple local images and a global image feature corresponding to the global image; Input the text label into the text encoder and output the text feature; The image features and the text features are input into an alignment decoder for fusion, and the position information of the object corresponding to the text label in the image to be detected is determined.
2. The visual inspection method according to claim 1, characterized in that: The alignment decoder is built based on a cross-attention mechanism; The step of inputting the image features and the text features into an alignment decoder for fusion to achieve visual detection of the image to be detected includes: The image features and the text features are fused using a cross-attention mechanism to determine the position information of the object corresponding to the text label in the image to be detected.
3. The visual inspection method according to claim 2, characterized in that: The cross-attention mechanism is adopted to fuse the image features and the text features to determine the position information of the object corresponding to the text label in the image to be detected, including: Determining an attention weight of the image feature according to the image feature and the text feature; According to the attention weight of the image feature and the text label, the position information of the object corresponding to the text label in the image to be detected is determined.
4. The visual inspection method according to claim 1, characterized in that: The text label is input into the text encoder, and the text feature is output, including: Performing an extended label description on the text label through a prompting project to obtain an extended text label; The expanded text label is input into the text encoder, and the text feature is output.
5. The visual inspection method according to any one of claims 1 to 4, characterized in that: The image to be detected is segmented to obtain multiple local images, including: The width W of the image to be detected is divided into x parts in equal proportion, and the height H of the image to be detected is divided into y parts in equal proportion, to obtain n partial images, each of which has a width of W / x and a height of H / y, wherein x≥1, y≥1, n=x×y, and n≥2.
6. The visual inspection method according to claim 5, characterized in that: The dimension of the global image is (W / x, H / y, C), where C is the number of channels; The step of inputting the multiple local images and the global image into a picture encoder to obtain picture features includes: The multiple local images and the global image are stitched together to obtain an image matrix, where the dimension of the image matrix is (W / x, H / y, C(x×y+1)); The image matrix is input into the picture encoder to obtain the picture features, wherein the picture encoder is constructed based on at least one of a convolutional neural network, an autoencoder, a recurrent neural network, and a long short-term memory network.
7. A visual inspection device, characterized in that: include: A first image processing module is used to segment the image to be detected to obtain multiple local images, wherein the multiple local images have the same size; A second image processing module is used to resize the image to be detected to obtain a global image with the same size as the local image; A picture feature extraction module, configured to input the multiple local images and the global image into a picture encoder to obtain picture features, wherein the picture features include multiple local image features corresponding to the multiple local images and a global image feature corresponding to the global image; A text feature extraction module is used to input text labels into a text encoder and output text features; The feature fusion module is used to input the image features and the text features into the alignment decoder for fusion, so as to realize visual detection of the image to be detected.
8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to perform the visual inspection method according to any one of claims 1 to 6 by executing the executable instructions.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the visual inspection method according to any one of claims 1 to 6 is implemented.
10. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the visual inspection method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Polyp segmentation method and device and storage medium
CN111986204A
Image segmentation method and device, equipment and storage medium
CN114581462A
Image segmentation method, remote sensing image segmentation method and device
CN115761222A
Garbage detection method and system, electronic equipment and storage medium
CN117422886A
Target detection network training method and device, electronic equipment and storage medium
CN117611898A
Cited By
Image privacy positioning identification method and device based on multi-modal large model
CN119863691A
