Visual detection method and apparatus, device, storage medium, and program product

Abstract

A visual detection method, comprising: segmenting a picture to be detected to obtain a plurality of local images, wherein the plurality of local images have the same size; performing size adjustment on the picture to be detected to obtain a global image having the same size as each local image; inputting the plurality of local images and the global image into a picture encoder to obtain picture features, wherein the picture features comprise a plurality of local image features corresponding to the plurality of local images and a global image feature corresponding to the global image; inputting a text label into a text encoder to output text features; and inputting the picture features and the text features into an alignment decoder for fusion to obtain a processed picture.

Description

Vision detection method, device, equipment, storage medium and program product

[0001] The present disclosure is based on and claims priority to Chinese Patent Application No. 202411237521.9, filed on September 4, 2024, entitled “Vision detection method, device, equipment, storage medium and program product”, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the technical field of vision detection, and in particular to a vision detection method, a vision detection device, an electronic device, a computer-readable storage medium and a computer program product. BACKGROUND

[0003] With the development of multi-modal large models in the field of vision detection, in order to overcome the limitations of traditional vision models in understanding and interpreting complex scenes, by integrating multiple types of data modalities such as images, texts, etc., a deeper and more comprehensive understanding of the environment and content is achieved. However, the object detection effect of the picture edge position based on the multi-modal vision detection algorithm in the related art is poor, which reduces the ability of the model to perceive complex details in the graph. SUMMARY

[0004] The present disclosure aims to provide a vision detection method, a vision detection device, an electronic device, a computer-readable storage medium and a computer program product, which at least partially overcome the problem of poor object detection effect of the picture edge unknown in the related art.

[0005] According to one aspect of the present disclosure, a vision detection method is provided, comprising: segmenting a to-be-detected picture to obtain a plurality of local images, wherein the sizes of the plurality of local images are the same; performing size adjustment on the to-be-detected picture to obtain a global image with the same size as the local image; inputting the plurality of local images and the global image into a picture encoder to obtain picture features, the picture features including a plurality of local image features corresponding to the plurality of local images and a global image feature corresponding to the global image; inputting a text label into a text encoder to output a text feature; and inputting the picture feature and the text feature into an alignment decoder for fusion to determine the position information of the object corresponding to the text label in the to-be-detected picture.

[0006] In an embodiment of the present disclosure, the alignment decoder is constructed based on a cross-attention mechanism; wherein the fusion of the picture feature and the text feature into the alignment decoder to determine the position information of the object corresponding to the text label in the to-be-detected picture comprises: fusing the picture feature and the text feature by using the cross-attention mechanism to determine the position information of the object corresponding to the text label in the to-be-detected picture.

[0007] In an embodiment of the present disclosure, the fusion of the picture feature and the text feature by using the cross-attention mechanism to determine the position information of the object corresponding to the text label in the to-be-detected picture comprises: determining the attention weight of the picture feature according to the picture feature and the text feature; and determining the position information of the object corresponding to the text label in the to-be-detected picture according to the attention weight of the picture feature and the text label.

[0008] In an embodiment of the present disclosure, the input of the text label into the text encoder to output the text feature comprises: expanding the text label by prompt engineering to obtain an expanded text label; and inputting the expanded text label into the text encoder to output the text feature.

[0009] In an embodiment of the present disclosure, the segmentation of the to-be-detected picture to obtain a plurality of local images comprises: equally dividing the width W of the to-be-detected picture into x parts, and equally dividing the height H of the to-be-detected picture into y parts to obtain n local images, wherein the width of each local image is W / x and the height of each local image is H / y, x≥1, y≥1, n=x×y, and n≥2.

[0010] In an embodiment of the present disclosure, the dimension of the global image is (W / x, H / y, C), wherein C is the number of channels, and the input of the plurality of local images and the global image into the picture encoder to obtain the picture feature comprises: splicing the plurality of local images and the global image to obtain an image matrix, wherein the dimension of the image matrix is (W / x, H / y, C(x×y+1)); and inputting the image matrix into the picture encoder to obtain the picture feature, wherein the picture encoder is constructed based on at least one of a convolutional neural network, an autoencoder, a recurrent neural network, and a long short-term memory network.

[0011] According to another aspect of the present disclosure, there is provided a visual detection device, comprising: a first picture processing module configured to segment a picture to be detected to obtain a plurality of local images, wherein the plurality of local images have the same size; a second picture processing module configured to resize the picture to be detected to obtain a global image having the same size as the local images; a picture feature extraction module configured to input the plurality of local images and the global image into a picture encoder to obtain picture features, the picture features comprising a plurality of local image features corresponding to the plurality of local images and a global image feature corresponding to the global image; a text feature extraction module configured to input a text label into a text encoder to output a text feature; and a feature fusion module configured to input the picture features and the text feature into an alignment decoder to fuse the picture features and the text feature to implement visual detection of the picture to be detected.

[0012] According to a further aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory storing executable instructions of the processor; the processor being configured to execute the visual detection method described above via execution of the executable instructions.

[0013] According to a further aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the visual detection method described above.

[0014] According to a further aspect of the present disclosure, there is provided a computer program product having stored thereon a computer program, the computer program being executed by a processor to implement the visual detection method described above.

[0015] It should be understood that the general description above and the detailed description below are only exemplary and explanatory and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure. It is readily apparent to one of ordinary skill in the art that the accompanying drawings only illustrate some embodiments of the present disclosure and other drawings can be obtained by one of ordinary skill in the art without creative effort based on the accompanying drawings.

[0017] FIG. 1 shows an exemplary application system architecture schematic diagram of a visual detection method in an embodiment of the present disclosure.

[0018] FIG. 2 shows a schematic flowchart of a visual detection method in an embodiment of the present disclosure.

[0019] FIG. 3 shows a schematic diagram of an example of a visual detection method in an embodiment of the present disclosure.

[0020] FIG. 4 shows a schematic flow chart of another visual detection method in an embodiment of the present disclosure.

[0021] FIG. 5 shows a schematic diagram of an example of another visual detection method in an embodiment of the present disclosure.

[0022] FIG. 6 shows a schematic diagram of a visual detection device in an embodiment of the present disclosure.

[0023] FIG. 7 shows a structural block diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations.

[0025] In addition, the accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and serve to explain the principles of the present disclosure. The same reference numbers in different drawings represent the same or similar elements.

[0026] FIG. 1 shows a schematic diagram of an example application system architecture to which the visual detection method in an embodiment of the present disclosure can be applied. As shown in FIG. 1, the system architecture can include a terminal device 101, a network 102, and a server 103.

[0027] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0028] In some embodiments, the wireless network or wired network described above uses standard communications technologies and / or protocols. The network typically is the Internet, but can also include any combination of networks, including, but not limited to, local area networks (LAN), metropolitan area networks (MAN), wide area networks (WAN), mobile, wired or wireless networks, private networks or virtual private networks (VPN), any combination thereof, etc. In some embodiments, technologies and / or formats including, but not limited to, hypertext markup language (HTML), extensible markup language (XML), etc. are used to represent data exchanged over the network. In addition, all or some links can be encrypted using conventional encryption technologies such as, but not limited to, secure socket layer (SSL), transport layer security (TLS), virtual private networks (VPN), Internet Protocol Security (IPSec), etc. In other embodiments, custom and / or proprietary data communications technologies and / or formats can be used instead of, or in addition to, the ones described above.

[0029] The terminal device 101 can be various electronic devices, including but not limited to a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a wearable device, an augmented reality device, a virtual reality device, an image acquisition device, etc. The image acquisition device can be a camera, etc.

[0030] In one embodiment, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the difference of terminal platforms, the specific form of the client of the application can also be different, for example, the client of the application can be a mobile phone client, a PC client, etc.

[0031] The server 103 can be a server providing various services, for example, a background management server providing support for the device operated by the user using the terminal device 101. The background management server can analyze and process the received data such as to-be-processed pictures, etc., and feed back the processing result to the terminal device 101.

[0032] In an embodiment, the server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0033] Those skilled in the art can know that the number of terminal devices, networks and servers in FIG. 1 is only illustrative, and any number of terminal devices, networks and servers can be provided according to actual needs. The embodiments of the present disclosure do not limit this.

[0034] With the rapid development of 5G technology and artificial intelligence, video has become ubiquitous in work and life. In current video application scenarios, especially in video monitoring scenarios, the point information of the monitoring camera is missing. It is not possible to obtain the information included in the pictures collected by the monitoring camera. At present, the only way is to watch the monitoring video by people and manually mark the point information of the camera and the environment and content of the images captured by the camera.

[0035] The 5G era has given rise to a large number of video-oriented applications, such as smart and safe cities, the Internet, unmanned driving, and video monitoring based on image and video content. The application scenarios are very extensive, and image and video-related applications will become one of the main sources of incremental traffic in the 5G and post-5G eras.

[0036] In related technologies, a multi-modal visual detection algorithm usually directly inputs an image into a detection network, outputs a detection result, and judges the position and category of a detection target by extracting image features. In a multi-modal visual detection method, a prompt text is encoded by a text encoder to obtain text features, the text features are deeply fused with picture features obtained by a picture encoder to obtain position information of an object corresponding to the text, and positioning is performed. However, when encoding is performed by the picture encoder, the object information at the edge position of the picture is easily lost, the object detection effect at the center position of the picture is good, and the object detection effect at the edge position of the picture is poor.

[0037] To at least solve the above technical problems, the visual detection method provided by the embodiments of the present disclosure includes the following steps: segmenting a to-be-detected picture to obtain a plurality of local images, the plurality of local images having the same size; adjusting the size of the to-be-detected picture to obtain a global image having the same size as the local images; inputting the plurality of local images and the global image into a picture encoder to obtain picture features, the picture features including a plurality of local image features corresponding to the plurality of local images and a global image feature corresponding to the global image; inputting a text label into a text encoder to output a text feature; and inputting the picture features and the text feature into an alignment decoder for fusion to determine position information of an object corresponding to the text label in the to-be-detected picture. The embodiments of the present disclosure can better extract the detailed features of a high-resolution picture by image block segmentation and image size adjustment, and improve the ability of the model to perceive complex details in the picture.

[0038] Under the above system architecture, the visual detection method provided by the embodiments of the present disclosure can be executed by any electronic device with computing processing capability.

[0039] In some embodiments, the visual detection method provided by the embodiments of the present disclosure can be executed by the terminal device of the above system architecture; in other embodiments, the visual detection method can also be executed by the server of the above system architecture; in other embodiments, the visual detection method can also be realized by the terminal device and the server of the above system architecture through interaction.

[0040] In the following, the visual detection method in the example embodiment will be described in more detail with reference to the accompanying drawings and embodiments.

[0041] As shown in FIG. 2, the visual detection method according to one embodiment of the present disclosure includes the following steps:

[0042] S202, segmenting a to-be-detected picture to obtain a plurality of local images, wherein the plurality of local images have the same size.

[0043] In one embodiment, the to-be-detected picture can be a picture collected by an image collection device, or a video frame determined by a video collected by the image collection device. The to-be-detected picture collected by the image collection device can be stored in a database, so that the server extracts the corresponding to-be-detected picture from the database.

[0044] The to-be-detected picture can include one or more to-be-detected objects, and the to-be-detected object can be a person, an animal, an article, etc., which is not limited in the present disclosure.

[0045] In the present disclosure, the purpose of visual detection of the to-be-detected picture is to identify the to-be-detected object in the to-be-detected picture, and the identified information includes but is not limited to the type of the to-be-detected object, the position of the to-be-detected object in the to-be-detected picture, etc.

[0046] In S202, the to-be-detected picture can be segmented in the following manner: the width W of the to-be-detected picture is equally segmented into x parts, and the height H of the to-be-detected picture is equally segmented into y parts, to obtain n local images, each local image has a width of W / x and a height of H / y, where x≥1, y≥1, n=x×y, and n≥2.

[0047] For example, the width W of the to-be-detected picture is 800, and the height H is 450. If the width of the to-be-detected picture is equally segmented into 2 parts (x=2), and the height H is equally segmented into 2 parts (y=2), then 4 (n=2×2=4) local images can be obtained, each local image has a size of: width 400 and height 225. The 4 local images are numbered respectively, denoted as Pi, i={0, 1, …3}.

[0048] For example, for the above to-be-detected picture, if the width of the to-be-detected picture is equally segmented into 4 parts (x=4), and the height H is equally segmented into 3 parts (y=3), then n=12 local images can be obtained, each local image has a size of: width 200 and height 150. The local images are numbered respectively, denoted as Pi, i={0, 1, …11}.

[0049] In an embodiment, the to-be-detected picture can be cut by using a fully convolutional network (FCN), a U-Net (a fully convolutional neural network composed of an encoder and a decoder), a DeepLab (a semi-supervised model combining a deep convolutional neural network and a probabilistic graph model), or the like.

[0050] It should be noted that the cutting manner and the cutting number of the to-be-detected picture can be determined according to actual needs. For example, when the width direction of the to-be-detected picture includes more to-be-detected objects, the cutting number in the width direction can be appropriately increased; when the height direction of the to-be-detected picture includes angle to-be-detected objects, the cutting number in the height direction can be appropriately increased.

[0051] In S204, the to-be-detected picture is resized to obtain a global image with the same size as the local image.

[0052] In an embodiment, the neural network can be used to resize the to-be-detected picture. That is, the width of the to-be-detected picture is reduced to 1 / x of the original, and the height is reduced to 1 / y of the original, to obtain a global image.

[0053] The dimension of the global image can be represented as (W / x, H / y, C), wherein W / x represents the number of pixels in the horizontal direction, H / y represents the number of pixels in the vertical direction, and C is the number of channels. For example, the number of channels C of a black and white image is 1, and the number of channels C of an RGB color image is 3. For an RGB color image, the value range of each channel is: R is between 0 and 255, G is between 0 and 255, and B is between 0 and 255.

[0054] In one embodiment, the global image can be numbered and denoted as Pn.

[0055] S206, inputting the plurality of local images and the global image into a picture encoder to obtain picture features, the picture features including a plurality of local image features corresponding to the plurality of local images and a global image feature corresponding to the global image.

[0056] In one embodiment, the picture encoder is used for image processing of the input image, extracting image features and performing classification or recognition. The picture encoder can be constructed based on at least one of a convolutional neural network (CNN), an autoencoder, a recurrent neural network (RNN), and a long short-term memory (LSTM).

[0057] The convolutional neural network is mainly used for a deep learning model for image processing, extracts image features through multiple convolutional layers and pooling layers, and performs classification or regression through a fully connected layer, and has strong feature extraction capability.

[0058] The autoencoder is an unsupervised learning model composed of an encoder and a decoder. The encoder compresses the input data into low-dimensional encoding, and the decoder restores the low-dimensional encoding to reconstructed data. The autoencoder can be used for data dimensionality reduction, feature extraction, and other tasks.

[0059] The recurrent neural network is a neural network model suitable for processing sequence data. The output of the recurrently linked hidden layer is used as the input of the next time, so that the information of the variable-length input sequence can be processed, and the time-dependent relationship in the sequence can be captured.

[0060] The long short-term memory network is an improved RNN model that solves the problem of small or explosive gradient in RNN. By introducing a memory unit and a gating mechanism, the long short-term memory network can better handle long-time dependencies.

[0061] It should be noted that the picture feature extraction task can be performed by combining multiple CNNs, autoencoders, RNNs, and LSTMs.

[0062] In an embodiment, each local image Pi can determine a corresponding one local image feature Vi, i={0, 1, … n-1}. The global image Pn corresponds to at least one global image feature Vn. The picture features Vi, i={0, 1, … n}, can be determined according to actual conditions, which are not limited in the present disclosure.

[0063] In an embodiment, the inputting the plurality of local images and the global image into the picture encoder to obtain the picture features in S206 includes: splicing the plurality of local images and the global image to obtain an image matrix, the dimension of the image matrix being (W / x, H / y, C(x×y+1)); and inputting the image matrix into the picture encoder to obtain the picture features.

[0064] S208, inputting the text label into a text encoder to output a text feature.

[0065] In an embodiment, the text label can be an object type or an object name corresponding to the to-be-detected object, and is used to mark the to-be-detected object. The text label is used to describe the name or type of the to-be-detected object that can be included in the corresponding scene in the to-be-detected image.

[0066] The text encoder can convert the input text label into a high-dimensional vector representation to capture semantic information in the text label. The text encoder converts the input text through a series of hidden layers to extract semantic information at different levels and obtain a text feature, which is convenient for subsequent processing and analysis.

[0067] In an embodiment, the text encoder can be constructed based on a Transformer model.

[0068] In S208, the text label can be expanded to obtain an expanded text label by a prompt engineering, and the expanded text label is input into the text encoder to output a text feature.

[0069] The prompt engineering, also known as the context prompt, is a method of guiding the behavior of a large language model (LLM) towards a specific result by updating the weights or parameters of the model, and is a process of achieving the desired result of effective communication with artificial intelligence (AI). The text label can be expanded into a description sentence, such as "The pixels of CLASS_NAME", for example, "The pixels of sofa" and "The pixels of desk".

[0070] The text feature can be represented as T, T={T1, T2, … Tk}, and Tj is the jth text feature.

[0071] It should be noted that the number k of text features can be the same as the number of picture features, or different, and the present disclosure does not make specific limitations thereto.

[0072] S210, inputting the picture features and the text features into the alignment decoder to fuse, and determining the position information of the object corresponding to the text label in the to-be-detected picture.

[0073] In an embodiment, the alignment encoder can be constructed based on a cross-attention mechanism. In the Transformer architecture, the cross-attention mechanism refers to the cross-attention layer between the encoder and the decoder, in which the decoder performs attention adjustment on the output of the encoder to obtain the encoder information related to the current decoding position. In the encoder-decoder architecture of the Transformer architecture, the encoder is responsible for encoding the input sequence into a series of feature vectors (for example, the text features obtained by the above-mentioned text encoder, the picture features obtained by the picture encoder), and the decoder is responsible for generating an output sequence (i.e., the position information of the object corresponding to the text features in the to-be-detected picture) step by step according to the above-mentioned feature vectors (text features and picture features).

[0074] In an embodiment, the S210 described above fuses the picture features and the text features into the alignment decoder to determine the position information of the object corresponding to the text features in the to-be-detected picture, including: fusing the picture features and the text features by using the cross-attention mechanism to determine the position information of the object corresponding to the text label in the to-be-detected picture.

[0075] The calculation process of the cross-attention mechanism includes:

[0076] The encoder input can be represented as enc_inputs, and the size is (batch_size, seq_len_enc, hidden_dim);

[0077] The decoder input can be represented as dec_inputs, and the size is (batch_size, seq_len_dec, hidden_dim);

[0078] The decoder generates a query vector query (for example, the picture features described above) at each position, which is used for attention weight calculation at all positions of the encoder;

[0079] The encoder generates a set of key vectors keys (which can be the text vectors described above) and value vectors values (which can be the text labels described above) at all positions;

[0080] The dot product operation can be performed using the query vector query and the key vector keys, and the attention weight can be determined by the softmax function;

[0081] The attention weight is multiplied with the value vector, and the sum of the results is obtained as the output of the encoder adjustment, i.e. the position information of the object corresponding to the text feature in the to-be-detected picture.

[0082] In some embodiments, the above process can be understood as: the working process of the model generalization system in the intelligent video monitoring can be divided into data collection (text label, to-be-processed image), data analysis (generating text features and image features), model inference (i.e. fusing the picture features and the text features to determine the position information of the object corresponding to the text label in the to-be-detected picture).

[0083] It should be noted that the alignment encoder can also be implemented in other ways, which are not limited in the present disclosure.

[0084] As shown in FIG. 3, in one embodiment, the to-be-detected picture is an indoor picture as shown in FIG. 3, 4 local images are obtained by cutting, the to-be-detected picture is resized to obtain a global image, the global image and the 4 local images are input into the picture encoder to obtain image features, the image features include global image features corresponding to the global image and 4 local image features corresponding to the 4 local images; the text label (such as sofa, desk, etc.) is expanded by the prompt engineering, and the expanded text label is input into the text encoder to obtain the text features, and the text features and the picture features are input into the alignment decoder for fusion to determine the position information of the object corresponding to the text label in the to-be-detected picture.

[0085] It should be noted that the order of the extraction steps of the image features and the extraction steps of the text features is not limited.

[0086] In the embodiments of the present disclosure, the to-be-detected picture is segmented to obtain a plurality of local images, and the sizes of the plurality of local images are the same; the to-be-detected picture is resized to obtain a global image with the same size as the local image; the plurality of local images and the global image are input into the picture encoder to obtain picture features, the picture features include a plurality of local image features corresponding to the plurality of local images and a global image feature corresponding to the global image; the text label is input into the text encoder to output the text features; the picture features and the text features are input into the alignment decoder for fusion to determine the position information of the object corresponding to the text label in the to-be-detected picture. The present disclosure can better extract the detailed features of high-resolution pictures through image block segmentation and image size adjustment, and improve the ability of the model to perceive complex details in the image.

[0087] FIG. 4 shows a flowchart of another visual detection method in an embodiment of the present disclosure. On the basis of the embodiment of FIG. 2, S210 is further refined into S212-S214 to limit the position information of the object corresponding to the text feature in the picture to be detected obtained by fusing the picture feature and the text feature through the cross-attention mechanism. As shown in FIG. 4, in an embodiment, the method comprises S202-S208, S212-S214. Wherein:

[0088] S212, determining the attention weight of the picture feature according to the picture feature and the text feature;

[0089] S214, determining the position information of the object corresponding to the text label in the picture to be detected according to the attention weight of the picture feature and the text label.

[0090] It should be noted that the specific implementation mode of S202-S208 in the embodiment of the present disclosure is the same as that of S202-S208 in the foregoing embodiment, which will not be described here.

[0091] In an embodiment, the above-mentioned determining the attention weight of the picture feature according to the picture feature and the text feature comprises: determining a first function value of a projection head that maps the picture feature to the same dimension as the text feature; determining a second function value of a projection head that maps the text feature to the same dimension as the picture feature; and obtaining the weight of the picture feature according to the first function value and the second function value.

[0092] As shown in FIG. 5, in an embodiment, the weight Weigth i of the i-th picture feature is determined by Formula One: i = Softmax(g v (T)g w (V i )) (Formula One)

[0093] Wherein, the function g(·) represents a projection head that maps the input to the same dimension, the function g v (·) represents a projection head that maps the text feature to the same dimension as the picture feature, and the function g w (·) represents a projection head that maps the picture feature to the same dimension as the text feature; Softmax(·) is a normalized exponential function, which is a generalization of the logistic function, and can display the results of multi-classification in the form of probability; T is all text features.

[0094] In S214, the output Output of the alignment decoder can be determined by Formula Two:

[0095] Wherein, Decoder represents the alignment decoder; T1,...,Tk These represent text labels, and x and y represent the number of width and height segments of the image to be detected, respectively; V i Represents i image features.

[0096] In this embodiment of the disclosure, adaptive weights are used to redistribute the weights of image features, and after being fused with text features, the accuracy of visual detection is improved.

[0097] To deepen the understanding of the visual inspection method disclosed herein, specific examples are provided below.

[0098] Example 1:

[0099] Step 1: Divide the image Img into multiple local images on an average basis.

[0100] The image to be processed has a width of 800 and a height of 450. The width is divided into two parts and the height is divided into two parts in equal proportion, resulting in four local images. The size of each local image is 400 in width and 225 in height.

[0101] Step 2: Resize the image to be processed to the size of the local image to obtain the global image (400, 225, 3).

[0102] Step 3: Concatenate the local image and the global image to obtain an image matrix with dimensions (400, 225, 15). Input the matrix into the image encoder to obtain image features, which include local image features and global image features.

[0103] Step 4: Expand the label description using the prompting process.

[0104] The pixels of the sofa.

[0105] The pixels of the desk.

[0106] ...

[0107] Step 5: Input the extended label description into the text encoder to obtain text features.

[0108] Step 6: Input the text features and image features into the alignment decoder and fuse them. The alignment decoder consists of two cross-attention mechanisms, in which the importance of image features is assigned through adaptive weights.

[0109] The attention weight of the i-th image feature is calculated using Formula 3: Weigth i =Softmax(g v (T)g w (V i (Formula 3)

[0110] where i = {1, 2,... 5}.

[0111] The output of the alignment decoder is determined by Equation Four:

[0112] Decoder denotes the alignment decoder, T refers to all text features, T1,..., T k denotes each text label feature, V i denotes the i-th image feature. Function g v and g w are projection heads that map the input to the same dimension.

[0113] Example Two:

[0114] Step One: Cut the image Img evenly into multiple local images.

[0115] The width of the image to be processed is 800 and the height is 450. The width is divided into 4 equal parts and the height is divided into 3 equal parts, resulting in 12 local images, each with a size of width 200 and height 150.

[0116] Step Two: Resize the image to be processed to the size of the local image to obtain a global image (200, 150, 3).

[0117] Step Three: Concatenate the local image and the global image to obtain an image matrix with dimensions (200, 150, 36), input the image encoder to obtain the image features, which include local image features and global image features.

[0118] Step Four: Expand the label description by the prompt engineering.

[0119] The pixels of sofa.

[0120] The pixels of desk.

[0121] ...

[0122] Step Five: Input the expanded label description into the text encoder to obtain the text features.

[0123] Step Six: Input the text features and image features into the alignment decoder for fusion. The alignment decoder consists of two cross-attention mechanisms, and the importance of the image features is determined by adaptive weight distribution.

[0124] The attention weight of the i-th image feature is calculated by Equation Five: Weigth i = Softmax(g v (T)g w (Vi )) (Equation Five)

[0125] where i = {1, 2, …, 12}.

[0126] The output of the alignment decoder is determined by Equation Four:

[0127] Decoder denotes the alignment decoder, T refers to all text features, T1, …, T12 refer to each text label feature, V refers to the vocabulary, and T1, …, T12 are the text label features. k V refers to the vocabulary, and T1, …, T12 are the text label features. i Ti refers to the i-th image feature. The function g v and g w are projection heads that map the input to the same dimension.

[0128] Based on the same inventive concept, the embodiments of the present disclosure also provide a visual detection device, as described in the following embodiments. Since the principle of solving problems of the device embodiments is similar to the above-mentioned method embodiments, the implementation of the device embodiments can be referred to the implementation of the above-mentioned method embodiments, and the repeated parts will not be described here.

[0129] FIG. 6 shows a schematic diagram of a visual detection device according to an embodiment of the present disclosure. As shown in FIG. 6, the device comprises:

[0130] A first image processing module 610 is configured to segment a to-be-detected image to obtain a plurality of local images, wherein the plurality of local images have the same size.

[0131] A second image processing module 620 is configured to adjust the size of the to-be-detected image to obtain a global image having the same size as the local images.

[0132] An image feature extraction module 630 is configured to input the plurality of local images and the global image into an image encoder to obtain image features, wherein the image features comprise a plurality of local image features corresponding to the plurality of local images and a global image feature corresponding to the global image.

[0133] A text feature extraction module 640 is configured to input a text label into a text encoder to output text features.

[0134] A feature fusion module 650 is configured to input the image features and the text features into an alignment decoder to fuse the image features and the text features, thereby realizing visual detection of the to-be-detected image.

[0135] It should be noted that the first picture processing module 610, the second picture processing module 620, the picture feature extraction module 630, the text feature extraction module 640 and the feature fusion module 650 correspond to S202-S210 in the method embodiment, and the above modules and the examples and application scenarios realized by the corresponding steps are the same, but are not limited to the content disclosed in the above method embodiment. It should be noted that the above modules as part of the device can be executed in a computer system such as a group of computer executable instructions.

[0136] In one embodiment, the alignment decoder is constructed based on a cross-attention mechanism; wherein the feature fusion module 650 is configured to fuse the picture feature and the text feature by using the cross-attention mechanism to determine the position information of the object corresponding to the text label in the to-be-detected picture.

[0137] In one embodiment, the feature fusion module 650 is configured to determine the attention weight of the picture feature according to the picture feature and the text feature; and determine the position information of the object corresponding to the text label in the to-be-detected picture according to the attention weight of the picture feature and the text label.

[0138] In one embodiment, the text feature extraction module 640 is configured to expand the text label by prompt engineering to obtain an expanded text label; and input the expanded text label into a text encoder to output the text feature.

[0139] In one embodiment, the first picture processing module 610 is configured to divide the width W of the to-be-detected picture into x equal parts, divide the height H of the to-be-detected picture into y equal parts, and obtain n local images, wherein the width of each local image is W / x and the height is H / y, x≥1, y≥1, n=x×y, n≥2.

[0140] In one embodiment, the dimension of the global image is (W / x, H / y, C), wherein C is the number of channels; the picture feature extraction module 630 is configured to splice the plurality of local images and the global image to obtain an image matrix, wherein the dimension of the image matrix is (W / x, H / y, C(x×y+1)); and input the image matrix into a picture encoder to obtain the picture feature, wherein the picture encoder is constructed based on at least one of a convolutional neural network, an autoencoder, a recurrent neural network, and a long short-term memory network.

[0141] Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method or a program product. Therefore, various aspects of the present disclosure can be embodied as a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.

[0142] An electronic device 700 according to this implementation of the present disclosure is described below with reference to FIG. 7. FIG. 7 shows the electronic device 700 merely as an example, and should not bring any limitation to the functions and usage scope of the embodiments of the present disclosure.

[0143] As shown in FIG. 7, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 can include, but are not limited to, the at least one processing unit 710 described above, the at least one storage unit 720 described above, and a bus 730 connecting different system components, including the storage unit 720 and the processing unit 710.

[0144] The storage unit stores program codes executable by the processing unit 710, so that the processing unit 710 performs the steps described in the above “Exemplary Method” section according to various exemplary implementations of the present disclosure. For example, the processing unit 710 can perform the following steps of the above method embodiments: segmenting a to-be-detected picture to obtain a plurality of local images, the plurality of local images having the same size; resizing the to-be-detected picture to obtain a global image having the same size as the local images; inputting the plurality of local images and the global image into a picture encoder to obtain picture features, the picture features including a plurality of local image features corresponding to the plurality of local images and a global image feature corresponding to the global image; inputting a text label into a text encoder to output a text feature; and inputting the picture features and the text feature into an alignment decoder for fusion to determine position information of an object corresponding to the text label in the to-be-detected picture.

[0145] The storage unit 720 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 7201 and / or a cache memory 7202, and can further include a read-only memory (ROM) 7203.

[0146] The storage unit 720 can further include a program / utility 7204 having a set of program modules 7205, including but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof can include implementation of a network environment.

[0147] The bus 730 can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit bus, or a local bus using any of a variety of bus architectures.

[0148] The electronic device 700 can also communicate with one or more external devices 740 such as a keyboard or pointing device, a Bluetooth device, or a database, and / or one or more devices that enable a user to interact with the electronic device 700 and / or one or more devices (e.g., a router, a modem, a server, etc.) that enable the electronic device 700 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface(s) 750. Still yet, the electronic device 700 can communicate with one or more networks, such as one or more local area networks (LANs), one or more wide area networks (WANs), and / or the Internet, through a network adapter 760. As depicted, the network adapter 760 communicates with the other components of the electronic device 700 via the bus 730. It should be appreciated that the network adapter 760 and / or the bus 730 can be implemented using one or more types of technology, including, but not limited to, Ethernet, Token Ring, FDDI, Wi-Fi, IEEE 802.11, Bluetooth, and / or any other suitable type of technology.

[0149] From the above description of the embodiments, those skilled in the art will readily appreciate that the example embodiments described herein can be implemented by software and / or by hardware coupled with software. Accordingly, the technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0150] In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer program product, which includes a computer program that, when executed by a processor, implements the above-mentioned visual detection method.

[0151] In the example embodiments of the present disclosure, a computer-readable storage medium is also provided, which can be a readable signal medium or a readable storage medium. The computer-readable storage medium stores a program product capable of implementing the above-mentioned method of the present disclosure. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code for causing a terminal device to perform the steps according to various example embodiments of the present disclosure described in the above “Example Method” section of the specification when the program product is run on the terminal device.

[0152] More specific examples of the computer-readable storage medium in the present disclosure can include but are not limited to an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] In the present disclosure, a computer readable storage medium can include a data signal carrying the readable program code in a baseband or in a carrier wave. Such a propagated data signal can take a wide variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. A computer readable storage medium can also be any non-transitory computer readable medium which can be considered to be a tangible storage medium. A computer readable storage medium can be any medium that can be read by a machine learner, e.g., a computer. The computer readable storage medium can include a hard disk, a floppy disk, a magnetic disk, an optical disk, a compact disk, a CD-ROM, a DVD, a RAM, a ROM, a FLASH memory, a portable computer diskette, a RAM with one or several wires, a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] In one embodiment, the program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the foregoing.

[0155] In practice, the program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0156] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, such a division is not mandatory. Indeed, according to an embodiment of the present disclosure, the features and functionalities of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functionalities of one module or unit described above can be further divided into a plurality of modules or units.

[0157] Moreover, although individual steps of the methods in the present disclosure are described in a particular order in the drawings, this is not required or implied, nor is it necessary to perform all of the steps shown to achieve the desired result. Additionally or alternatively, certain steps can be omitted, combined into a single step, performed in a different order, broken down into multiple steps, and / or the like.

[0158] Through the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0159] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such departures from the present disclosure that come within known use or custom in the art to which the present disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A visual inspection method, wherein, The method comprises the following steps: segmenting a to-be-detected picture to obtain a plurality of local images, wherein the plurality of local images have the same size; adjusting the size of the to-be-detected picture to obtain a global image having the same size as the local images; inputting the plurality of local images and the global image into a picture encoder to obtain picture features, wherein the picture features comprise a plurality of local image features corresponding to the plurality of local images and a global image feature corresponding to the global image; inputting a text label into a text encoder to output a text feature; fusing the picture features and the text feature in an alignment decoder to determine position information of an object corresponding to the text label in the to-be-detected picture.

2. The visual inspection method of claim 1, wherein, The alignment decoder is constructed based on a cross-attention mechanism; wherein the step of fusing the picture features and the text feature in the alignment decoder to determine the position information of the object corresponding to the text label in the to-be-detected picture comprises: fusing the picture features and the text feature by using a cross-attention mechanism to determine the position information of the object corresponding to the text label in the to-be-detected picture.

3. The visual inspection method of claim 2, wherein, The step of fusing the picture features and the text feature by using a cross-attention mechanism to determine the position information of the object corresponding to the text label in the to-be-detected picture comprises: determining an attention weight of the picture features according to the picture features and the text feature; determining the position information of the object corresponding to the text label in the to-be-detected picture according to the attention weight of the picture features and the text label.

4. The visual inspection method of claim 1, wherein, The step of inputting a text label into a text encoder to output a text feature comprises: extending the text label by a prompt engineering to obtain an extended text label; inputting the extended text label into the text encoder to output the text feature.

5. The visual inspection method according to any one of claims 1 to 4, wherein, The step of segmenting a to-be-detected picture to obtain a plurality of local images comprises: equally dividing the width W of the to-be-detected picture into x parts and equally dividing the height H of the to-be-detected picture into y parts to obtain n local images, wherein each local image has a width of W / x and a height of H / y, x≥1, y≥1, n=x×y, and n≥2.

6. The visual inspection method of claim 5, wherein, The global image has a dimension of (W / x, H / y, C), wherein C is the number of channels. The step of inputting the plurality of local images and the global image into a picture encoder to obtain picture features comprises: splicing the plurality of local images and the global image to obtain an image matrix, wherein the image matrix has a dimension of (W / x, H / y, C(x×y+1)); inputting the image matrix into the picture encoder to obtain the picture features, wherein the picture encoder is constructed based on at least one of a convolutional neural network, an autoencoder, a recurrent neural network, and a long short-term memory network.

7. A vision inspection apparatus wherein, The method comprises the following steps: a first picture processing module is configured to segment a to-be-detected picture to obtain a plurality of local images, wherein the plurality of local images have the same size; a second picture processing module is configured to adjust the size of the to-be-detected picture to obtain a global image having the same size as the local images; The picture feature extraction module is configured to input the plurality of local images and the global image into a picture encoder to obtain picture features, wherein the picture features comprise a plurality of local image features corresponding to the plurality of local images and a global image feature corresponding to the global image. The text feature extraction module is configured to input the text label into a text encoder to output a text feature. The feature fusion module is configured to input the picture features and the text feature into an alignment decoder to fuse the picture features and the text feature, and to implement visual detection on the to-be-detected picture.

8. An electronic device, comprising: The visual detection method comprises the following steps: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the visual detection method according to any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, wherein, The computer program is executed by the processor to implement the visual detection method according to any one of claims 1-6.

10. A computer program product having stored thereon a computer program, wherein, The computer program is executed by the processor to implement the visual detection method according to any one of claims 1-6.

Images