A method, device and equipment for open set target detection based on cloud-edge-end architecture

Abstract

The present invention discloses an open set target detection method, device and equipment based on a cloud-edge-end architecture, and relates to the field of visual language detection technology. The cloud server can obtain the text input by the user uploaded by the edge device and extract the language features in the text, and then perform classification mapping based on the preset category table to determine whether the categories described in the text all appear in the preset category table. If there are new categories, the visual features extracted by the edge device from the image input by the user are obtained, so as to perform feature fusion of the language features and the visual features to obtain the visual language feature encoding, and determine the target to be detected corresponding to the text in the image according to the visual language feature encoding and send it to the edge device. If there is no new category, the edge device determines the target to be detected corresponding to at least some categories in the preset category table according to the image input by the user. The present invention improves the practicality of open set target detection in the actual scenario of the cloud-edge-end architecture.

Description

Technical Field

[0001] The present invention relates to the field of visual language detection technology, and in particular to an open set target detection method, device and equipment based on a cloud-edge-end architecture. Background Art

[0002] At present, in the research of traditional machine learning visual detection models, it is usually assumed that the test samples and application scenarios of the model come from known categories and familiar scenes in the model training stage. This type of algorithm is called closed set detection. However, this assumption only exists in the laboratory and is often not true in applications facing real-world scenarios, because the real world is open, unknown, and novel.

[0003] For example, in the field of autonomous driving, closed-set object detection can only identify object categories in the training data, and any new or abnormal objects (such as unknown road obstacles or new traffic signs) may not be correctly identified, resulting in inappropriate responses, which may trigger inappropriate driving decisions, increase the risk of accidents and have a profound impact on the safety and reliability of the autonomous driving system. In addition, as the environment changes, such as road conditions, updates to traffic rules or new traffic facilities, the closed-set object detection model needs to be retrained to include these new data, which is time-consuming and costly.

[0004] The introduction of open-set object detection models aims to improve the system's ability to recognize a wide range of categories, including those that the model has never learned during the training phase. The introduction of such models significantly enhances the robustness and adaptability of the system, enabling it to better cope with changing environments and conditions. In the past year, the introduction of representative visual language models such as the OWL-ViT (Open World Vision Transformer) model and the Ferret model has marked a major development in open-set object detection and visual representation learning. These models combine image content with language descriptions. In practical applications, they can not only recognize a large number of known categories, but also process and classify categories that have not been seen during training.

[0005] The Ferret model is a large multimodal language model (LLM) that integrates image and text understanding capabilities, and performs well in analyzing specific areas of images and responding to context. The OWL-ViT model is a Transformer-based object detection model that uses a multimodal architecture similar to CLIP (Contrastive Language-Image Pre-training), has zero-sample object detection capabilities, and can identify new categories without specific training.

[0006] The visual language model uses deep learning technology to learn to recognize and understand the relationship between image content and language description through joint training of image and text data. This enables the model to not only recognize objects in the image, but also understand the semantic information of these objects. Compared with closed-set detection, open-set object detection can improve classification accuracy. By jointly training visual information and related text descriptions, the model can more accurately recognize and classify objects in images when performing classification tasks. This is because language information provides additional context for visual objects, helping the model to better understand the scene. In addition, open-set object detection can also enrich training data. Visual language models are usually trained using large-scale image-text pairing datasets, which include not only information about the image itself, but also descriptions of the image content. The richness of this data enables the model to learn more dimensional information, thereby improving its generalization ability in complex environments.

[0007] However, visual language models require a lot of computing resources and storage space, especially when performing real-time data processing and response. Often, visual language models may not be able to achieve real-time object detection and image analysis without the support of high-performance GPUs. The limited computing resources and storage space of mobile devices or edge devices limit the practicality of visual language models on mobile devices or edge devices. In other words, the current visual language models are difficult to deploy directly into actual scenarios of cloud-edge architecture due to their large model size and complex structure. Summary of the invention

[0008] Based on this, it is necessary to provide an open set target detection method, device and equipment based on cloud-edge architecture to address the above technical problems.

[0009] The present invention adopts the following technical solutions:

[0010] The present invention provides an open set target detection method based on a cloud-edge-end architecture, wherein a cloud server can obtain a text input by a user uploaded by an edge device and extract language features from the text, and the edge device can simultaneously obtain an image input by the user, wherein the text includes the category of the target to be detected in the image. Then, the cloud server can classify and map the language features based on a preset category table to determine whether the categories described in the text all appear in the preset category table. If there are new categories, the visual features extracted by the edge device from the image input by the user are obtained, thereby obtaining enhanced language features and enhanced visual features by performing feature fusion on the language features and the visual features through a feature fusion device, and determining enhanced visual features matching the enhanced language features according to the similarity between the enhanced visual features and the enhanced language features through a query module guided by language, obtaining visual language feature encoding, and determining the target to be detected in the image corresponding to the text according to the visual language feature encoding through a feature decoder and sending it to the edge device. If there are no new categories, the edge device is enabled to perform target detection according to the image input by the user, and the edge device is deployed with an image detection model, which is obtained by training the corresponding images of each category in the preset category table. Among them, the feature fuser is constructed by connecting two parallel self-attention modules, followed by a cross-attention module from image to text and a cross-attention module from text to image in series, and then connecting two parallel multi-layer perceptrons in series.

[0011] The present invention provides an open set target detection device based on a cloud-edge-end architecture, comprising:

[0012] A cloud acquisition module is used to acquire the text input by the user uploaded by the edge device and extract the language features in the text; wherein the edge device simultaneously acquires the image input by the user, and the text includes the category of the target to be detected in the image;

[0013] The judgment and detection module is used to classify and map the language features according to the preset category table to determine whether the categories described in the text all appear in the preset category table; if so, an instruction is sent to the edge device to indicate that the user input content does not include the target corresponding to the new category; the edge device performs target detection according to the image input by the user; if not, an instruction is sent to the edge device to indicate that the user input content includes the target corresponding to the new category, and the visual features extracted by the edge device from the image input by the user are obtained; the language features and the visual features are fused by the feature fusion device to obtain enhanced language features and enhanced visual features, and the enhanced visual features matching the enhanced language features are determined by the query module guided by the language according to the similarity between the enhanced visual features and the enhanced language features to obtain the visual language feature encoding, and the target to be detected corresponding to the text in the image is determined by the feature decoder according to the visual language feature encoding and sent to the edge device; wherein the feature fusion device is constructed by two parallel self-attention modules, followed by a cross-attention module from image to text and a cross-attention module from text to image in series, and two parallel multi-layer perceptrons.

[0014] The present invention provides an open set target detection device based on a cloud-edge-end architecture, comprising:

[0015] The edge acquisition module is used to acquire the text and image input by the user, wherein the text includes the category of the target to be detected in the image;

[0016] A feature extraction module is used to extract visual features from the image and upload the text to the cloud server, so that the cloud server extracts language features from the text and classifies and maps the language features according to a preset category table to determine whether the categories described in the text all appear in the preset category table;

[0017] A detection module, configured to, upon receiving an instruction from a cloud server indicating that the user input content does not include an object corresponding to a new category, perform object detection based on the image input by the user, determine an object to be detected, and return the result to the user;

[0018] The transmission module is used to upload the visual features to the cloud server when receiving an instruction sent by the cloud server that represents the user input content including the target corresponding to the new category, so that the cloud server can fuse the language features and the visual features through the feature fuser to obtain the enhanced language features and the enhanced visual features, and determine the enhanced visual features that match the enhanced language features according to the similarity between the enhanced visual features and the enhanced language features through the language-guided query module, obtain the visual language feature encoding, and determine the target to be detected in the image corresponding to the text according to the visual language feature encoding through the feature decoder; receive the target to be detected in the image corresponding to the text sent by the cloud server and return it to the user.

[0019] The present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the open set target detection method based on the cloud-edge architecture is implemented.

[0020] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the open set target detection method based on the cloud-edge architecture is implemented.

[0021] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0022] The present invention splits the open set object detection task combining text and image, and splits the visual language model at the same time. Since the computing resources of edge devices are generally tight, the image processing task is placed on the edge device, and features of the image are extracted through the edge device. Cloud servers generally have more computing resources, so the text processing task that requires a lot of computing resources is placed on the cloud server.

[0023] Furthermore, in practical applications, open-set target detection may have new categories that are not learned during model training, or may not have new categories. On the one hand, the present invention deploys a pre-trained image detection model on the edge device, and on the other hand, deploys a preset category table of text in the cloud server, wherein the image detection model is obtained by training the corresponding images of each category in the preset category table. Thus, in the process of processing the text by the cloud server with more computing resources, the feature information contained in the text is used to guide the entire open-set target detection task.

[0024] When the cloud server finds that no new categories appear during text processing, it means that the image detection model deployed on the edge device has learned all the information about the targets to be detected in the image during training. At this time, it can be regarded as closed-set target detection. No text is needed to guide the open-set target detection process. Target detection can be performed directly through the edge device, further improving the detection response speed when there are no new categories.

[0025] When a new category appears in the text processing of the cloud server, the edge device uploads the visual features corresponding to the image to the cloud server, and cross-processes the visual features and text features based on the image-to-text cross-attention module and the text-to-image cross-attention module in the feature fusion module, and deeply fuses the two, so that the visual language feature encoding is further matched through the language-guided query module, and decoded through the feature decoder. Through the above processing, the interactivity and depth in the multimodal data processing of text and images are improved, and the powerful computing power of the cloud server is effectively utilized to improve the efficiency and accuracy of open-set target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0027] Figure 1 A schematic diagram of the flow of an OWL-ViT model provided by the present invention;

[0028] Figure 2 A schematic diagram of the process of an open set target detection method based on a cloud-edge-device architecture provided by the present invention on a cloud server;

[0029] Figure 3 A schematic diagram of the process of an open set target detection method based on a cloud-edge-device architecture provided by the present invention on an edge device;

[0030] Figure 4 A schematic diagram of a process flow of an edge device performing a task provided by the present invention;

[0031] Figure 5 A schematic diagram of a process flow of a cloud server executing a task provided by the present invention;

[0032] Figure 6 A schematic diagram of the overall structure of a visual language model provided by the present invention;

[0033] Figure 7 A schematic diagram of an algorithm flow of a visual feature extractor and a visual detection module provided by the present invention;

[0034] Figure 8 A schematic diagram of the algorithm flow of a language feature extractor and a visual detection module provided by the present invention;

[0035] Fig. 9 A schematic diagram of the algorithm structure of a feature fusion device provided by the present invention;

[0036] Fig.10A schematic diagram of an algorithm flow of a language-guided query module provided by the present invention;

[0037] Fig.11 A schematic diagram of the algorithm structure of a feature decoder provided by the present invention;

[0038] Fig.12 A schematic diagram of an open-set target detection device based on a cloud-edge-end architecture on a cloud server side provided by the present invention;

[0039] Fig.13 A schematic diagram of an open-set target detection device based on a cloud-edge-end architecture on an edge device side provided by the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] At present, in open set target detection, taking the OWL-ViT model as an example, the basic technical route of the visual language model is: First, the visual language model requires the user to input prompt words to locate the target position in the image and confirm the target category. The prompt words are open and can be changed according to the user scenario. The visual language model will input the input prompt words and images into two feature extractors, the language feature extractor and the visual feature extractor, respectively. These two extractors are also called encoders to obtain processed visual features and language features. This is followed by feature fusion to interact with the features of the two modalities. Next, the visual features are input into the detection head of the target detection to locate the target position. In addition, the language features are encoded into a query embedding sequence to interact with the image features for target classification. Figure 1 As shown, Figure 1 The figure is a flow chart of an OWL-ViT model in the present invention.

[0042] Due to their large model size and complex structure, visual language models are difficult to deploy directly into actual scenarios of cloud-edge architecture. At the same time, fine-tuning visual language models also faces many challenges. Due to the complexity of the model and the large number of parameters, the fine-tuning process not only requires a lot of time and computing resources, but also often requires deep expertise and technology to adjust the model to a specific task or data set. This increases the technical threshold for model deployment and maintenance, which is particularly unfavorable for commercial applications that require rapid iteration and deployment.

[0043] In addition, the complexity and large parameter size of visual language models make it difficult to perform effective model splitting, which limits their application in cloud-edge collaborative computing environments. These models are often optimized and trained as a whole, which means that their different components (such as visual processing units and language processing units) are tightly coupled and difficult to separate. Therefore, to deploy some parts of these models on edge devices while keeping other computationally intensive parts in the cloud, complex model reconstruction and optimization are required.

[0044] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0045] Figure 2 The following is a flow chart of an open set target detection method based on a cloud-edge-device architecture in the present invention on a cloud server, which specifically includes the following steps:

[0046] S101: Obtain a text input by a user uploaded to an edge device, and extract language features in the text; the text includes a category of a target to be detected in an image uploaded by the user to the edge device.

[0047] S102: Classify and map the language features according to the preset category table to determine whether the categories described in the text all appear in the preset category table. If so, execute step S103; if not, execute step S104.

[0048] S103: Send an instruction to the edge device indicating that the user input content does not include a target to be detected of a new category; enable the edge device to perform target detection based on the image input by the user through an image detection model; the image detection model is trained by images corresponding to each category in a preset category table.

[0049] S104: Send an instruction to the edge device that represents the user input content including the new category of the target to be detected, and obtain the visual features extracted by the edge device from the user input image; perform feature fusion on the language features and the visual features through the feature fuser to obtain enhanced language features and enhanced visual features, and determine the enhanced visual features that match the enhanced language features according to the similarity between the enhanced visual features and the enhanced language features through the language-guided query module to obtain the visual language feature encoding, and determine the target to be detected corresponding to the text in the image according to the visual language feature encoding through the feature decoder and send it to the edge device.

[0050] based on Figure 2 The open set target detection method based on the cloud-edge architecture on the cloud server side is shown, and the present invention also provides an explanation on the edge device side.

[0051] Figure 3The following is a flow chart of an open set target detection method based on a cloud-edge-device architecture in an edge device in the present invention, which specifically includes the following steps:

[0052] S201: Acquire text and an image input by a user, wherein the text includes a category of an object to be detected in the image.

[0053] S202: extract visual features from the image and upload the text to the cloud server, so that the cloud server extracts language features from the text and classifies and maps the language features according to a preset category table to determine whether the categories described in the text all appear in the preset category table.

[0054] S203: When receiving an instruction from the cloud server indicating that the user input content does not include a new category of targets to be detected, determining the targets to be detected based on visual features through an image detection model and returning it to the user; the image detection model is trained by images corresponding to each category in a preset category table.

[0055] S204: When receiving an instruction sent by the cloud server indicating that the user input content includes a new category of targets to be detected, uploading the visual features to the cloud server, allowing the cloud server to fuse the language features and the visual features through a feature fuser to obtain enhanced language features and enhanced visual features, and determining the enhanced visual features that match the enhanced language features based on the similarity between the enhanced visual features and the enhanced language features through a language-guided query module, obtaining a visual language feature encoding, and determining the targets to be detected in the image corresponding to the text based on the visual language feature encoding through a feature decoder; receiving the targets to be detected in the image corresponding to the text sent by the cloud server and returning them to the user.

[0056] Figure 4 is a flow chart of an edge device executing a task in the present invention, Figure 5 A schematic diagram of a process of executing a task by a cloud server in the present invention.

[0057] In large-scale visual language models, the model's ability to recognize and judge objects and targets in the open world mainly comes from the powerful expressive power and contextual understanding ability of the language model in the visual language model. As a tool for processing natural language, the language model can understand and generate descriptions related to images, which makes it possible to identify unlabeled or unseen categories. For example, when using a visual language model for image classification or target detection, the language model can understand rich text description information through natural language processing, and combine this information with the visual features of the image to achieve the recognition of unknown or rare targets in the image. This capability enables the visual language model to no longer be limited to an inherent or limited set of labels, but to work effectively in a wider range of dynamically changing environments.

[0058] In order to apply large-scale visual language models to actual scenarios, this paper combines the visual language model with the cloud-edge architecture coordinated computing technology, disassembles the model structure, and places it on the cloud server and edge device respectively, so as to ensure the stability of target detection accuracy while reducing the deployment cost of the visual language model. Figure 6 As shown, Figure 6 is a schematic diagram of the overall structure of a visual language model in the present invention, Figure 6 In the figure, the structures in the dashed box are located on the edge device, and the rest of the structures are located on the cloud server.

[0059] Specifically, for feature extraction of text and images input by users, the present invention adopts a dual-stream network architecture, which includes a visual feature extractor and a language feature extractor, which are used to process images and texts respectively, thereby establishing associations between different modalities.

[0060] Among them, visual feature extractor: In one or more embodiments of the present invention, the visual feature extractor is placed on the edge device. Depthwise separable convolution can be selected as the core technology of this part. The use of depthwise separable convolution can not only effectively improve the efficiency of visual feature extraction, but also significantly reduce the number of parameters. Depthwise separable convolution decomposes the traditional convolution operation into two steps: depthwise convolution and pointwise convolution, which respectively handle spatial relationships and cross-channel feature combinations. The advantage of this structure is that it greatly reduces the computing cost and the size of the model, so that the model can process user input images in real time on the edge device while maintaining high accuracy. This is particularly important for cloud-edge collaborative computing because it allows fast pre-processing at the edge, and if necessary, key features or pre-processed data can be transmitted to the cloud for further analysis and processing. This approach not only optimizes data flow and computing load, but also helps to improve the response speed and processing efficiency of the entire system.

[0061] Language feature extractor: In one or more embodiments of the present invention, when constructing a language feature extractor, a BERT (Bidirectional Encoder Representations from Transformers) network model can be selected as the infrastructure. The backbone network of this structure is large in scale and needs to respond to unknown and diverse text prompts input by users. Therefore, it is placed on a cloud server, which sends instructions to edge devices while responding to user needs to meet the needs of cloud-edge collaborative computing.

[0062] The BERT network model is a Transformer-based model that learns rich language representations on large-scale text datasets through pre-training, and can then be fine-tuned to adapt to various specific downstream tasks. This pre-training plus fine-tuning strategy enables the BERT network model to perform better in many natural language processing tasks. Compared with other natural language processing backbone networks, the BERT network model can be deployed to fixed-demand tasks more quickly. In addition, the bidirectional encoding capability of the BERT network model, that is, considering both the left and right sides of the context at the same time, provides strong support for more accurate capture and utilization of language information, especially in understanding and generating image-related descriptions.

[0063] Through this dual-stream network architecture, the present invention can process the images and texts input by the user in parallel, and then integrate the features of the images and texts through the subsequent fusion layer. This processing method not only improves the processing efficiency, but also enhances the expressiveness of the vision and language model when facing complex multimodal data, enabling it to more accurately classify, detect or describe the image content. The advantage of this architecture is that it can naturally fuse information from different sources, providing a solid foundation for solving complex visual language tasks.

[0064] Furthermore, on the cloud server side, it is also necessary to determine whether the text prompt word input by the user belongs to the "common category". In one or more embodiments of the present invention, the part of the visual language model deployed on the cloud server side can maintain a common category table. For example, the class in the category table can be the target detection classification in an existing data set (such as the COCO data set), and an additional empty class can also be set. After the language feature extractor outputs the language feature, the language feature can be mapped to a category through a decoder, so that the text prompt words input by the user can be mapped one by one to the common category table.

[0065] Therefore, the cloud server can traverse the feature sequences of each category name in the preset category table to determine whether the similarity threshold between the language feature and the feature sequence of each category name is greater than the similarity threshold (such as 0.7). If not, it is determined that there is at least part of the content in the text that does not correspond to all categories in the preset category table.

[0066] In addition, in order to avoid the problem that the user inputs a long sentence or the key category prompt words in the text are not coherent in the input, in one or more embodiments of the present invention, the soft token prediction method in the Modulated Detection for End-to-End Multi-Modal Understanding (MDETR) algorithm can be referred to, and the prompt word predictions at all positions in the input text are given weights that conform to uniform distribution. If a certain output category or multiple output categories of the prompt word do not belong to the 80 categories in the COCO dataset, but belong to the empty category, a "new category" instruction can be returned to the edge device to indicate that the user input content includes the target corresponding to the new category, so that the edge device uploads the visual features to the cloud server, and the cloud server can input the visual features extracted by the edge device and the language features extracted by itself into the feature fusion module for the next step. On the contrary, if all the output categories of the prompt word can correspond one-to-one with the categories in the common category table, a "no new category" instruction is returned to indicate that the user input content does not include the target corresponding to the new category.

[0067] Correspondingly, the edge device can judge whether it is necessary to upload the visual features to the cloud server for further processing according to the "new class" and "no new class" responses sent from the cloud, or put it into the detection head deployed in the edge device (i.e., the image detection model, which is trained by the corresponding images of each category in the preset category table) to directly output the target category and target position corresponding to the text in the image. There are two visual modules in the edge device, one visual detection backbone (deep separable convolutional visual feature extractor) and the other is the visual detection head. The detection head can use the traditional target detection design. For example, the YOLO detection head can be used here. This structure extracts features through a series of convolutional layers, and then uses the anchor box to generate the coordinates, category probability and confidence of the bounding box, and finally outputs the detection result.

[0068] Figure 7 is a schematic diagram of an algorithm flow of a visual feature extractor and a visual detection module in the present invention, Figure 8 The figure is a schematic diagram of the algorithm flow of a language feature extractor and a visual detection module in the present invention.

[0069] After completing the feature extraction of the image and text input by the user through the above, when the cloud server determines that the user input content includes targets corresponding to new categories, the edge device can upload the extracted visual features to the cloud server, and the cloud server will further process the features of both.

[0070] A feature fusion device can be deployed in the cloud server, and its input is the visual features output by the visual feature extractor and the language features output by the language feature extractor.

[0071] In order to effectively combine visual features and language features, a feature fuser can be constructed based on the GLIP (Grounded Language-Image Pre-training) model. Based on the feature fusion technology used in the GLIP model, it is improved. As an advanced multimodal learning framework, GLIP's feature fuser integrates visual features and language features in an innovative way to improve the overall recognition and classification performance.

[0072] In one or more embodiments of the present invention, the previously extracted visual features and language features are fused by a feature fuser. The feature fuser can be constructed by two parallel self-attention modules, followed by a cross-attention module from image to text and a cross-attention module from text to image, and two parallel multi-layer perceptrons. The feature fuser uses an attention mechanism, which is a powerful neural network structure that can dynamically calculate focus points between single instances in the sequence data within the model. In the feature fuser, the first self-attention module first performs attention weighting on the language features corresponding to different positions in the text, and the second self-attention module performs attention weighting on the visual features corresponding to different positions in the image.

[0073] Then, according to the query vector corresponding to the weighted visual features, the key vector and the value vector corresponding to the weighted language features, the visual features weighted by the attention weights of the text corresponding to the image are determined through the image-to-text cross-attention module. And according to the query vector corresponding to the weighted language features, the key vector and the value vector corresponding to the weighted visual features, the language features weighted by the attention weights of the image corresponding to the text are determined through the text-to-image cross-attention module. That is, the attention mechanism is not only used for the traditional one-way feature fusion from vision to language (that is, the visual features are used as the query key Q, and the language features are used as the key K and the value V to calculate the weighted attention of the language), but also adds a cross-attention calculation process, setting the language features as the query key Q, and the visual features as the key K and the value V, to increase the self-attention calculation of the image. That is, it is extended to bidirectional feature fusion. The application of this bidirectional self-attention mechanism not only improves the interactivity and depth of the visual language model in multimodal data processing, but also effectively utilizes the powerful computing power of the cloud server to improve processing efficiency and accuracy.

[0074] Finally, the first multi-layer perceptron extracts features based on the weighted visual features of the text corresponding to the image, and obtains enhanced visual features; the second multi-layer perceptron extracts features based on the weighted language features of the image corresponding to the text, and obtains enhanced language features. The series connection of multi-layer perceptrons enables higher flexibility and performance in processing complex visual language tasks. Fig. 9 As shown, Fig. 9 This is a schematic diagram of the algorithm structure of a feature fusion device in the present invention.

[0075] After feature fusion is performed by the feature fuser, the enhanced visual features that match the enhanced language features can be further determined based on the enhanced language features through a language-guided query module according to the similarity between the enhanced visual features and the enhanced language features, so as to obtain the visual language feature encoding.

[0076] The language-guided query module calculates the similarity between the enhanced visual features and the enhanced language features, sorts the similarities and selects the most matching enhanced visual features, thus realizing the association analysis between the image and text input by the user, and generating an enhanced visual feature index, which is used as the initialization query for the subsequent feature decoder. The enhanced visual feature index generated by the language-guided query module is called the visual language feature encoding. For example, at the current stage, 900 visual language feature encodings can be generated for each input image. Fig.10 As shown, Fig.10 The figure is a flow chart of the algorithm of a language-guided query module in the present invention. When calculating the similarity between the enhanced language features and the enhanced visual features, the similarity result can be calculated by dot product. Subsequently, a preset number of visual language feature codes with high similarity can be selected by similarity sorting for target positioning and target classification tasks in subsequent steps.

[0077] After the language-guided query module obtains multiple visual language feature codes based on the language, the visual language feature codes can be further decoded by a feature decoder. The feature decoder may include a self-attention module, an image cross-attention module, a text cross-attention module and a multi-layer perceptron connected in series. In the feature decoder, first, the visual language feature codes can be weighted by attention by a third self-attention module.

[0078] Then, according to the query vector corresponding to the weighted visual language feature encoding, the key vector and the value vector corresponding to the enhanced visual feature, the image cross-attention module determines the first weighted fusion feature corresponding to the visual language feature encoding. And according to the query vector corresponding to the first weighted fusion feature, the key vector and the value vector corresponding to the enhanced language feature, the text cross-attention module determines the second weighted fusion feature corresponding to the visual language feature encoding.

[0079] Finally, feature extraction is performed according to the second weighted fusion feature through the third multi-layer perceptron, and the target to be detected corresponding to the text in the image is determined through the regional regressor and the classifier based on the enhanced language feature.

[0080] Among them, first, the paired visual language feature encodings are sent to the feature decoder and subjected to a self-attention process. During this process, the various elements within the visual language feature encoding interact with each other, enhancing the connection and consistency between the features. Subsequently, these visual language feature encodings processed by self-attention first interact with the enhanced visual features. At this time, the output in the feature decoder acts as the query vector Q, and the enhanced visual features participate in the calculation as key-value pairs K and V to capture and strengthen the contextual details related to the visual information. After completing the processing of the image information, the decoder uses the updated visual feature encoding as a new query vector Q and interacts with the enhanced language features for the second time. Similarly, the enhanced language features play the role of key-value pairs K and V at this stage, thereby further deepening the understanding and integration of the text content. Finally, the output features are passed through the third multi-layer perceptron to output the final query key output of the feature decoder. The number of query keys output by the decoder is consistent with the number of keys generated by the query module in the previous step, and is used for subsequent regression and classification tasks. As Fig.11 As shown, Fig.11 It is a schematic diagram of the algorithm structure of a feature decoder in the present invention.

[0081] Through the hierarchical and complementary processing steps of the feature decoder, not only the conversion process from raw input to complex feature representation is optimized, but also the accuracy and adaptability of the final visual language model output are effectively improved through precise attention regulation.

[0082] When training the visual language model in one or more embodiments of the present invention, it can be based on the following loss function.

[0083] In traditional object detection networks, the input image is first processed by a feature extractor consisting of a convolutional neural network to extract the deep features of the image. These extracted features are then fed into two specialized predictors: a region regressor ( ), responsible for predicting the location of the target; the second is the classifier ( ), used to determine the category of the target. The outputs of these two predictors will be used to calculate two types of losses: classification loss and regression loss The classification loss is responsible for evaluating the accuracy of the classification results, while the regression loss is responsible for evaluating the accuracy of the target position prediction. It is expressed as: .

[0084] In one or more embodiments of the present invention, the feature fuser, the language-guided query module, and the feature decoder are trained in the following manner:

[0085] First, obtain the sample image and the sample text corresponding to each target in the sample image, and mark the corresponding relationship between the position of the real bounding box of each target in the sample image and the real text description of the real bounding box.

[0086] Then, the language features of the sample text are extracted, and the visual features extracted by the edge device from the sample image are obtained; based on the language features of the sample text and the visual features of the sample image, the position of the predicted bounding box of the target to be detected corresponding to the text and the predicted text description of the predicted bounding box are determined through a feature fuser, a language-guided query module and a feature decoder.

[0087] Finally, the mean absolute error (MAE) loss and the generalized intersection over union (GIOU) loss are determined according to the deviation between the position of the true bounding box and the position of the predicted bounding box, and the regression loss is determined according to the mean absolute error loss and the generalized intersection over union loss; the classification loss is determined according to the deviation between the true text description of the true bounding box and the predicted text description of the predicted bounding box; the total loss is determined according to the regression loss and the classification loss, and the feature fuser, the language-guided query module and the feature decoder are trained with the goal of minimizing the total loss.

[0088] In order to enable the visual language model to have the ability of open set object detection, the present invention makes necessary modifications to the classification loss function so that it can align the region classification task with the features extracted by the language feature extractor.

[0089] Specifically, this process involves matching language cues to region categories in the image. Defining Image Features , which comes from the image feature extractor For images Processing: ; Also defines language features , which is done by a language feature extractor Prompt words The calculation results are: .

[0090] On this basis, the output classification score of the visual language model is expressed as The loss function of the classifier can be expressed as ,in is a target matrix calculated by matching between categories and regions, is the cross entropy loss function. Specifically, the construction of this matrix in the present invention follows the idea of ​​DETR, and the target matrix is ​​calculated by the binary Hungarian matching method. However, since the lengths of the prompt words provided by the language model in the open-set target detection task vary, the classification categories of the target matrix are not fixed. Therefore, the present invention innovatively reconstructs this matrix.

[0091] In existing studies, the length of the classification matrix is ​​generally consistent with the length of the number of classifications to be matched. Let the length of the number of classifications be Then the matrix dimension of the output classification score in the traditional model is .in is the number of query keys output by the decoder, corresponding to .

[0092] In the visual language model of the present invention, the length of the text prompt word is difficult to be fixed, and it may be a sentence or a phrase, and often multiple prompt words represent a type.

[0093] Therefore, when using the binary Hungarian matching algorithm, the original target matrix The dimension is expanded to ,in Is the length of the input prompt word. Normally The length is greater than When calculating the loss function matching, if a phrase matches an image region (positive match), all the prompt words corresponding to the phrase are marked as positive in the target matrix, and all additional marks are negatively matched with all image features. This setting ensures that the calculation of the loss function can accurately reflect the degree of matching of the visual language model to the phrase in the image region and the relevant text.

[0094] In the present invention, the regression loss function The method in the DETR model is adopted to regress the bounding box using the query key provided by the feature decoder. Specifically, the present invention uses the mean absolute error loss (L1 loss) and GIOU loss to optimize the prediction accuracy of the bounding box. The L1 loss helps to reduce the mean absolute error between the predicted box and the true box, while the GIOU loss further considers the intersection-over-union ratio between the bounding boxes, as well as their shape and size differences, which makes the model more accurate when dealing with objects with complex overlaps and different scales. This enables the model to effectively identify and process unseen categories in a wider range of application scenarios, improving the model's adaptability to novel and changing scenarios.

[0095] based on Figure 2In the open set target detection method based on the cloud-edge architecture shown, the cloud server can obtain the text input by the user uploaded by the edge device and extract the language features in the text. The edge device will simultaneously obtain the image input by the user, and the text includes the category of the target to be detected in the image. Then the cloud server can classify and map the language features based on the preset category table to determine whether the categories described in the text all appear in the preset category table. If there are new categories, the visual features extracted by the edge device from the image input by the user are obtained, so that the language features and visual features are fused by the feature fusion device to obtain enhanced language features and enhanced visual features, and the enhanced visual features matching the enhanced language features are determined by the query module guided by the language according to the similarity between the enhanced visual features and the enhanced language features, and the visual language feature encoding is obtained, and the target to be detected corresponding to the text in the image is determined by the feature decoder according to the visual language feature encoding and sent to the edge device. If there is no new category, the edge device is enabled to perform target detection based on the image input by the user. The edge device is deployed with an image detection model, which is trained by the images corresponding to each category in the preset category table. Among them, the feature fuser is constructed by connecting two parallel self-attention modules, followed by a cross-attention module from image to text and a cross-attention module from text to image in series, and then connecting two parallel multi-layer perceptrons in series.

[0096] The present invention splits the open set object detection task combining text and image, and splits the visual language model at the same time. Since the computing resources of edge devices are generally tight, the image processing task is placed on the edge device, and features of the image are extracted through the edge device. Cloud servers generally have more computing resources, so the text processing task that requires a lot of computing resources is placed on the cloud server.

[0097] Furthermore, in practical applications, open-set target detection may have new categories that are not learned during model training, or may not have new categories. On the one hand, the present invention deploys a pre-trained image detection model on the edge device, and on the other hand, deploys a preset category table of text in the cloud server, wherein the image detection model is obtained by training the corresponding images of each category in the preset category table. Thus, in the process of processing the text by the cloud server with more computing resources, the feature information contained in the text is used to guide the entire open-set target detection task.

[0098] When the cloud server finds that no new categories appear during text processing, it means that the image detection model deployed on the edge device has learned all the information about the targets to be detected in the image during training. At this time, it can be regarded as closed-set target detection. No text is needed to guide the open-set target detection process. Target detection can be performed directly through the edge device, further improving the detection response speed when there are no new categories.

[0099] When a new category appears in the text processing of the cloud server, the edge device uploads the visual features corresponding to the image to the cloud server, and cross-processes the visual features and text features based on the image-to-text cross-attention module and the text-to-image cross-attention module in the feature fusion module, and deeply fuses the two, so that the visual language feature encoding is further matched through the language-guided query module, and decoded through the feature decoder. Through the above processing, the interactivity and depth in the multimodal data processing of text and images are improved, and the powerful computing power of the cloud server is effectively utilized to improve the efficiency and accuracy of open-set target detection.

[0100] Cloud-edge collaborative computing is a computing architecture that distributes data processing tasks between cloud servers and edge devices. In this mode, edge devices usually handle tasks that are more real-time and require high response speed, while tasks with higher complexity and greater computing requirements are transmitted to the cloud for processing. This division of labor utilizes the powerful computing power of the cloud and the geographical advantages of edge devices close to users, which can effectively reduce latency and improve data processing efficiency. It can also optimize bandwidth usage and reduce energy consumption by reducing the amount of transmitted data.

[0101] Deploying the open-set object detection visual language model of the present invention in a cloud-edge collaborative computing environment provides a series of significant advantages. On the one hand, by optimizing the visual language model architecture and algorithm, the edge model deployed on the edge device can achieve high detection accuracy while maintaining lightweight, solving the problem of poor performance of traditional edge models due to their small scale; on the other hand, the deployment flexibility of edge devices allows the system to be quickly deployed in various environments, whether in urban monitoring systems or in wildlife monitoring in remote areas, it can work effectively.

[0102] In addition, cloud servers can handle more complex language understanding tasks and conduct in-depth learning, which is particularly important for open-set object detection because it involves identifying and understanding categories that do not appear in the training data. The powerful computing power of cloud servers allows models to be trained on larger data sets and use more complex algorithms, thereby improving overall detection accuracy and the generalization ability of the model. At the same time, cloud servers can also update models and algorithms in real time to ensure that the system always runs the most optimized and latest technologies to cope with changing application needs and environments.

[0103] In order to efficiently utilize cloud-edge resources, the present invention first places the language feature extractor on the cloud server. Language processing usually requires a lot of computing resources and storage space, especially when using deep learning models such as BERT network models for complex natural language processing tasks. These models usually have many parameters and high hardware requirements. Edge devices such as smartphones or embedded systems may not be able to meet these requirements. Secondly, language feature extractors often need to access a large amount of corpus data for effective language understanding, and the storage and processing of this data may exceed the capabilities of edge devices. Finally, placing the language feature extractor in the cloud can keep the model up to date, because updating and maintaining the model in the cloud is much easier and more cost-effective than updating it separately on each edge device. This ensures that the most advanced language processing technology is used to provide more accurate services.

[0104] In order to effectively realize the collaborative computing of cloud, edge and terminal, this paper specially designs a lightweight visual feature extractor based on depthwise separable convolution. This design makes the visual feature extractor not only have lower computational complexity, but also more suitable for deployment on resource-constrained edge devices to efficiently extract visual features of images.

[0105] In addition, the input and output tensor dimensions of the visual feature extractor on the edge device and the language feature extractor on the cloud server are consistent. Let the visual feature extractor on the edge device be represented as , and then the language feature extractor of the cloud server is expressed as .

[0106] Get the input image on the edge device and input text After that, the image is input to the visual feature extractor of the edge device to obtain visual features: ; At the same time, the input text Transmitted to the cloud server, the cloud server receives the input text Then the language features are obtained through the language feature extractor: .

[0107] In practical applications, open-set object detection algorithms do not always encounter completely new categories that have never been seen before, and often need to process some common categories. When processing common categories, the present invention needs to make full use of edge devices to avoid repeated calculations in the cloud. To this end, the present invention uses a preset category table, for example, using the classification standard of the COCO dataset.

[0108] In the cloud server, the present invention newly constructs a language feature decoder, which is connected in series with the language feature extractor. This language feature decoder is consistent with the general language model, that is, the final text output is generated through the self-attention mechanism and the inter-layer attention mechanism. The input is sent to the language feature decoder, which matches these language features with all categories in the predefined category table by calculating similarity. If each phrase or associated phrase in the input text prompt word or sentence can be completely matched with one or more categories in the category table, the system will output a "no new category" instruction, which means that all the recognized categories are known categories. On the contrary, if the mapping confidence of the input prompt is low, or a paragraph or phrase in the prompt word cannot be matched with any category in the category table, the system will output a "new category" instruction, which means that the text prompt word entered by the user may have an unknown category.

[0109] When the system outputs the "no new class" instruction, the instruction is sent to the edge device. On the edge device, the pre-trained detection head based on the COCO dataset will receive and process the visual features output by the feature extractor on the edge. This process involves locating and classifying objects in the image, and then the edge device feeds back the detection results (including the location and category of the object) to the user. This design ensures that the system can respond quickly to user needs while reducing the amount of data transmission and improving processing efficiency.

[0110] When the system outputs the "new class" instruction, the instruction is also sent to the edge device. Next, the edge device uploads the visual features extracted by the visual feature extractor to the cloud server. The cloud server then combines these visual features with the language features extracted by itself and inputs them into the feature fuser, language-guided query module, and feature decoder for processing. These components jointly perform the open set prediction step to identify and process new categories. This process ensures that the system can effectively process and identify unknown categories, improving the adaptability and accuracy of the model.

[0111] In addition, the present invention provides an application scenario in a safe production scenario, wherein the user's text input is a task to be inspected in the safe production scenario, such as manual behavior detection, and the image input is a video frame captured by a camera, and the image size can be flexibly input.

[0112] For example, the input text can be "person with goggles" or "person operating crane", and some additional text can be entered to describe the details, such as "person wearing red safety helmet". In complex safety scenarios, a single frame of image can perform multiple detection tasks at the same time. In this case, different classes in the input text need to be separated by the symbol ".", such as "person with goggles . belt conveyor". When this term is entered, the algorithm will output the location of the person wearing goggles and the location of the belt conveyor at the same time.

[0113] For the feature extraction module, in the industrial safety production scenario, each frame of the image to be inspected captured by the camera will be sent to the visual feature extractor, and the visual features will be obtained through deep separable convolution. The visual feature extractor is deployed in the edge device, which is generally located at the scene of safety production, close to the location of the camera.

[0114] The language feature extractor is placed in the cloud server due to its large backbone network structure. The edge device sends a collaborative instruction to the cloud server, that is, sends text to the cloud server to obtain the target location. The text entered by the user will pass through the BERT network model in the cloud server to extract language features. In addition, this language feature will pass through a language feature decoder and determine whether the prompt word or text entered by the user is in the common category table maintained by the current algorithm. In the application scenario of safe production, if the user inputs "person. belt conveyor. person with goggles", after passing through the language feature decoder, the following steps will be generated one by one: the language feature of "person" will be more than 90% similar to the same "person" in the common category table, so the worker category will be considered to be in the common category table, and the cloud device will issue a "no new category" instruction to the edge device for subsequent reasoning; the language features generated by "belt conveyor" and "person with goggles" after the language feature extractor cannot achieve 50% similarity with any class in the common category table, so the cloud device will issue a "new category" instruction.

[0115] After the system outputs a new class of instructions, the cloud server will receive the visual features of the video frames in the safety production scene uploaded by the edge device, and then pass the language features extracted from the text prompts entered by the user together with the visual features into the language-guided query module, and then input the fused features into the feature decoder, and finally output the position coordinate information of the object in each text prompt in the original video frame. The position coordinate information is sent to the edge device in the form of a detection box, that is, the position of the detection box determined by the upper left corner coordinates and the lower right corner coordinates.

[0116] Compared with traditional vertical-specific target detection algorithms, the open set target detection model based on cloud-edge collaboration in the selection and configuration of hardware devices in the present invention has significant inherent advantages over traditional vertical-specific target detection algorithms. Under this architecture, edge devices only need to deploy a lightweight visual detection model to start preliminary detection, which greatly reduces the demand for edge computing resources. Larger detection models are deployed on cloud servers, and these models are only called for deep reasoning when faced with more complex situations. This deployment strategy not only optimizes the use of resources and reduces the burden on edge devices, but also enables the system to respond more flexibly to various complex scenarios, improving overall processing efficiency and accuracy.

[0117] As for the design of visual language models, in real-world application scenarios such as safe production, the phenomena that visual language model design needs to deal with are often complex and diverse. In most cases, traditional vertical field-specific algorithms need to be deployed in multiples to barely meet a single business need. In contrast, open set target detection algorithms have significant flexibility and wide applicability, and one algorithm can cover multiple business scenarios. For example, in safe production, the same open set target detection algorithm can be used to monitor workers' manual behavior and is also suitable for equipment safety environment monitoring. In addition, the applicability of the same algorithm is not limited to safe production scenarios, but can also be applied to other fields such as autonomous driving.

[0118] In terms of visual language model deployment, the cloud-edge open-set object detection algorithm can meet the different needs of multiple scenarios through a unified model. In this configuration, the edge device only needs to deploy a small visual feature extractor to start executing the entire algorithm. This deployment strategy not only maintains the reasoning speed of the visual language model, but also the accuracy of the visual language model is supported and guaranteed by the large language feature extractor deployed on the cloud server and the large related model of feature fusion. Through this cloud-edge collaborative approach, efficient and accurate data processing can be achieved, effectively combining the real-time response capability of edge computing and the powerful computing power of cloud computing.

[0119] In terms of visual language model optimization, in the subsequent optimization operation and maintenance phase of the visual language model, the cloud-edge open set target detection method provided by the present invention has significant advantages over the traditional target detection method. First, even in the face of different industrial environments, or detecting targets in different states, the visual language model of the present invention can change the output direction of the detection through the support of text prompts, and realize the characteristics of detecting any transaction, which greatly reduces the update and optimization frequency of the visual language model. Secondly, since the main part of the visual language model is deployed on the cloud server, it can be easily updated and optimized in a centralized manner without updating each edge device one by one. This centralized management method not only improves the update efficiency, but also ensures that all edge devices can synchronize the latest visual language model improvements and corrections. In addition, the powerful computing resources in the cloud can be used to analyze a large amount of data collected from the edge, help identify potential problems and improvement space of the algorithm, and further improve the overall performance and accuracy of the visual language model. Through this cloud-edge collaborative mode, the maintenance and optimization process of the visual language model is more efficient and flexible, greatly reducing the operating cost and complexity, and also enhancing the adaptability and reliability of the visual language model. This gives the open-set target detection method based on the cloud-edge architecture obvious advantages in continuous operation and maintenance, and is particularly suitable for modern industrial applications that require rapid iteration and continuous optimization.

[0120] The above is an open set target detection method based on the cloud-edge-end architecture provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding open set target detection device based on the cloud-edge-end architecture on the cloud server side, such as Fig.12 shown.

[0121] Fig.12 A schematic diagram of an open set target detection device based on a cloud-edge-end architecture on a cloud server side provided by the present invention includes:

[0122] The cloud acquisition module 301 is used to acquire the text input by the user uploaded by the edge device and extract the language features in the text; the text includes the category of the target to be detected in the image uploaded by the user to the edge device;

[0123] The judgment and detection module 302 is used to classify and map the language features according to the preset category table to determine whether the categories described in the text all appear in the preset category table; if so, an instruction is sent to the edge device to indicate that the user input content does not include the target to be detected of the new category; the edge device performs target detection based on the image input by the user through the image detection model; the image detection model is trained by the images corresponding to each category in the preset category table; if not, an instruction is sent to the edge device to indicate that the user input content includes the target to be detected of the new category, and the visual features extracted by the edge device from the image input by the user are obtained; through the feature fusion device The language features and the visual features are fused to obtain enhanced language features and enhanced visual features, and the enhanced visual features that match the enhanced language features are determined by a language-guided query module based on the similarity between the enhanced visual features and the enhanced language features to obtain visual language feature encoding, and the target to be detected corresponding to the text in the image is determined by a feature decoder based on the visual language feature encoding and sent to the edge device; wherein the feature fuser is constructed by two parallel self-attention modules, followed by a cross-attention module from image to text and a cross-attention module from text to image, and then two parallel multi-layer perceptrons.

[0124] The present invention also provides a corresponding open set target detection device based on the cloud-edge-end architecture on the edge device side, such as Fig.13 shown.

[0125] Fig.13 A schematic diagram of an open set target detection device based on a cloud-edge-end architecture on an edge device side provided by the present invention includes:

[0126] The edge acquisition module 401 is used to acquire text and images input by a user, wherein the text includes the category of the target to be detected in the image;

[0127] The feature extraction module 402 is used to extract visual features from the image and upload the text to the cloud server, so that the cloud server extracts language features from the text and classifies and maps the language features according to the preset category table to determine whether the categories described in the text all appear in the preset category table;

[0128] The detection module 403 is used to determine the target to be detected by using an image detection model according to visual features and return it to the user when receiving an instruction sent by the cloud server indicating that the user input content does not include a target to be detected of a new category; the image detection model is trained by using images corresponding to each category in a preset category table;

[0129] The transmission module 404 is used to upload the visual features to the cloud server when receiving an instruction sent by the cloud server that represents the user input content including a new category of targets to be detected, so that the cloud server can fuse the language features and the visual features through a feature fuser to obtain enhanced language features and enhanced visual features, and determine the enhanced visual features that match the enhanced language features according to the similarity between the enhanced visual features and the enhanced language features through a language-guided query module to obtain the visual language feature encoding, and determine the targets to be detected in the image corresponding to the text according to the visual language feature encoding through a feature decoder; receive the targets to be detected in the image corresponding to the text sent by the cloud server and return them to the user.

[0130] For the specific limitations of the open-set target detection device based on the cloud-edge-end architecture on the cloud server side and the edge device side, please refer to the limitations of the open-set target detection method based on the cloud-edge-end architecture mentioned above, which will not be repeated here. Each module in the above-mentioned open-set target detection device based on the cloud-edge-end architecture can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0131] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 2 or Figure 3 The open set target detection method based on cloud-edge architecture is provided.

[0132] The present invention also provides a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 2 or Figure 3 The open set target detection method based on cloud-edge architecture is provided.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0134] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. An open set target detection method based on cloud-edge-device architecture, characterized in that: The cloud server performs the following steps, including: Obtaining text input by a user uploaded by an edge device and extracting language features from the text; the text includes the category of the target to be detected in the image uploaded by the user to the edge device; Classify and map the language features according to the preset category table to determine whether the categories described in the text all appear in the preset category table; If so, an instruction is sent to the edge device indicating that the user input content does not include the target to be detected of the new category; the edge device is enabled to perform target detection based on the image input by the user through an image detection model; the image detection model is trained by images corresponding to each category in a preset category table; If not, an instruction is sent to the edge device to characterize that the user input content includes the new category of the target to be detected, and the visual features extracted by the edge device from the user input image are obtained; the language features and the visual features are fused by the feature fusion device to obtain enhanced language features and enhanced visual features, and the enhanced visual features that match the enhanced language features are determined by the query module guided by the language according to the similarity between the enhanced visual features and the enhanced language features, and the visual language feature encoding is obtained, and the target to be detected corresponding to the text in the image is determined by the feature decoder according to the visual language feature encoding and sent to the edge device; The feature fusion device is constructed by connecting two parallel self-attention modules, followed by a cross-attention module from image to text and a cross-attention module from text to image, and connecting two parallel multi-layer perceptrons in series. The feature fuser, language-guided query module and feature decoder are trained in the following way: Obtain sample images and sample texts corresponding to each target in the sample images, and mark the corresponding relationship between the position of the real bounding box of each target in the sample image and the real text description of the real bounding box; Extracting language features of the sample text and obtaining visual features extracted by the edge device from the sample image; determining the position of the predicted bounding box of the target to be detected corresponding to the text and the predicted text description of the predicted bounding box according to the language features of the sample text and the visual features of the sample image through a feature fuser, a language-guided query module, and a feature decoder; The mean absolute error loss and the generalized intersection-over-union loss are determined according to the deviation between the position of the true bounding box and the position of the predicted bounding box, so as to determine the regression loss according to the mean absolute error loss and the generalized intersection-over-union loss; the classification loss is determined according to the deviation between the true text description of the true bounding box and the predicted text description of the predicted bounding box; the total loss is determined according to the regression loss and the classification loss, and the feature fuser, the language-guided query module and the feature decoder are trained with the goal of minimizing the total loss.

2. The open set target detection method based on cloud-edge-device architecture according to claim 1, characterized in that: The preset category table includes the correspondence between the category names of multiple categories and the feature sequences of the category names; The classifying and mapping the language features according to the preset category table to determine whether the categories described in the text all appear in the preset category table specifically includes: Traverse the feature sequences of each category name in the preset category table, and determine whether the similarity threshold between the language feature and the feature sequence of each category name is greater than the similarity threshold; If not, it is determined that at least part of the content in the text does not appear in the preset category table.

3. The open set target detection method based on cloud-edge-device architecture according to claim 1, characterized in that: The feature fusion of the language features and the visual features by the feature fusion device to obtain the enhanced language features and the enhanced visual features specifically includes: The first self-attention module is used to weight the language features corresponding to different positions of the text, and the second self-attention module is used to weight the visual features corresponding to different positions of the image; According to the query vector corresponding to the weighted visual features, the key vector and the value vector corresponding to the weighted language features, the visual features weighted by the attention weights of the text corresponding to the image are determined through the image-to-text cross attention module; According to the query vector corresponding to the weighted language feature, the key vector and the value vector corresponding to the weighted visual feature, the language feature weighted by the attention weight of the image corresponding to the text is determined through the text-to-image cross attention module; The first multi-layer perceptron performs feature extraction based on visual features weighted by the attention weights of the text corresponding to the image, thereby obtaining enhanced visual features. The second multi-layer perceptron performs feature extraction based on language features weighted by the attention weights of the image corresponding to the text, thereby obtaining enhanced language features.

4. The open set target detection method based on cloud-edge-device architecture according to claim 3, characterized in that: The feature decoder includes a self-attention module, an image cross-attention module, a text cross-attention module and a multi-layer perceptron connected in series; The step of determining the target to be detected corresponding to the text in the image according to the visual language feature coding by the feature decoder specifically includes: The visual language feature encoding is weighted by attention through the third self-attention module; According to the query vector corresponding to the weighted visual language feature encoding, the key vector and the value vector corresponding to the enhanced visual feature, a first weighted fusion feature corresponding to the weighted visual language feature encoding of the image is determined through an image cross attention module; According to the query vector corresponding to the first weighted fusion feature, the key vector and the value vector corresponding to the enhanced language feature, a text cross-attention module is used to determine a second weighted fusion feature corresponding to the text after the visual language feature encoding is weighted; Feature extraction is performed according to the second weighted fusion feature through a third multi-layer perceptron, and based on the enhanced language feature, a target to be detected corresponding to the text in the image is determined through a regional regressor and a classifier.

5. An open set target detection method based on cloud-edge-device architecture, characterized in that: The edge device performs the following steps, including: Acquire text and an image input by a user, wherein the text includes the category of the target to be detected in the image; Extract visual features from the image and upload the text to the cloud server, so that the cloud server extracts language features from the text and classifies and maps the language features according to a preset category table to determine whether the categories described in the text all appear in the preset category table; When receiving an instruction sent by the cloud server indicating that the user input content does not include a target to be detected of a new category, determining the target to be detected by an image detection model based on visual features and returning it to the user; the image detection model is trained by images corresponding to each category in a preset category table; When receiving an instruction sent by the cloud server indicating that the user input content includes a new category of objects to be detected, uploading the visual features to the cloud server, allowing the cloud server to fuse the language features and the visual features through a feature fusion device to obtain enhanced language features and enhanced visual features, and determining the enhanced visual features that match the enhanced language features according to the similarity between the enhanced visual features and the enhanced language features through a language-guided query module, obtaining a visual language feature encoding, and determining the object to be detected in the image corresponding to the text according to the visual language feature encoding through a feature decoder; receiving the object to be detected in the image corresponding to the text sent by the cloud server and returning it to the user; The feature fuser, language-guided query module and feature decoder are trained in the following way: Obtain sample images and sample texts corresponding to each target in the sample images, and mark the corresponding relationship between the position of the real bounding box of each target in the sample image and the real text description of the real bounding box; Extracting language features of the sample text and obtaining visual features extracted by the edge device from the sample image; determining the position of the predicted bounding box of the target to be detected corresponding to the text and the predicted text description of the predicted bounding box according to the language features of the sample text and the visual features of the sample image through a feature fuser, a language-guided query module, and a feature decoder; The mean absolute error loss and the generalized intersection-over-union loss are determined according to the deviation between the position of the true bounding box and the position of the predicted bounding box, so as to determine the regression loss according to the mean absolute error loss and the generalized intersection-over-union loss; the classification loss is determined according to the deviation between the true text description of the true bounding box and the predicted text description of the predicted bounding box; the total loss is determined according to the regression loss and the classification loss, and the feature fuser, the language-guided query module and the feature decoder are trained with the goal of minimizing the total loss.

6. The open set target detection method based on cloud-edge-device architecture according to claim 1 or 5, characterized in that: The language features in the text are extracted through the pre-trained BERT network model, and the visual features in the image are extracted through the pre-trained deep separable convolutional network model.

7. An open set target detection device based on cloud-edge-device architecture, characterized in that: include: The cloud acquisition module is used to obtain the user input text uploaded by the edge device and extract the language features in the text; The text includes the category of the target to be detected in the image uploaded by the user to the edge device; A judgment and detection module is used to classify and map language features according to a preset category table to determine whether the categories described in the text all appear in the preset category table; If yes, an instruction is sent to the edge device indicating that the user input content does not include the new category of targets to be detected; The edge device performs target detection according to the image input by the user through an image detection model; the image detection model is trained by the images corresponding to each category in the preset category table; If not, an instruction is sent to the edge device to characterize that the user input content includes a new category of targets to be detected, and the visual features extracted by the edge device from the user input image are obtained; the language features and the visual features are fused by a feature fusion device to obtain enhanced language features and enhanced visual features, and the enhanced visual features that match the enhanced language features are determined by a query module guided by a language according to the similarity between the enhanced visual features and the enhanced language features, and the visual language feature encoding is obtained, and the target to be detected corresponding to the text in the image is determined by a feature decoder according to the visual language feature encoding and sent to the edge device; wherein the feature fusion device is constructed by two parallel self-attention modules, followed by a cross-attention module from image to text and a cross-attention module from text to image in series, and two parallel multi-layer perceptrons in series; The feature fuser, language-guided query module and feature decoder are trained in the following way: Obtain sample images and sample texts corresponding to each target in the sample images, and mark the corresponding relationship between the position of the real bounding box of each target in the sample image and the real text description of the real bounding box; Extract the language features of the sample text and obtain the visual features extracted by the edge device from the sample image; According to the language features of the sample text and the visual features of the sample image, the position of the predicted bounding box corresponding to the object to be detected in the text and the predicted text description of the predicted bounding box are determined through a feature fuser, a language-guided query module and a feature decoder; The mean absolute error loss and the generalized intersection-over-union loss are determined according to the deviation between the position of the true bounding box and the position of the predicted bounding box, so as to determine the regression loss according to the mean absolute error loss and the generalized intersection-over-union loss; the classification loss is determined according to the deviation between the true text description of the true bounding box and the predicted text description of the predicted bounding box; the total loss is determined according to the regression loss and the classification loss, and the feature fuser, the language-guided query module and the feature decoder are trained with the goal of minimizing the total loss.

8. An open set target detection device based on cloud-edge-end architecture, characterized in that: include: The edge acquisition module is used to acquire the text and image input by the user, wherein the text includes the category of the target to be detected in the image; A feature extraction module is used to extract visual features from the image and upload the text to the cloud server, so that the cloud server extracts language features from the text and classifies and maps the language features according to a preset category table to determine whether the categories described in the text all appear in the preset category table; A detection module, for determining the target to be detected by an image detection model based on visual features and returning the target to the user when receiving an instruction from a cloud server indicating that the user input content does not include a target to be detected of a new category; the image detection model is trained by images corresponding to each category in a preset category table; The transmission module is used for uploading the visual features to the cloud server when receiving an instruction sent by the cloud server indicating that the user input content includes a new category of the target to be detected, so that the cloud server performs feature fusion on the language features and the visual features through a feature fusion device to obtain enhanced language features and enhanced visual features, and determines the enhanced visual features that match the enhanced language features according to the similarity between the enhanced visual features and the enhanced language features through a query module guided by language, obtains the visual language feature encoding, and determines the target to be detected in the image corresponding to the text according to the visual language feature encoding through a feature decoder; receives the target to be detected in the image corresponding to the text sent by the cloud server and returns it to the user; The feature fuser, language-guided query module and feature decoder are trained in the following way: Obtain sample images and sample texts corresponding to each target in the sample images, and mark the corresponding relationship between the position of the real bounding box of each target in the sample image and the real text description of the real bounding box; Extract the language features of the sample text and obtain the visual features extracted by the edge device from the sample image; According to the language features of the sample text and the visual features of the sample image, the position of the predicted bounding box corresponding to the object to be detected in the text and the predicted text description of the predicted bounding box are determined through a feature fuser, a language-guided query module and a feature decoder; The mean absolute error loss and the generalized intersection-over-union loss are determined according to the deviation between the position of the true bounding box and the position of the predicted bounding box, so as to determine the regression loss according to the mean absolute error loss and the generalized intersection-over-union loss; the classification loss is determined according to the deviation between the true text description of the true bounding box and the predicted text description of the predicted bounding box; the total loss is determined according to the regression loss and the classification loss, and the feature fuser, the language-guided query module and the feature decoder are trained with the goal of minimizing the total loss.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Images