Homologous image multi-task analysis method and device, storage medium and electronic equipment

Abstract

Embodiments of the invention disclose a homologous image multi-task analysis method and apparatus, a storage medium and an electronic device. The method comprises the steps of firstly obtaining an image block of an input image and a target task configured for the input image; then determining a task analysis prompt based on each target task of the input image; and finally, based on the image blocks and task analysis prompts, generating an input image multi-task analysis result.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, storage medium and electronic device for multi-task analysis of homologous images. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology and the Internet, a large amount of image and text data has provided rich training resources for machine learning models, greatly improving the performance of the models. In particular, for the image-text contrast learning base model, it can be applied to new tasks without a large amount of downstream task data. Among them, CLIP (Contrastive Language-Image Pre-training) is a representative model that uses image-text pairing data on the Internet for pre-training, that is, data that matches images with their corresponding text descriptions. CLIP uses a contrast learning strategy to make the representations of images and texts describing the same content close in the latent space, while the representations of images and texts describing different content remain far apart. This training method enables the model to learn with zero samples, that is, it can be directly applied to new tasks that have never been seen, such as image classification.

[0003] However, although the CLIP model has shown strong versatility in a variety of tasks, its generalization ability may be limited when dealing with certain specific, highly specialized tasks or fields. Therefore, there is an urgent need for a solution that can support multiple pluggable tasks while keeping computing resources basically unchanged, so as to effectively cope with the ever-changing task requirements and improve the flexibility and scalability of the model. Summary of the invention

[0004] The embodiment of this specification provides a method for multi-task parsing of homologous images. The method determines a task parsing hint for each target task, and generates a multi-task parsing result of the input image based on the image block of the input image and the task parsing hint, thereby improving the processing efficiency when coordinating multiple tasks and flexibly coping with different tasks. The method includes:

[0005] Obtaining an image patch of an input image and a target task assigned to the input image;

[0006] Determining a task parsing hint for each target task based on the input image;

[0007] Based on the image block and the task parsing hint, a multi-task parsing result of the input image is generated.

[0008] Further, in some implementations, generating the input image multi-task parsing result based on the image block and the task parsing hint includes:

[0009] Extracting feature data of each current image block to obtain different types of feature information corresponding to each current image block;

[0010] The features of the same type contained in each of the current image blocks are merged to obtain a new image block;

[0011] Fusing all feature information in the new image block with each current task analysis prompt respectively to obtain fused data corresponding to each current task analysis prompt;

[0012] The new image block is used as the current image block, and the fused data is used as the current task analysis prompt, and the above steps are repeated until the output conditions are met, and the multi-task analysis result of the input image is output based on the fused data corresponding to the current task analysis prompt obtained last time.

[0013] Furthermore, in some implementations, before outputting the input image multi-task parsing result based on the fusion data corresponding to the current task parsing prompt obtained last time, the method further includes:

[0014] The fused data corresponding to the current task analysis prompt obtained last time is classified and processed to obtain the input image multi-task analysis result.

[0015] Furthermore, in some embodiments,

[0016] Before obtaining the image block of the input image, the method further includes:

[0017] The input image is segmented to obtain image blocks.

[0018] The embodiment of this specification also provides a method for multi-task parsing of homologous images, the method comprising:

[0019] Obtaining an image patch of an input image and a target task assigned to the input image;

[0020] Determining a task parsing hint for each target task based on the input image;

[0021] The image block and the task parsing prompt are input into a multi-task parsing model to generate a multi-task parsing result of the input image.

[0022] Further, in some embodiments, the multi-task parsing model includes a plurality of sequentially superimposed feature extraction sub-networks based on a self-attention mechanism;

[0023] The step of inputting the image block and the task parsing hint into the multi-task parsing model to generate the input image multi-task parsing result comprises:

[0024] Inputting the current image block and the current task analysis prompt into the current feature extraction subnetwork based on the self-attention mechanism to obtain fused data corresponding to the current task analysis prompt and multiple new image blocks;

[0025] The new image block is used as the current image block, and the fused data is used as the current task analysis prompt, and the above steps are repeated until the output conditions are met, and the multi-task analysis result of the input image is output based on the fused data corresponding to the current task analysis prompt obtained last time.

[0026] Further, in some embodiments, the step of inputting the current image block and the current task analysis prompt into the current feature extraction subnetwork based on the self-attention mechanism to obtain the fused data corresponding to the current task analysis prompt and a plurality of new image blocks comprises:

[0027] The following operations are performed using the current feature extraction subnetwork based on the self-attention mechanism:

[0028] Extracting feature data of each current image block to obtain different types of feature information corresponding to each current image block;

[0029] Fusing the same type of features in each of the current image blocks to obtain a new image block;

[0030] All feature information in the new image block is fused with each current task analysis prompt respectively to obtain fused data corresponding to each current task analysis prompt.

[0031] Further, in some embodiments, the multi-task parsing model further includes a classifier, and the classifier corresponds one-to-one with the task parsing prompt;

[0032] Before outputting the input image multi-task parsing result based on the fusion data corresponding to the current task parsing prompt obtained last time, the method further includes:

[0033] The fused data corresponding to the current task parsing prompt obtained by the last feature extraction subnetwork based on the self-attention mechanism is input into the classifier to obtain the multi-task parsing result of the input image.

[0034] Furthermore, in some implementations, before acquiring the plurality of image blocks of the input image, the method further includes:

[0035] The input image is segmented to obtain image blocks.

[0036] The embodiment of this specification also proposes a homologous image multi-task parsing device, including:

[0037] A first data acquisition module, used to acquire an image block of an input image and a target task configured for the input image;

[0038] A first prompt generation module, configured to determine a task parsing prompt based on each target task of the input image;

[0039] The first result generation module is used to generate the multi-task parsing result of the input image based on the image block and the task parsing prompt.

[0040] The embodiment of this specification also provides a homologous image multi-task parsing device, including:

[0041] A second data acquisition module, used for acquiring a plurality of image blocks of an input image and a plurality of target tasks configured for the input image;

[0042] A second prompt generation module, configured to determine a task parsing prompt based on each target task of the input image;

[0043] The second result generation module is used to input the image block and the task parsing prompt into the multi-task parsing model to generate the multi-task parsing result of the input image.

[0044] The embodiments of the present specification also provide a computer program product, wherein the computer program product stores at least one instruction, and the at least one instruction is suitable for being loaded by a processor and executing the above method steps.

[0045] The embodiments of the present specification also provide a storage medium, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the above method.

[0046] An embodiment of the present specification also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0047] In the embodiments of the present specification, on the one hand, by determining task analysis prompts for each task, it is possible to flexibly adapt to various different task requirements. At the same time, in a multi-task scenario, each target task has its own unique task analysis prompt. Therefore, the embodiments of the present specification can better coordinate the relationship between the various tasks, reasonably allocate computing resources and processing time, and effectively avoid conflicts and interference between multiple tasks. On the other hand, since images in reality are extremely diverse, including different shooting angles, lighting conditions, resolutions, image content, etc., the embodiments of the present specification can better cope with this diversity based on image blocks and task analysis prompts, that is, for different images, the processing method and feature extraction strategy of the image blocks can be adaptively adjusted according to the task analysis prompts; and the processing method based on image blocks and task analysis prompts can achieve parallel processing and local optimization, that is, multiple image blocks can be processed at the same time, fully utilizing computing resources and improving processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of the first homologous image multi-task parsing method provided in the embodiment of this specification;

[0049] Figure 2 A flowchart of a second homologous image multi-task parsing method provided in an embodiment of this specification;

[0050] Figure 3 A schematic diagram of the flow of a third method for multi-task parsing of homologous images provided in an embodiment of this specification;

[0051] Figure 4 A flowchart of a fourth homologous image multi-task parsing method provided in an embodiment of this specification;

[0052] Figure 5 A flowchart of a fifth homologous image multi-task parsing method provided in an embodiment of this specification;

[0053] Figure 6 A flowchart of a sixth homologous image multi-task parsing method provided in an embodiment of this specification;

[0054] Figure 7 A flowchart of the seventh homologous image multi-task parsing method provided in the embodiments of this specification;

[0055] Figure 8 A flowchart of an eighth homologous image multi-task parsing method provided in an embodiment of this specification;

[0056] Fig. 9 A schematic diagram of the structure of a multi-task parsing model provided in an embodiment of this specification;

[0057] Fig.10 A schematic diagram of the structure of a homologous image multi-task parsing device provided in an embodiment of this specification;

[0058] Fig.11 A schematic diagram of the structure of another homologous image multi-task parsing device provided in an embodiment of this specification;

[0059] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0061] See also Figure 1 , which is a flowchart of a method for multi-task analysis of homologous images in an embodiment of this specification. In an embodiment of this specification, the method for multi-task analysis of homologous images is applied to a multi-task analysis device for homologous images or an electronic device equipped with the multi-task analysis device for homologous images. Figure 1 The process shown in the figure is described in detail, and the multi-task parsing method of homologous images may specifically include the following steps:

[0062] S102, obtaining an image block of an input image and a target task configured for the input image.

[0063] The input image can be any image captured by an optical device or other image acquisition device. For example, colorful natural scenery photography, simple geometric images, high-definition professional medical images, casual photos of daily life, large scene images with rich details, close-up images focusing only on tiny objects, etc., can all be used as input images to enter the subsequent processing flow.

[0064] The image blocks of the input image can be understood as sub-regions with independent features formed by dividing the input image. Therefore, each image block contains the pixel information of the input image in a specific local area. These pixels are adjacent to each other in spatial position and have certain correlation and similarity characteristics. In order to facilitate the separate processing and analysis of each image block, for example, in image recognition tasks, by extracting and analyzing the features of each image block, objects or patterns in the image can be more accurately identified, especially for multi-target recognition in complex scenes, which can help reduce computational complexity and improve recognition accuracy.

[0065] The target task represents a specific direction or requirement for processing and analyzing the input image. For example, if the target task is to identify the facial features of a person in the input image, then this task makes it clear that the core of subsequent processing is to revolve around the relevant information of the person's face. The setting of the target task is targeted and varies according to different application scenarios and user needs. It can be various specific operation intentions such as image classification, object detection, image editing, etc.

[0066] The target tasks assigned to the input images can be flexibly adjusted according to different needs, so the embodiments of this specification can be easily applied to multi-task scenarios in various fields. For example, in the security field, multiple target tasks such as face recognition and behavior monitoring can be realized at the same time; in the e-commerce field, multiple task analysis such as commodity recognition and image search optimization can be performed. Rapid switching and customization of target tasks according to actual needs greatly enhances the flexibility and adaptability of the multi-task analysis method of homologous images provided in this specification.

[0067] S104, determining a task analysis hint based on each target task of the input image.

[0068] Task analysis prompts are a set of directional information generated to guide the processing of image data for specific tasks, including guidance on key points, feature types, and processing methods that need to be paid attention to for the current task.

[0069] In some embodiments, the task analysis prompts may be derived from pre-set algorithmic rules, a pattern library based on historical data and experience summary, or intelligent prompts generated by a machine learning model based on the training of a large number of similar tasks. It can be seen that the task analysis prompts are guiding information generated based on the target task of the input image, which can clearly define the key points and processing methods that need to be paid attention to for each target task. Therefore, the task analysis prompts determined according to different target tasks are highly targeted and can guide the subsequent analysis process to focus on key information related to the target task. For example, in medical imaging diagnosis, when the target task is to detect tumors, the task analysis prompts will prompt the system to focus on areas in the image where tumor features may exist, such as image blocks with abnormal grayscale values, irregular shapes and other features, thereby improving the accuracy of tumor detection.

[0070] With the guidance of task analysis prompts, image blocks and information related to the target task can be quickly screened out during the analysis process, avoiding unnecessary processing of irrelevant information. This can reduce the waste of computing resources and improve the efficiency of analysis. For example, in an image search task, the task analysis prompts generated based on the search keywords can quickly locate image blocks that may contain relevant information, speeding up the search.

[0071] S106, generating a multi-task analysis result of the input image based on the image blocks and the task analysis prompts.

[0072] In one or more embodiments of the present specification, feature extraction can be performed on each image block, and the analysis results of the target task can be generated according to the feature type specified in the task analysis prompt. After the analysis results of each target task are completed, all analysis results are summarized and sorted to obtain the multi-task analysis results of the input image.

[0073] In one or more embodiments of the present specification, feature extraction can be performed on each image block, and the features extracted from each image block can be integrated to generate a parsing result for the target task based on the feature type specified in the task parsing prompt. After completing the parsing results for each target task, all parsing results are summarized and organized to obtain a multi-task parsing result for the input image.

[0074] The embodiments of this specification have wide application value in many fields. For example, in the medical field, the multi-task analysis results can help doctors diagnose and analyze diseases in medical images, such as identifying the lesion area in the image, measuring the size and morphological characteristics of the lesion, and evaluating the condition based on this information; in the field of intelligent transportation, through multi-task analysis of traffic monitoring images, vehicle identification, traffic flow statistics, and traffic incident detection can be realized, providing an important basis for traffic management and planning.

[0075] See also Figure 2 , is a flow chart of a method for multi-task analysis of homologous images provided in an embodiment of this specification. Figure 2 As shown, the method comprises the following steps:

[0076] S202, obtaining an image block of an input image and a target task configured for the input image.

[0077] For step S202, please refer to the detailed description of step S102 in another embodiment of this specification, which will not be repeated here.

[0078] S204, determining a task parsing hint based on each target task of the input image.

[0079] For step S204, please refer to the detailed description of step S104 in another embodiment of this specification, which will not be repeated here.

[0080] S206, extracting feature data of each current image block to obtain different types of feature information corresponding to each current image block;

[0081] Different types of feature information may include color features, texture features, shape features, spatial features, etc. In one or more embodiments of the present specification, corresponding feature extraction algorithms may be used for different types of feature information. For example, for color features, the statistical information of the image block on each color channel (such as mean, standard deviation, histogram, etc.) may be directly calculated; for texture features, the grayscale co-occurrence matrix algorithm or the local binary pattern algorithm may be applied. In one or more embodiments of the present specification, some machine learning and deep learning methods may also be used for feature extraction, such as using a convolutional neural network (CNN) to automatically learn representative features from an image block.

[0082] Each feature can describe the characteristics of an image block from a different perspective. Combining multiple features can provide a more comprehensive understanding of the content and structure of an image block. For example, color features alone may not be able to distinguish objects with similar shapes but different colors, while shape features alone may not be able to handle situations where the shapes are similar but the textures are different. By combining multiple features, the limitations of a single feature can be compensated and the accuracy of image analysis and recognition can be improved.

[0083] S208: Fusing the features of the same type contained in each of the current image blocks to obtain a new image block.

[0084] That is, based on the feature information corresponding to each current image block, the features of the same type are grouped together, for example, the color mean features of all image blocks are fused, the gray level co-occurrence matrix eigenvalues ​​are fused, and so on.

[0085] In one or more embodiments of the present specification, a simple weighted fusion method, a fusion method based on principal component analysis (PCA), a feature fusion method based on deep learning, etc. may be used for fusion.

[0086] In this way, the obtained new image blocks have more accurate description capabilities. For example, in image classification tasks, using the fused new image blocks for training and classification can improve the accuracy and robustness of classification. In image stitching and restoration tasks, the stitched or restored areas can be made more consistent with the surrounding image areas in terms of color, texture, and shape, thereby improving the overall quality of the image. Therefore, the new image blocks can provide smoother and more natural transition effects. In target detection and recognition tasks, the new image blocks can provide richer feature information, which helps to more accurately locate and identify target objects. For example, by fusing color and shape features, the target object can be better distinguished from the background, thereby improving the accuracy of target detection.

[0087] S210, fusing all feature information in the new image block with each current task analysis prompt respectively to obtain fused data corresponding to each current task analysis prompt.

[0088] In one or more embodiments of the present specification, before all features in the new image block are fused with each current task analysis prompt, in order to ensure the accuracy and effectiveness of the fusion process, all features in the new image block can also be standardized. For example, for the color mean feature, its value range may be between 0 and 255, while the grayscale co-occurrence matrix eigenvalue of the texture feature may fluctuate within a larger value range. Through standardization methods, such as mapping the eigenvalue to between 0 and 1 or performing standardization with a mean of 0 and a variance of 1, different features are made comparable in value.

[0089] It is understandable that when there are multiple target tasks, there are also multiple task analysis prompts corresponding to the target tasks. In some embodiments, the number of target tasks and task analysis prompts can correspond one to one. In other embodiments, one target task can correspond to multiple task analysis prompts.

[0090] On this basis, the fused data corresponding to each current task analysis prompt is the fused data corresponding to the task analysis prompt corresponding to each target task. In other words, one task analysis prompt corresponds to one fused data.

[0091] S212, taking the new image block as the current image block and the fused data as the current task analysis prompt, repeating the above steps until the output conditions are met, and outputting the input image multi-task analysis result based on the fused data corresponding to the current task analysis prompt obtained last time.

[0092] It is understandable that each repetition of the above steps is a process of continuous refinement and optimization. By using new image blocks and fused data as the basis for a new round of processing, the understanding of image features and the accuracy of task parsing can be gradually corrected and improved. For example, in the target detection task, the first round of processing may preliminarily determine some possible target areas, but through subsequent repeated steps, the boundaries and detailed features of the target can be further accurately located. As the number of repetitions increases, it can continuously adapt to the complexity and diversity of the image, better extract and fuse the features related to the task, and thus improve the quality and accuracy of the final multi-task parsing results.

[0093] Here, the output condition may be set to reach a preset number of repetitions, or, in some embodiments, if the repetition step is performed by some network models, the output condition may be set to reach the output layer of the network model.

[0094] See also Figure 3 , is a flow chart of a method for multi-task analysis of homologous images provided in an embodiment of this specification. Figure 3 As shown, the method comprises the following steps:

[0095] S302, obtaining an image block of an input image and a target task configured for the input image.

[0096] For step S302, please refer to the detailed description of step S102 in another embodiment of this specification, which will not be repeated here.

[0097] S304: Determine a task analysis prompt based on each target task of the input image.

[0098] For step S304, please refer to the detailed description of step S104 in another embodiment of this specification, which will not be repeated here.

[0099] S306: extract feature data of each current image block to obtain different types of feature information corresponding to each current image block.

[0100] For step S306, please refer to the detailed description of step S206 in another embodiment of this specification, which will not be repeated here.

[0101] S308: Fusing the features of the same type contained in each of the current image blocks to obtain a new image block.

[0102] For step S302, please refer to the detailed description of step S208 in another embodiment of this specification, which will not be repeated here.

[0103] S310, fusing all the features in the new image block with the current task analysis prompts respectively to obtain fused data corresponding to the current task analysis prompts.

[0104] For step S310, please refer to the detailed description of step S210 in another embodiment of this specification, which will not be repeated here.

[0105] S312, taking the new image block as the current image block, and taking the fused data as the current task parsing prompt, and repeating the above steps until the output condition is met.

[0106] For step S312, please refer to the detailed description of step S212 in another embodiment of this specification, which will not be repeated here.

[0107] S324, classifying and processing the fused data corresponding to the current task analysis prompt obtained last time to obtain a multi-task analysis result of the input image.

[0108] The last obtained current task analysis hint is the current task analysis hint obtained when the output conditions are met. After the previous series of steps, fully integrated and optimized feature data has been obtained, but since the target task may not be unique, it is necessary to perform final classification and summary according to the target task to obtain specific multi-task analysis results to meet the needs of different application scenarios.

[0109] In some embodiments, classification can be performed using machine learning algorithms, such as decision trees, support vector machines, etc., and classification can also be performed using deep learning algorithms, such as convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory networks (LSTM), etc., or ensemble learning classification methods, such as random forest classification.

[0110] See also Figure 4 , is a flow chart of a method for multi-task analysis of homologous images provided in an embodiment of this specification. Figure 4 As shown, the method comprises the following steps:

[0111] S402, segmenting the input image to obtain image blocks.

[0112] In some embodiments, in order to better analyze and process the input image, it is necessary to perform segmentation processing on it. There are various ways of segmentation processing, which can be flexibly selected according to actual needs and conditions.

[0113] In a feasible implementation, a computer system can be used to perform automated segmentation operations on the input image. When the system is used for segmentation, a series of advanced image segmentation algorithms can be used to achieve accurate segmentation. For example, by an algorithm that identifies the boundaries of different regions in an image, the algorithm can accurately locate the boundary lines between various regions based on the differences in the color, brightness, texture and other features of the pixels in the image, and then segment the input image into multiple independent regions with specific features along these boundary lines. These regions may represent different objects in the image, different background elements, or image blocks with similar features. For another example, a segmentation algorithm based on cluster analysis can group pixels in an image according to similarity to form different cluster regions, each of which corresponds to a specific part of the input image.

[0114] In a feasible implementation, the input image may also be segmented manually, for example, relying on rich experience and in-depth understanding of the image content, professional image editing tools may be used to manually draw boundary lines or select specific areas to perform fine segmentation on the input image.

[0115] It can be seen that no matter what segmentation method is used, each image block formed by segmentation contains relatively local and feature-concentrated information, so feature extraction and processing can be performed more carefully based on image blocks. For example, in image recognition tasks, for an input image containing multiple objects, after dividing it into image blocks, the edges, textures and other features of each object can be more accurately identified, avoiding the mutual interference that may occur when the entire image is processed together, thereby greatly improving the accuracy of single object recognition.

[0116] S404, obtaining an image block of an input image and a target task configured for the input image.

[0117] For step S404, please refer to the detailed description of step S102 in another embodiment of this specification, which will not be repeated here.

[0118] S406: Determine a task analysis prompt based on each target task of the input image.

[0119] For step S406, please refer to the detailed description of step S104 in another embodiment of this specification, which will not be repeated here.

[0120] S408: Generate a multi-task analysis result of the input image based on the image block and the task analysis prompt.

[0121] For step S408, please refer to the detailed description of step S106 in another embodiment of this specification, which will not be repeated here.

[0122] See also Figure 5 , is a flow chart of a method for multi-task analysis of homologous images provided in an embodiment of this specification. Figure 5 As shown, the method comprises the following steps:

[0123] S502, obtaining an image block of an input image and a target task configured for the input image.

[0124] The input image can be any image captured by an optical device or other image acquisition device. For example, colorful natural scenery photography, simple geometric images, high-definition professional medical images, casual photos of daily life, large scene images with rich details, close-up images focusing only on tiny objects, etc., can all be used as input images to enter the subsequent processing flow.

[0125] The image blocks of the input image can be understood as sub-regions with independent features formed by dividing the input image. Therefore, each image block contains the pixel information of the input image in a specific local area. These pixels are adjacent to each other in spatial position and have certain correlation and similarity characteristics. In order to facilitate the separate processing and analysis of each image block, for example, in image recognition tasks, by extracting and analyzing the features of each image block, objects or patterns in the image can be more accurately identified, especially for multi-target recognition in complex scenes, which can help reduce computational complexity and improve recognition accuracy.

[0126] The target task represents a specific direction or requirement for processing and analyzing the input image. For example, if the target task is to identify the facial features of a person in the input image, then this task makes it clear that the core of subsequent processing is to revolve around the relevant information of the person's face. The setting of the target task is targeted and varies according to different application scenarios and user needs. It can be various specific operation intentions such as image classification, object detection, image editing, etc.

[0127] The target tasks assigned to the input images can be flexibly adjusted according to different needs, so the embodiments of this specification can be easily applied to multi-task scenarios in various fields. For example, in the security field, multiple target tasks such as face recognition and behavior monitoring can be realized at the same time; in the e-commerce field, multiple task analysis such as commodity recognition and image search optimization can be performed. Rapid switching and customization of target tasks according to actual needs greatly enhances the flexibility and adaptability of the multi-task analysis method of homologous images provided in this specification.

[0128] S504: Determine a task analysis prompt based on each target task of the input image.

[0129] Task analysis prompts are a set of directional information generated to guide the processing of image data for specific tasks, including guidance on key points, feature types, and processing methods that need to be paid attention to for the current task.

[0130] In some embodiments, the task analysis prompts may be derived from pre-set algorithmic rules, a pattern library based on historical data and experience summary, or intelligent prompts generated by a machine learning model based on the training of a large number of similar tasks. It can be seen that the task analysis prompts are guiding information generated based on the target task of the input image, which can clearly define the key points and processing methods that need to be paid attention to for each target task. Therefore, the task analysis prompts determined according to different target tasks are highly targeted and can guide the subsequent analysis process to focus on key information related to the target task. For example, in medical imaging diagnosis, when the target task is to detect tumors, the task analysis prompts will prompt the system to focus on areas in the image where tumor features may exist, such as image blocks with abnormal grayscale values, irregular shapes and other features, thereby improving the accuracy of tumor detection.

[0131] With the guidance of task analysis prompts, image blocks and information related to the target task can be quickly screened out during the analysis process, avoiding unnecessary processing of irrelevant information. This can reduce the waste of computing resources and improve the efficiency of analysis. For example, in an image search task, the task analysis prompts generated based on the search keywords can quickly locate image blocks that may contain relevant information, speeding up the search.

[0132] S506: Input the image block and the task analysis prompt into a multi-task analysis model to generate a multi-task analysis result of the input image.

[0133] In some embodiments, the multi-task parsing model can be a multi-task model based on a deep learning convolutional neural network architecture, using a plurality of parallel convolutional layer branch structures, each branch is dedicated to processing a specific type of feature or task-related information. For example, one branch is used to extract color features of an image, another branch is used to extract texture features, and another branch is used to extract shape features, etc. These branches extract features from image blocks at different levels, and then fuse the extracted features.

[0134] In other embodiments, the multi-task parsing model can also be a sequence processing multi-task model based on a recurrent neural network, which is suitable for processing image data with time series or sequence features, such as video image analysis. In this case, the model needs to consider the temporal correlation and dynamic changes between image frames. Multiple branches can be connected to the output layer of the RNN unit to form a model structure. First, feature extraction is performed on each image block to obtain a feature vector sequence, and then these feature vectors are input into the RNN unit in sequence. The RNN unit can remember historical information and update the internal state according to the current input and the previous state. At each time step, the output of the RNN unit can be used as an intermediate result for different task parsing. Multiple branches connected to the output layer of the RNN, each branch corresponds to a task. For example, one branch is used to predict the motion trajectory of an object in a video (target tracking task), and another branch is used to identify key events in a video (event detection task), etc.

[0135] See also Figure 6 , is a flow chart of a method for multi-task parsing of homologous images provided in an embodiment of this specification. The multi-task parsing model includes a plurality of feature extraction sub-networks based on a self-attention mechanism that are sequentially superimposed; Figure 6 As shown, the method comprises the following steps:

[0136] S602, obtaining an image block of an input image and a target task configured for the input image.

[0137] For step S602, please refer to the detailed description of step S502 in the first embodiment of this specification, which will not be repeated here.

[0138] S604: Determine a task analysis prompt based on each target task of the input image.

[0139] For step S604, please refer to the detailed description of step S504 in the first embodiment of this specification, which will not be repeated here.

[0140] S606, input the current image block and the current task analysis prompt into the current feature extraction subnetwork based on the self-attention mechanism to obtain the fused data corresponding to the current task analysis prompt and multiple new image blocks.

[0141] The self-attention mechanism can automatically calculate the importance weight of each part based on the relationship between the image block and the task parsing hint.

[0142] The feature extraction sub-network can be designed as a neural network structure containing multiple levels, such as convolutional layers, fully connected layers, etc., while integrating a self-attention mechanism module.

[0143] By re-weighting and adjusting the image patch features through the self-attention mechanism, the image patch can be re-divided and new image patches can be generated according to the importance and relevance of the features. For example, if the self-attention mechanism finds that a certain area in the image patch has unique features and is highly relevant to the task parsing hint, then this area may be extracted separately as a new image patch for further processing.

[0144] At the same time, the self-attention mechanism can also merge or split multiple image blocks according to the similarity and correlation between image blocks to generate new image block combinations.

[0145] Each new image patch may contain specific features or key information related to the task, which helps the model to understand the image content more deeply and complete the task. For example, in the image segmentation task, the new image patch can more accurately represent the different regions and object boundaries in the image. In the multi-task processing scenario, different new image patches can be used for different tasks respectively to improve the efficiency and accuracy of task processing. The new image patch can also serve as the basis for the iterative processing of the feature extraction subnetwork based on the self-attention mechanism. By continuously generating new image patches and updating feature representations, the model's understanding of the image and task processing capabilities can be gradually optimized.

[0146] S608, taking the new image block as the current image block and the fused data as the current task analysis prompt, repeating the above steps until the output conditions are met, and outputting the input image multi-task analysis result based on the fused data corresponding to the current task analysis prompt obtained last time.

[0147] Each repetition is a further refinement and optimization of the image patch and task parsing prompt. As the repetition proceeds, the self-attention mechanism can continuously adjust the focus on image features, and generate more accurate new image patches and updated versions of task parsing prompts based on the fused data. For example, in the first round of processing, some features of the image may not be accurately focused on, but through multiple iterations, the model can gradually focus on the features most relevant to the task, so that the new image patch contains more accurate information.

[0148] In addition, on the one hand, different input images may contain various complex scenes, objects, and feature combinations. Repeating steps can make the model gradually adapt to these changes, and have a better understanding and extraction ability for details and potential patterns in complex images. For example, in an image containing multiple overlapping objects, the initial processing may be difficult to accurately distinguish the features of each object, but through multiple iterations, the model can gradually optimize the division and feature extraction of image blocks corresponding to each object based on the feedback of fused data. On the other hand, in practical applications, image data may be affected by factors such as noise, illumination changes, and perspective changes. By repeating steps, the multi-task parsing model can learn feature patterns and task parsing methods in different situations, thereby improving adaptability and robustness to various interference factors.

[0149] See also Figure 7 , is a flow chart of a method for multi-task parsing of homologous images provided in an embodiment of this specification. The multi-task parsing model includes a plurality of feature extraction sub-networks based on a self-attention mechanism that are sequentially superimposed; Figure 7 As shown, the method comprises the following steps:

[0150] S702, obtaining an image block of an input image and a target task configured for the input image.

[0151] For step S702, please refer to the detailed description of step S502 in the first embodiment of this specification, which will not be repeated here.

[0152] S704: Determine a task analysis prompt based on each target task of the input image.

[0153] For step S704, please refer to the detailed description of step S504 in the first embodiment of this specification, which will not be repeated here.

[0154] S706: Extract feature data of each current image block to obtain different types of feature information corresponding to each current image block.

[0155] Different types of feature information may include color features, texture features, shape features, spatial features, etc. In one or more embodiments of the present specification, corresponding feature extraction algorithms may be used for different types of feature information. For example, for color features, the statistical information of the image block on each color channel (such as mean, standard deviation, histogram, etc.) may be directly calculated; for texture features, the gray level co-occurrence matrix algorithm or the local binary pattern algorithm may be applied. In one or more embodiments of the present specification, some machine learning and deep learning methods may also be used for feature extraction. For example, a convolutional neural network may be used to automatically learn representative features from an image block.

[0156] Each feature can describe the characteristics of an image block from a different perspective. Combining multiple features can provide a more comprehensive understanding of the content and structure of an image block. For example, color features alone may not be able to distinguish objects with similar shapes but different colors, while shape features alone may not be able to handle situations where the shapes are similar but the textures are different. By combining multiple features, the limitations of a single feature can be compensated and the accuracy of image analysis and recognition can be improved.

[0157] S708: Fusing the features of the same type contained in each of the current image blocks to obtain a plurality of new image blocks.

[0158] That is, based on the feature information corresponding to each current image block, the features of the same type are grouped together, for example, the color mean features of all image blocks are fused, the gray level co-occurrence matrix eigenvalues ​​are fused, and so on.

[0159] In one or more embodiments of the present specification, a simple weighted fusion method, a fusion method based on principal component analysis (PCA), a feature fusion method based on deep learning, etc. may be used for fusion.

[0160] In this way, the obtained new image blocks have more accurate description capabilities. For example, in image classification tasks, using the fused new image blocks for training and classification can improve the accuracy and robustness of classification. In image stitching and restoration tasks, the stitched or restored areas can be made more consistent with the surrounding image areas in terms of color, texture, and shape, thereby improving the overall quality of the image. Therefore, the new image blocks can provide smoother and more natural transition effects. In target detection and recognition tasks, the new image blocks can provide richer feature information, which helps to more accurately locate and identify target objects. For example, by fusing color and shape features, the target object can be better distinguished from the background, thereby improving the accuracy of target detection.

[0161] S710: Fusing all feature information in the new image block with each current task analysis prompt respectively to obtain fused data corresponding to the current task analysis prompt.

[0162] In one or more embodiments of the present specification, before all features in the new image block are fused with each current task analysis prompt, in order to ensure the accuracy and effectiveness of the fusion process, all features in the new image block can also be standardized. For example, for the color mean feature, its value range may be between 0 and 255, while the grayscale co-occurrence matrix eigenvalue of the texture feature may fluctuate within a larger value range. Through standardization methods, such as mapping the eigenvalue to between 0 and 1 or performing standardization with a mean of 0 and a variance of 1, different features are made comparable in value.

[0163] It is understandable that when there are multiple target tasks, there are also multiple task analysis prompts corresponding to the target tasks. In some embodiments, the number of target tasks and task analysis prompts can correspond one to one. In other embodiments, one target task can correspond to multiple task analysis prompts.

[0164] On this basis, the fused data corresponding to each current task analysis prompt is the fused data corresponding to the task analysis prompt corresponding to each target task. In other words, one task analysis prompt corresponds to one fused data.

[0165] In one or more implementations of the present specification, S706 to S710 may be implemented using a feature extraction subnetwork based on a self-attention mechanism.

[0166] The self-attention mechanism can automatically calculate the importance weight of each part based on the relationship between the image block and the task parsing hint.

[0167] The feature extraction sub-network can be designed as a neural network structure containing multiple levels, such as convolutional layers, fully connected layers, etc., while integrating a self-attention mechanism module.

[0168] In one or more embodiments, the feature extraction subnetwork based on the self-attention mechanism can be set to a convolutional neural network, a recurrent neural network, or a Transformer Layer that includes a self-attention mechanism. In the embodiment of this specification, the feature extraction subnetwork based on the self-attention mechanism is set to a Transformer Layer. It can be imagined that multiple sequentially superimposed feature extraction subnetworks based on the self-attention mechanism are generated based on the CLIP base model, and a detailed description is given below based on this.

[0169] By re-weighting and adjusting the image patch features through the self-attention mechanism, the image patch can be re-divided and new image patches can be generated according to the importance and relevance of the features. For example, if the self-attention mechanism finds that a certain area in the image patch has unique features and is highly relevant to the task parsing hint, then this area may be extracted separately as a new image patch for further processing.

[0170] At the same time, the self-attention mechanism can also merge or split multiple image blocks according to the similarity and correlation between image blocks to generate new image block combinations.

[0171] Each new image patch may contain specific features or key information related to the task, which helps the model to understand the image content more deeply and complete the task. For example, in the image segmentation task, the new image patch can more accurately represent the different regions and object boundaries in the image. In the multi-task processing scenario, different new image patches can be used for different tasks respectively to improve the efficiency and accuracy of task processing. The new image patch can also serve as the basis for the iterative processing of the feature extraction subnetwork based on the self-attention mechanism. By continuously generating new image patches and updating feature representations, the model's understanding of the image and task processing capabilities can be gradually optimized.

[0172] S712, taking the new image block as the current image block and the fused data as the current task analysis prompt, repeating the above steps until the output conditions are met, and outputting the input image multi-task analysis result based on the fused data corresponding to the current task analysis prompt obtained last time.

[0173] It is understandable that each repetition of the above steps is a process of continuous refinement and optimization. By using new image blocks and fused data as the basis for a new round of processing, the understanding of image features and the accuracy of task parsing can be gradually corrected and improved.

[0174] In one or more embodiments of the present specification, each feature extraction sub-network based on the self-attention mechanism can implement S706 to S710. That is, the content output by each feature extraction sub-network based on the self-attention mechanism includes a new image block and fusion data, and this output content is used as the input of the next feature extraction sub-network based on the self-attention mechanism, and is propagated in sequence, thereby saving computing resources.

[0175] See also Figure 8 , is a flow chart of a method for multi-task parsing of homologous images provided in an embodiment of this specification. The multi-task parsing model includes a plurality of sequentially superimposed feature extraction sub-networks and classifiers based on a self-attention mechanism; Figure 8 As shown, the method comprises the following steps:

[0176] S802, obtaining an image block of an input image and a target task configured for the input image.

[0177] For step S802, please refer to the detailed description of step S502 in another embodiment of this specification, which will not be repeated here.

[0178] S804: Determine a task analysis prompt based on each target task of the input image.

[0179] For step S804, please refer to the detailed description of step S504 in another embodiment of this specification, which will not be repeated here.

[0180] S806, extracting feature data of each current image block to obtain different types of feature information corresponding to each current image block;

[0181] For step S806, please refer to the detailed description of step S706 in another embodiment of this specification, which will not be repeated here.

[0182] S808: Fusing the same type of features in each of the current image blocks to obtain a new image block.

[0183] For step S808, please refer to the detailed description of step S708 in another embodiment of this specification, which will not be repeated here.

[0184] S810, fusing all feature information in the new image block with each current task analysis prompt respectively to obtain fused data corresponding to each current task analysis prompt.

[0185] For step S810, please refer to the detailed description of step S710 in another embodiment of this specification, which will not be repeated here.

[0186] S812, taking the new image block as the current image block, and taking the fused data as the current task parsing prompt, and repeating the above steps until the output condition is met.

[0187] For step S812, please refer to the detailed description of step S712 in another embodiment of this specification, which will not be repeated here.

[0188] S814, inputting the fused data corresponding to the current task parsing prompt obtained by the last feature extraction subnetwork based on the self-attention mechanism into the classifier to obtain the multi-task parsing result of the input image.

[0189] The last obtained current task analysis hint is the current task analysis hint obtained when the output conditions are met. After the previous series of steps, fully integrated and optimized feature data has been obtained, but since the target task may not be unique, it is necessary to perform final classification and summary according to the target task to obtain specific multi-task analysis results to meet the needs of different application scenarios.

[0190] refer to Fig. 9 As shown in Figure 1, a schematic diagram of the structure of a multi-task parsing model is given. Fig. 9 In the multi-task parsing model 9, at least a plurality of self-attention mechanism-based feature extraction subnetworks 901 and classifiers 902 are stacked in sequence. The image block 903 and the task parsing prompt 904 are input into a feature extraction subnetwork 901 based on the self-attention mechanism. The feature extraction subnetwork 901 based on the self-attention mechanism performs feature extraction and fusion processing to obtain a new image block 905 and fusion data 906. The new image block 905 and fusion data 906 are input into the next feature extraction subnetwork 901 based on the self-attention mechanism, until the last feature extraction subnetwork 901 based on the self-attention mechanism outputs the fusion data to the classifier. The multi-task parsing result of the input image is obtained through classification processing by the classifier. In this process, a fusion data is generated for each task parsing prompt, and the fusion data contains all the features of the new image block and the task parsing prompt. Of course, Fig. 9 The structure of the multi-task parsing model 900 is merely an illustrative example, and the present embodiment is not limited thereto.

[0191] Based on this, the new image block is completely determined by the output of the previous feature extraction subnetwork based on the self-attention mechanism, and is not affected by any task parsing prompts; in contrast, the task parsing prompts or fused data depend on the output of the previous feature extraction subnetwork based on the self-attention mechanism and the upper-layer output of the task parsing prompts or fused data themselves. These outputs are synthesized through a clever aggregation mechanism to form an intermediate state representation of the task parsing prompts or fused data. This design ensures that the representation of the intermediate state of the original image and its final representation remain unchanged with the participation of the task parsing prompts or fused data, so that the representation can be shared in various downstream tasks. This is particularly important when performing multiple tasks that require prompt learning. This means that the backbone network including multiple feature extraction subnetworks based on the self-attention mechanism superimposed in sequence only needs to perform one forward propagation, which greatly saves valuable computing resources. At the same time, since the fused data simultaneously considers the information from the intermediate layer of the feature extraction subnetwork based on the self-attention mechanism and the output of the fused data itself, it can not only bring new feature gains to the final representation of the image, but also maintain the fitting ability of the prompt learning in the downstream tasks. This complementarity enables the embodiments of this specification to improve efficiency while also enhancing the adaptability and expressiveness of diversified tasks.

[0192] In addition, the classifier input of the embodiment of this specification comes from the final representation of the fused data. Fig. 9 As shown, the embodiment of this specification incorporates the refined representation of data as the decisive feature input. This is based on a key consideration: when designing task parsing prompts, the fitting effects from elementary to advanced levels are fully considered, and this cumulative impact greatly enhances the expressiveness of the representation. In addition, since the representation of the task parsing prompt is also closely combined with the original output of each feature extraction sub-network based on the self-attention mechanism, this brings strong robustness to the final representation. The combination of this robustness and expressiveness makes the embodiment of this specification particularly suitable for downstream tasks with scarce labeled samples, where the model must make accurate predictions with limited data support. Therefore, through such a design, not only can the efficient use of the feature extraction sub-network based on the self-attention mechanism be achieved, but also through the layer-by-layer refinement of the task parsing prompt representation, a richer and more robust feature foundation is provided for complex downstream tasks. This method, which combines robustness and adaptability, provides a powerful solution framework for facing changing task requirements and data conditions.

[0193] The task analysis prompts between tasks are completely decoupled: Fig. 9One of the highlights of the demonstration is that the task parsing prompts between tasks are designed to be completely decoupled and independent of each other. There are two core considerations behind this design. First, given that the training difficulty of the CLIP base model is relatively high and the deployment cost is expensive, the goal of the embodiment of this specification is to achieve a "one-time deployment, multiple uses" mode. That is, after the base model is deployed, multiple tasks can be "plug and play" without any changes to it. This approach ensures that even when new tasks are added or some services are offline, the existing online tasks can remain unchanged and proceed seamlessly. Another consideration is that although there is usually a certain positive interaction between tasks in the multi-task model, since the original CLIP base model has embedded highly robust feature representations-these features show strong adaptability to diverse tasks, there is no need to set interactions between different task parsing prompts. The embodiment of this specification makes full use of this advantage of the CLIP base model itself, allowing each task to be independently optimized through its own task parsing prompts without interfering with each other. Through this careful design, not only can the performance of downstream tasks be improved, but also computing resources can be effectively saved. The task analysis prompts for each task operate independently, which means that individual tasks can be flexibly adjusted or expanded without affecting the overall system. This architectural design achieves efficient collaboration while retaining the flexibility and scalability of the system, providing strong support for the ever-changing business needs.

[0194] See also Fig.10 , is a schematic diagram of the structure of a homologous image multi-task parsing device provided in an embodiment of this specification. Fig.10 As shown, the homologous image multi-task parsing device 1 can be implemented as all or part of an electronic device through software, hardware or a combination of both. According to some embodiments, the homologous image multi-task parsing device 1 includes a first data acquisition module 11, a first prompt generation module 12, and a first result generation module 13, specifically including:

[0195] A first data acquisition module 11, used to acquire an image block of an input image and a target task configured for the input image;

[0196] A first prompt generating module 12, configured to determine a task parsing prompt based on each target task of the input image;

[0197] The first result generating module 13 is used to generate the multi-task parsing result of the input image based on the image block and the task parsing prompt.

[0198] Optionally, the first result generating module 13 is specifically configured to:

[0199] Extracting feature data of each current image block to obtain feature information corresponding to each current image block, wherein the feature information includes multiple features;

[0200] Fusing the same type of features in each of the image blocks to obtain a new image block;

[0201] Fusing all features in the new image block with each current task analysis prompt respectively to obtain fused data corresponding to each current task analysis prompt;

[0202] The new image block is used as the current image block, and the fused data is used as the current task parsing prompt, and the above steps are repeated until the output condition is met, and the input image multi-task parsing result is output.

[0203] Optionally, the first result generating module 13 is further used for:

[0204] The fused data corresponding to the current task analysis prompt obtained last time is classified and processed to obtain the input image multi-task analysis result.

[0205] Optionally, the homologous image multi-task parsing device 1 further includes a first image segmentation module 14, which is used to perform segmentation processing on the input image to obtain image blocks.

[0206] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0207] See also Fig.11 , is a schematic diagram of the structure of a homologous image multi-task parsing device 2 provided in an embodiment of this specification. Fig.11 As shown, the homologous image multi-task parsing device 2 can be implemented as all or part of the electronic device through software, hardware or a combination of both. According to some embodiments, the homologous image multi-task parsing device 2 includes a second data acquisition module 21, a second prompt generation module 22, and a second result generation module 23, specifically including:

[0208] A second data acquisition module 21, used for acquiring a plurality of image blocks of an input image and a plurality of target tasks configured for the input image;

[0209] A second prompt generating module 22, configured to determine a task parsing prompt based on each target task of the input image;

[0210] The second result generating module 23 is used to input the image block and the task parsing hint into the multi-task parsing model to generate the multi-task parsing result of the input image.

[0211] Optionally, the multi-task parsing model includes a plurality of sequentially superimposed feature extraction sub-networks based on the attention mechanism, and the second result generating module 23 is specifically used for:

[0212] Inputting the current image block and the current task analysis prompt into the current feature extraction subnetwork based on the self-attention mechanism to obtain fused data corresponding to the current task analysis prompt and multiple new image blocks;

[0213] The new image block is used as the current image block, and the fused data is used as the current task parsing prompt, and the above steps are repeated until the output condition is met, and the input image multi-task parsing result is output.

[0214] Optionally, the second result generating module 23 is further specifically configured to:

[0215] The following operations are performed using the current feature extraction subnetwork based on the self-attention mechanism:

[0216] Extracting feature data of each current image block to obtain feature information corresponding to each current image block, wherein the feature information includes multiple features;

[0217] Fusing the same type of features in each of the image blocks to obtain a new image block;

[0218] All features in the new image block are fused with each current task analysis prompt respectively to obtain fused data corresponding to each current task analysis prompt.

[0219] Optionally, the multi-task parsing model also includes a classifier, which corresponds one-to-one to the task parsing prompt; the second result generation module 23 is also specifically used to: input the fusion data corresponding to the current task parsing prompt obtained by the last feature extraction subnetwork based on the self-attention mechanism into the classifier to obtain the multi-task parsing result of the input image.

[0220] Optionally, the homologous image multi-task parsing device 2 further includes a second image segmentation module 24, which is used to perform segmentation processing on the input image to obtain image blocks.

[0221] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0222] The present specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figures 1 to 8 The method of the embodiment shown in the figure can be specifically executed by referring to Figures 1 to 8 The specific description of the illustrated embodiment will not be repeated here.

[0223] The present specification also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figures 1 to 8 The method of the embodiment shown in the figure can be specifically executed by referring to Figures 1 to 8 The specific description of the illustrated embodiment will not be repeated here.

[0224] The embodiments of this specification also provide Fig.12 The structural diagram of the electronic device shown in FIG. Fig.12 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned voice activity detection method.

[0225] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0226] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0227] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0228] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0229] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0230] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0231] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0232] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0233] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0234] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0235] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0236] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0237] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0238] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0239] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0240] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0241] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for multi-task analysis of homologous images, the method comprising: Obtaining an image patch of an input image and a target task assigned to the input image; Determining a task parsing hint for each target task based on the input image; Based on the image block and the task parsing hint, a multi-task parsing result of the input image is generated.

2. The method according to claim 1, wherein generating the input image multi-task parsing result based on the image block and the task parsing hint comprises: Extracting feature data of each current image block to obtain different types of feature information corresponding to each current image block; The features of the same type contained in each of the current image blocks are merged to obtain a new image block; Fusing all feature information in the new image block with each current task analysis prompt respectively to obtain fused data corresponding to each current task analysis prompt; The new image block is used as the current image block, and the fused data is used as the current task analysis prompt, and the above steps are repeated until the output conditions are met, and the multi-task analysis result of the input image is output based on the fused data corresponding to the current task analysis prompt obtained last time.

3. The method according to claim 2, before outputting the input image multi-task parsing result based on the fusion data corresponding to the current task parsing prompt obtained last time, the method further comprises: The fused data corresponding to the current task analysis prompt obtained last time is classified and processed to obtain the input image multi-task analysis result.

4. The method according to any one of claims 1 to 3, before obtaining the image block of the input image, the method further comprises: The input image is segmented to obtain image blocks.

5. A method for multi-task analysis of homologous images, the method comprising: Obtaining an image patch of an input image and a target task assigned to the input image; Determining a task parsing hint for each target task based on the input image; The image block and the task parsing prompt are input into a multi-task parsing model to generate a multi-task parsing result of the input image.

6. The method according to claim 5, wherein the multi-task parsing model comprises a plurality of sequentially superimposed feature extraction sub-networks based on a self-attention mechanism; The step of inputting the image block and the task parsing hint into the multi-task parsing model to generate the input image multi-task parsing result comprises: Inputting the current image block and the current task analysis prompt into the current feature extraction subnetwork based on the self-attention mechanism to obtain fused data corresponding to the current task analysis prompt and multiple new image blocks; The new image block is used as the current image block, and the fused data is used as the current task analysis prompt, and the above steps are repeated until the output conditions are met, and the multi-task analysis result of the input image is output based on the fused data corresponding to the current task analysis prompt obtained last time.

7. According to the method of claim 6, the step of inputting the current image block and the current task analysis prompt into the current feature extraction subnetwork based on the self-attention mechanism to obtain the fused data corresponding to the current task analysis prompt and a plurality of new image blocks comprises: The following operations are performed using the current feature extraction subnetwork based on the self-attention mechanism: Extracting feature data of each current image block to obtain different types of feature information corresponding to each current image block; The features of the same type contained in each of the current image blocks are merged to obtain a new image block; All features in the new image block are fused with each current task analysis prompt respectively to obtain fused data corresponding to each current task analysis prompt.

8. The method according to claim 6, wherein the multi-task parsing model further comprises a classifier, and the classifier corresponds one-to-one with the task parsing prompt; Before outputting the input image multi-task parsing result based on the fusion data corresponding to the current task parsing prompt obtained last time, the method further includes: The fused data corresponding to the current task parsing prompt obtained by the last feature extraction subnetwork based on the self-attention mechanism is input into the classifier to obtain the multi-task parsing result of the input image.

9. A homologous image multi-task parsing device, comprising: A first data acquisition module, used to acquire an image block of an input image and a target task configured for the input image; A first prompt generation module, configured to determine a task parsing prompt based on each target task of the input image; The first result generation module is used to generate the multi-task parsing result of the input image based on the image block and the task parsing prompt.

10. A homologous image multi-task analysis device, the device comprising: A second data acquisition module, used for acquiring a plurality of image blocks of an input image and a plurality of target tasks configured for the input image; A second prompt generation module, configured to determine a task parsing prompt based on each target task of the input image; The second result generation module is used to input the image block and the task parsing prompt into the multi-task parsing model to generate the multi-task parsing result of the input image.

11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 4 or 5 to 8 are implemented.

12. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 4 or 5 to 8.

13. A computer program product having at least one instruction stored thereon, characterized in that: When the at least one instruction is executed by the processor, the steps of the method described in any one of claims 1 to 4 or 5 to 8 are implemented.

Images