Processing method and device based on computer vision large model, equipment and medium

Abstract

The embodiment of the invention provides a processing method and device based on a computer vision large model, equipment and a medium. The method comprises the following steps: constructing a knowledge graph related to an identification task; obtaining sample image data, fusing the sample image data and the knowledge graph, and training a preset computer vision large model by adopting the fused data; and obtaining target image data, calling the trained computer vision large model, and identifying the target image data to obtain an identification result. According to the embodiment of the invention, the method achieves the fusion of the computer vision large model and the knowledge graph for image recognition, can meet the demands of processing a complex scene or multi-modal information, and improves the capability of image recognition.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a processing method, device, equipment and medium based on a computer vision large model. Background Art

[0002] With the development of science and technology, image recognition tasks are increasingly dependent on models, such as the large computer vision (CV) model based on neural networks.

[0003] In practical applications, large computer vision models have achieved remarkable results in tasks such as image classification, object detection, and semantic segmentation. The models automatically learn image features in an end-to-end manner, greatly simplifying the manual feature design process in traditional computer vision tasks and surpassing the level of human experts in many cases. However, when processing complex scenes or multimodal information, existing large computer vision models still have many shortcomings and are difficult to meet the needs. Summary of the invention

[0004] In view of the above problems, a processing method, device, equipment and medium based on a computer vision large model are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A processing method based on a large computer vision model, the method comprising:

[0006] Build a knowledge graph related to the recognition task;

[0007] Acquire sample image data, fuse the sample image data with the knowledge graph, and use the fused data to train a preset computer vision large model;

[0008] The target image data is acquired, and the trained computer vision model is called to recognize the target image data to obtain a recognition result.

[0009] Optionally, the fusing the sample image data and the knowledge graph includes:

[0010] Determining attention scores of multiple regions in the sample image data based on an attention mechanism;

[0011] The weight of the relevant information in the knowledge graph is adjusted according to the attention score to fuse the sample image data and the knowledge graph.

[0012] Optionally, the sample image data includes unlabeled data, and before acquiring the target image data, calling the trained computer vision large model, identifying the target image data, and obtaining the identification result, the following is further included:

[0013] Acquire unlabeled data from the sample image data;

[0014] Generate pseudo labels for the unlabeled data according to the knowledge graph;

[0015] The computer vision large model is controlled to perform self-learning through the pseudo-labels.

[0016] Optionally, the sample image data includes annotated data, and before acquiring the target image data, calling the trained computer vision large model, identifying the target image data, and obtaining the identification result, the following is further included:

[0017] Acquire labeled data from the sample image data;

[0018] The preset large computer vision model is fine-tuned and trained according to the labeled data.

[0019] Optionally, before acquiring the target image data, calling the trained computer vision model, recognizing the target image data, and obtaining the recognition result, the method further includes:

[0020] In the process of training the large computer vision model, a multi-task learning framework is used to control the large computer vision model to learn multiple recognition tasks.

[0021] Optionally, after constructing the knowledge graph related to the recognition task, the method further includes:

[0022] Cross-domain migration of data in knowledge graphs in multiple fields.

[0023] Optionally, after acquiring the sample image data, the method further includes: preprocessing the sample image data; wherein the preprocessing includes any one or more of the following: standardization processing and normalization processing.

[0024] A processing device based on a large computer vision model, the device comprising:

[0025] Knowledge graph construction module, used to construct knowledge graphs related to recognition tasks;

[0026] A model fusion knowledge graph module is used to obtain sample image data, fuse the sample image data with the knowledge graph, and use the fused data to train a preset computer vision large model;

[0027] The model image recognition module is used to obtain target image data, call the trained computer vision large model, recognize the target image data, and obtain a recognition result.

[0028] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method described above when executed by the processor.

[0029] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0030] The embodiments of the present invention have the following advantages:

[0031] In an embodiment of the present invention, by constructing a knowledge graph related to the recognition task, sample image data is obtained, the sample image data and the knowledge graph are fused, and the fused data is used to train a preset computer vision big model, obtain target image data, call the trained computer vision big model, recognize the target image data, and obtain a recognition result. This realizes the fusion of the computer vision big model and the knowledge graph to perform image recognition, thereby being able to meet the needs of processing complex scenes or multimodal information and improving the ability of image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0033] Figure 1 is a flowchart of a processing method based on a computer vision large model provided by some embodiments of the present invention;

[0034] Figure 2 is a flowchart of another processing method based on a large computer vision model provided by some embodiments of the present invention;

[0035] Figure 3 is a flowchart of another processing method based on a large computer vision model provided by some embodiments of the present invention;

[0036] Figure 4 It is a structural block diagram of a processing device based on a computer vision large model provided by some embodiments of the present invention. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] When dealing with complex scenes or multimodal information, existing large computer vision models still have many deficiencies and are difficult to meet the needs. For example, when there is a complex background in the image or objects are occluded by each other, because the model mainly relies on local features in the image for recognition, and lacks effective use of global context information and the relationship between objects, it is difficult for the model to accurately identify the target object. For another example, because deep learning models require a large amount of labeled data to learn generalized feature representations, and in certain specific fields or long-tail categories, it is very difficult to obtain a large amount of high-quality labeled data, the performance of the model often drops significantly when faced with problems such as small sample learning and long-tail category recognition.

[0039] Based on this, the present invention solves the problem of poor performance of the model in complex scenarios by effectively introducing external knowledge into the large computer vision model, enhancing the model's understanding and generalization capabilities.

[0040] The present invention is exemplarily described below with reference to the accompanying drawings:

[0041] Reference Figure 1 , showing a flowchart of the steps of a processing method based on a large computer vision model provided by some embodiments of the present invention, a large computer vision model refers to a large deep learning model used in the field of computer vision, which has a large number of parameters and a deeper network structure, and can handle complex visual tasks such as image classification, target detection, semantic segmentation, etc. The large computer vision model is trained through a large-scale data set and can achieve a high accuracy rate in various visual tasks. With the improvement of hardware computing power and the optimization of algorithms, the large computer vision model can show better performance in high-resolution image processing.

[0042] Specifically, the following steps may be included:

[0043] Step 101, construct a knowledge graph related to the recognition task.

[0044] In practical applications, we can build a knowledge graph related to the recognition task, define entities, relationships and attributes, extract entities, attributes and relationships related to object categories from the knowledge base, organize this information into a knowledge graph structure, and use deep learning-based embedding methods to map them into vector space for subsequent fusion operations.

[0045] For example, the following formula is used to map entities and relations in the knowledge graph to a high-dimensional vector space:

[0046] h i =ReLU(W (0) v i +b (0) )

[0047] r ij =ReLU(W (1) e ij +b (1) )

[0048] Among them, v i and e ij Respectively represent the node v i and e ij The original representation of W (0) and W (1) is the weight matrix; b (0) , b (1) is the bias vector; h i and r ij is the final embedding vector.

[0049] In some examples, in the process of building a knowledge graph, in addition to extracting entities, attributes, and relationships related to object categories from the knowledge base, we should also consider how to effectively integrate information from multiple knowledge sources to build a richer and more comprehensive knowledge graph. For example, we can combine visual dictionaries, common sense knowledge bases, and even domain-specific databases to enrich the content of the graph to ensure that as much relevant information as possible is covered; in addition, considering the changes and development of the knowledge graph over time, a mechanism should be designed to regularly update the knowledge graph to ensure its timeliness and accuracy.

[0050] In some examples, when mapping entities and relations in knowledge graphs to high-dimensional vector spaces, in addition to using deep learning methods, you can also consider introducing additional feature engineering, such as rule-based methods or heuristic methods, to enhance the accuracy of the mapping; for example, you can use domain knowledge to define specific mapping rules, or use heuristic algorithms to handle mapping problems in special cases.

[0051] In addition, considering the complexity of the knowledge graph, the mapping process may encounter the problem of multiple meanings of entities and relationships. To solve this problem, multi-view embedding technology can be used to create multiple embedding vectors for each entity and relationship. Each vector represents different semantics or contexts. This can more accurately capture the meaning of entities and relationships in different scenarios and improve the performance of the model in complex tasks.

[0052] Finally, when mapping the knowledge graph to a high-dimensional vector space, the dimensionality optimization of the vector space should be considered. Too high a dimension will lead to increased computational costs, while too low a dimension may lead to information loss. Therefore, the vector dimension can be optimized through dimensionality reduction techniques such as PCA (Principal Component Analysis) and t-SNE (t-distributed Stochastic Neighbor Embedding) to reduce computational complexity while maintaining information integrity.

[0053] In the embodiment of the present invention, through these extension measures, a more complete knowledge graph can be constructed, and through a sophisticated mapping strategy, the information in the knowledge graph can be effectively integrated into the CV large model, thereby improving the recognition and generalization capabilities of the model.

[0054] In some embodiments of the present invention, after constructing the knowledge graph related to the recognition task, it also includes: cross-domain migration of data in the knowledge graphs of multiple fields.

[0055] In practical applications, by realizing cross-domain knowledge transfer, the knowledge graph of the source domain is aligned and mapped with the knowledge graph of the target domain, and then the aligned knowledge graph is integrated into the CV large model of the target domain, and the model is fine-tuned and trained using the labeled data of the target domain.

[0056] In some examples, cross-domain knowledge migration is achieved through the alignment and mapping of knowledge graphs, and the model performance is further optimized in the target domain. Specifically, the knowledge graph of the source domain is aligned and mapped with the knowledge graph of the target domain, and the following formula is used for knowledge migration:

[0057] z′ i =T·z i +b

[0058] Among them, z′ i is the migrated vector, T is the transformation matrix, and b is the bias vector.

[0059] In order to ensure the accuracy and effectiveness of knowledge transfer, the following specific measures can be taken:

[0060] 1. Feature space alignment: Before aligning the knowledge graphs of the source and target domains, the feature space must be aligned first; this means that it is necessary to find common feature representations between the two domains to ensure that the entities and relationships in the knowledge graphs can be accurately mapped between different domains; this can be achieved by calculating the similarity matrix of the entities in the two domains and using an optimization algorithm (such as the Hungarian algorithm) to find the best match.

[0061] 2. Transformation matrix learning: In order to learn the transformation matrix, a bilinear model or other matrix operation methods can be used to train the aligned entity pairs; specifically, the transformation matrix is ​​learned by minimizing the distance between the entity vectors before and after the migration, ensuring that the migrated entities maintain the original semantic relationship in the target domain.

[0062] 3. Bias vector calibration: The calibration of the bias vector is to compensate for the potential offset between the two domains. The value of the bias vector can be determined by calculating the average offset of the vectors before and after the migration on the aligned entity pairs to ensure that the migrated vector maintains the correct semantic position in the target domain.

[0063] 4. Post-migration evaluation: After migration, the knowledge graph needs to be evaluated to ensure that the migration effect meets expectations. A cross-validation method can be used to migrate on part of the data in the target domain and evaluate the accuracy and consistency of the migrated knowledge graph on another part of the data.

[0064] Step 102: obtain sample image data, fuse the sample image data with the knowledge graph, and use the fused data to train a preset computer vision large model.

[0065] In some embodiments of the present invention, the fusing of the sample image data and the knowledge graph includes: determining the attention scores of multiple regions in the sample image data based on an attention mechanism; and adjusting the weights of relevant information in the knowledge graph according to the attention scores to fuse the sample image data and the knowledge graph.

[0066] In practical applications, by designing a knowledge fusion strategy based on the attention mechanism, for each input image, the importance score of each region, namely the attention score (Attention Weights), is calculated, and then the relevant information in the knowledge graph is weighted and fused according to these scores.

[0067] In order to better utilize the information in the knowledge graph, a fusion algorithm based on the attention mechanism is used to process the input image; specifically, the importance scores of each region in the input image are first calculated, and then the weights of the knowledge graph information are adjusted according to these scores to be integrated into the feature representation of the model. In the embodiment of the present invention, this process not only enhances the model's understanding of the global context, but also helps the model to accurately identify the target object in a complex background.

[0068] In the specific implementation, a multi-head attention mechanism can be used, which can capture information from different representation subspaces, thereby improving the expressiveness of the model. Assume that the input image is divided into multiple regions, each region corresponds to a feature vector; for each region, calculate the attention score between it and all other regions, and the attention score reflects the correlation between the region and other regions; the attention score can be calculated using the dot product attention function:

[0069]

[0070] Where Q and K represent the query vector and key vector respectively. k is the dimension of the key vector, and the attention score matrix is ​​then normalized to a probability distribution through a softmax function and used as a weighting factor to combine the value vectors:

[0071] attention(Q,K,V)=softmax(score(Q,K))V

[0072] Finally, the weights calculated by the attention mechanism are applied to the relevant information in the knowledge graph to adjust the contribution of entities and relationships in the knowledge graph. For example, if the importance score of a certain area is high, then the entities and relationships related to that area in the knowledge graph will be given higher weights accordingly. In this way, the model can dynamically decide which pieces of knowledge are more important and adjust its decision-making process accordingly.

[0073] In some embodiments of the present invention, after acquiring the sample image data, the method further includes: preprocessing the sample image data; wherein the preprocessing includes any one or more of the following: standardization processing and normalization processing.

[0074] In practical applications, the input image data is first standardized to scale the pixel values ​​to the range of 0-1, and then normalized so that each pixel value is subtracted from the mean and divided by the standard deviation to eliminate the differences in lighting and contrast between different images.

[0075] In some examples, in order to ensure the consistency of image data and improve the efficiency of model training, the input image data is first standardized to scale the pixel values ​​to the range of 0 to 1; specifically, for each input image, each pixel value is divided by the maximum possible pixel intensity value (for 8-bit RGB images, this value is 255), thereby converting all pixel values ​​to between 0 and 1; this allows images from different sources or under different shooting conditions to have the same numerical range before being input into the model, thereby eliminating brightness differences between images.

[0076] In some examples, in order to further reduce the differences in lighting and contrast between images, normalization is performed. The normalization process includes two steps: subtracting the average pixel value of the image dataset from each pixel value, and then dividing the result by the standard deviation of the image dataset; after this processing, the distribution of the image data is close to the standard normal distribution, that is, the mean is 0 and the variance is 1; through such standardization and normalization processing, it can be ensured that the image data has consistent statistical characteristics in the entire dataset, which is critical to improving the training speed and stability of the model.

[0077] It should be noted that when calculating the mean and standard deviation, it is best to do so based on the entire training set or a large representative subset to ensure that these statistics reflect the true characteristics of the image dataset; for different color channels (such as red, green, and blue), the respective means and standard deviations can be calculated separately and applied to the corresponding channel data to preserve the relative proportion of color information;

[0078] In an embodiment of the present invention, through standardization and normalization processing, the image data is not only more consistent, but also easier to be learned by the neural network model, because the model no longer needs to deal with situations where the numerical range is too large or too small, and the risk of gradient disappearance or explosion during training is also reduced; in addition, these preprocessing steps also help the model to make more accurate predictions for new images during the testing phase, because it can convert new images into a form similar to the training data.

[0079] In some embodiments of the present invention, the sample image data includes labeled data, and before acquiring the target image data, calling the trained computer vision large model, identifying the target image data, and obtaining the recognition result, it also includes: acquiring the labeled data from the sample image data; and fine-tuning the preset computer vision large model according to the labeled data.

[0080] In practical applications, the labeled data in the labeled dataset can be used to pre-train the large CV model. For example, ResNet-50 is used as the basic network structure, and the pre-trained weights are used as initialization parameters to perform fine-tuning training on the ImageNet dataset.

[0081] In some examples, a pre-trained ResNet-50 model is used as a basis to learn feature representations of different categories of objects by fine-tuning on the ImageNet dataset.

[0082] In some examples, the following loss functions can be used for training:

[0083] L=L cv +αL kg

[0084] Among them, L cv is the loss function for computer vision tasks, and L kg is the KG embedding loss function and α is the balancing factor.

[0085] In order to further improve the generalization ability of the model, the following strategies are used to optimize the fine-tuning process:

[0086] 1. Learning rate scheduling: Set a higher learning rate initially to facilitate rapid convergence, and gradually reduce the learning rate as training progresses to ensure that the model can fine-tune parameters and find the optimal solution. In terms of specific implementation, an exponentially decaying learning rate strategy is adopted, that is, after a certain number of training batches, the learning rate decreases at a predetermined ratio.

[0087] 2. Data enhancement: By randomly cropping, flipping, rotating and other operations on the training data, the diversity of the data set is increased, thereby improving the robustness and generalization ability of the model; in addition, the color jitter technology is used, that is, randomly changing the brightness, contrast and saturation of the image, to further enhance the model's adaptability to different inputs.

[0088] 3. Regularization technology: Use Dropout technology to randomly shut down some neurons during training to reduce the model's excessive dependence on specific features and improve the model's generalization ability; at the same time, use L2 regularization to penalize the size of the weight vector to prevent overfitting.

[0089] 4. Batch Normalization: During the training process, the output of each layer is normalized to speed up the training and improve the stability of the model; batch normalization can reduce the problem of internal covariate shift and make the model easier to learn.

[0090] In some examples, for the knowledge graph embedding loss function Lkg , we can use methods based on triple loss or margin loss to ensure that the embedding vector can accurately represent the logical connection between entities and relations. For example, we can define the loss function as:

[0091]

[0092] Among them, (h, r, t) is the triple in the knowledge graph, m is the margin value, the sim function is used to calculate the similarity between two embedding vectors (such as cosine similarity), (h′, t′) is a negative sample pair, and the loss function encourages the embedding vector distance of the correct triple to be less than the distance of the wrong triple plus the margin value.

[0093] In some embodiments of the present invention, the sample image data includes unlabeled data. Before acquiring the target image data, calling the trained computer vision large model, and identifying the target image data to obtain the identification result, it also includes: acquiring unlabeled data from the sample image data; generating pseudo labels for the unlabeled data based on the knowledge graph; and controlling the computer vision large model to perform self-learning through the pseudo labels.

[0094] In practical applications, weakly supervised learning methods can be applied to unlabeled data. Prior knowledge in knowledge graphs can be used to generate pseudo labels for unlabeled data, and these pseudo labels can be used as supervisory signals to guide the model to self-learn. Weakly supervised learning methods can be used to reduce dependence on labeled data by generating pseudo labels and using them in subsequent training.

[0095] In some examples, the pseudo-label generation formula is as follows:

[0096]

[0097] Where P(y|x) represents the probability distribution of output y given input x;

[0098] In order to further improve the effect of weakly supervised learning, the specific implementation plan is as follows:

[0099] 1. Pseudo-label generation: After the initial training of the model, use the model to predict the unlabeled data; based on the prediction results, select those categories with the highest prediction probability as pseudo-labels; this is done to ensure that the generated pseudo-labels have high credibility, thereby reducing the impact of noise.

[0100] 2. Pseudo-label update: After each iteration of updating the model parameters, the latest model is used to re-predict the unlabeled data, and the pseudo-label is updated according to the new prediction results; as the model is continuously optimized, the accuracy of the pseudo-label is gradually improved.

[0101] 3. Data augmentation: Perform data augmentation on the unlabeled data, including random cropping, rotation, and flipping to generate more training samples. Then, pseudo-labels are generated again for these augmented samples to further enrich the training dataset.

[0102] 4. Consistency Regularization: When generating pseudo labels, consistency regularization techniques are used to constrain the output of the model. The specific method is to achieve consistency by minimizing the difference in predictions for the same input under different augmentation methods, thereby ensuring that the model produces similar predictions on different versions of the same input.

[0103] 5. Model integration: Use model integration technology to improve the reliability of pseudo labels. The specific operation is to train multiple base models and take the average of the prediction results of each base model to generate pseudo labels. This method can reduce the impact of single model prediction errors and improve the quality of pseudo labels.

[0104] In some embodiments of the present invention, before acquiring the target image data, calling the trained computer vision big model, recognizing the target image data, and obtaining the recognition result, it also includes: in the process of training the computer vision big model, using a multi-task learning framework to control the computer vision big model to learn multiple recognition tasks.

[0105] In practical applications, a multi-task learning framework is introduced on the basis of the CV large model, tasks such as object part recognition and scene recognition are added, and multi-task learning is achieved by sharing the underlying feature representation layer and setting task-specific output layers.

[0106] In some examples, a multi-task learning framework is used to simultaneously optimize multiple related tasks during training to improve the generalization ability of the model, and the multi-task loss function is calculated using the following formula:

[0107]

[0108] Among them, T is the task set, L t is the loss function of the task, λ t is the weight coefficient of the corresponding task.

[0109] Specifically, for task selection in the multi-task learning framework, in addition to the main object recognition task, auxiliary tasks such as object part recognition and scene recognition can also be added; these auxiliary tasks share the underlying feature representation layer with the main task, but set their own task-specific output layer at the top level.

[0110] In the embodiment of the present invention, by sharing the underlying feature representation layer, information sharing between different tasks can be promoted, thereby improving the model learning efficiency; and setting a task-specific output layer can ensure that each task has dedicated parameters to adapt to its specific needs.

[0111] During the training process, the weight coefficient of each task needs to be adjusted dynamically. This can be achieved by monitoring the performance of different tasks on the validation set. If an auxiliary task begins to affect the performance of the main task, the weight of the task is appropriately reduced, and vice versa. This ensures that the training focus of the model at different stages can be adjusted according to the actual situation, so as to achieve the best overall model performance.

[0112] In some examples, in order to avoid interference between different tasks, a gating mechanism can be used to control the information flow between different tasks. The gating mechanism uses the learned gating parameters to determine which information can flow from one task to another, thereby ensuring that each task can focus on its own learning goals while benefiting from the information of other tasks.

[0113] Step 103, obtain target image data, call the trained computer vision model, recognize the target image data, and obtain a recognition result.

[0114] In practical applications, by deploying the trained model to practical applications, the newly input target image data is recognized and analyzed in real time, and the recognition results are output.

[0115] In an embodiment of the present invention, by constructing a knowledge graph related to the recognition task, sample image data is obtained, the sample image data and the knowledge graph are fused, and the fused data is used to train a preset computer vision big model, obtain target image data, call the trained computer vision big model, recognize the target image data, and obtain a recognition result. This realizes the fusion of the computer vision big model and the knowledge graph to perform image recognition, thereby being able to meet the needs of processing complex scenes or multimodal information and improving the ability of image recognition.

[0116] Specifically, through the knowledge fusion strategy based on the attention mechanism, the importance scores of each area of ​​the input image are first calculated, and then the relevant information in the knowledge graph is weighted and fused according to these scores, so that the model not only focuses on the local features in the image, but also captures the global context information, which helps to accurately identify the target objects in complex backgrounds. This fusion algorithm based on the attention mechanism calculates the importance scores of each area in the input image and adjusts the weight of the knowledge graph information accordingly to integrate it into the feature representation of the model, thereby enhancing the model's understanding of the global context.

[0117] Moreover, by using weakly supervised learning methods, it uses the prior knowledge in the knowledge graph to generate pseudo labels for unlabeled data, and uses these pseudo labels as supervision signals to guide the model to self-learn, which reduces the need for a large amount of labeled data. It is particularly suitable for specific fields or long-tail categories where it is difficult to obtain a large amount of high-quality labeled data. By generating pseudo labels and using these labels in subsequent training, the model can still maintain good performance when data is scarce. This adaptive learning strategy allows the model to dynamically adjust the training process according to the structure and content of the knowledge graph, thereby performing well in recognition tasks of small samples and long-tail categories.

[0118] The following combination Figure 2 The present invention is exemplified as follows:

[0119] S1, input image data, standardize and normalize it, scale the image pixel values ​​to the interval 0-1, and then further subtract the mean from each pixel value and divide it by the standard deviation;

[0120] S2, build a knowledge graph related to the recognition task, define entities, relations and attributes;

[0121] S3. Use the labeled data set to pre-train the CV large model, use ResNet-50 as the basic network structure, use the pre-trained weights as initialization parameters, and perform fine-tuning training on the ImageNet data set;

[0122] S4. Design a knowledge fusion strategy based on the attention mechanism. For each input image, calculate the importance scores (attention weights) of each region, and then weight the fusion of relevant information in the knowledge graph according to these scores;

[0123] S5. Based on the CV large model, a multi-task learning framework is introduced to add tasks such as object part recognition and scene recognition, and multi-task learning is achieved by sharing the underlying feature representation layer and setting task-specific output layers;

[0124] S6. Apply weakly supervised learning methods on unlabeled data, use the prior knowledge in the knowledge graph to generate pseudo labels for unlabeled data, and use these pseudo labels as supervision signals to guide the model to self-learn;

[0125] S7, realize cross-domain knowledge transfer, align and map the knowledge graph of the source domain with the knowledge graph of the target domain, then integrate the aligned knowledge graph into the CV large model of the target domain, and fine-tune the model with the labeled data of the target domain;

[0126] S8. Deploy the trained model to practical applications, perform real-time recognition and analysis on new input images, and output the recognition results.

[0127] Reference Figure 3 , shows a flowchart of the steps of another processing method based on a large computer vision model provided by some embodiments of the present invention. A large computer vision model refers to a large deep learning model used in the field of computer vision. Such a model has a large number of parameters and a deeper network structure, and can handle complex visual tasks such as image classification, target detection, semantic segmentation, etc. The large computer vision model is trained with a large-scale data set and can achieve a high accuracy rate in various visual tasks. With the improvement of hardware computing power and the optimization of algorithms, the large computer vision model can show better performance in high-resolution image processing.

[0128] Specifically, the following steps may be included:

[0129] Step 301, construct a knowledge graph related to the recognition task.

[0130] In practical applications, we can build a knowledge graph related to the recognition task, define entities, relationships and attributes, extract entities, attributes and relationships related to object categories from the knowledge base, organize this information into a knowledge graph structure, and use deep learning-based embedding methods to map them into vector space for subsequent fusion operations.

[0131] For example, the following formula is used to map entities and relations in the knowledge graph to a high-dimensional vector space:

[0132] h i =ReLU(W (0) v i +b (0) )

[0133] r ij =ReLU(W (1) e ij +b (1) )

[0134] Among them, v i and e ij Respectively represent the node v i and e ij The original representation of W (0) and W (1) is the weight matrix; b (0) , b (1) is the bias vector; h i and r ij is the final embedding vector.

[0135] In some examples, in the process of building a knowledge graph, in addition to extracting entities, attributes, and relationships related to object categories from the knowledge base, we should also consider how to effectively integrate information from multiple knowledge sources to build a richer and more comprehensive knowledge graph. For example, we can combine visual dictionaries, common sense knowledge bases, and even domain-specific databases to enrich the content of the graph to ensure that as much relevant information as possible is covered; in addition, considering the changes and development of the knowledge graph over time, a mechanism should be designed to regularly update the knowledge graph to ensure its timeliness and accuracy.

[0136] In some examples, when mapping entities and relations in knowledge graphs to high-dimensional vector spaces, in addition to using deep learning methods, you can also consider introducing additional feature engineering, such as rule-based methods or heuristic methods, to enhance the accuracy of the mapping; for example, you can use domain knowledge to define specific mapping rules, or use heuristic algorithms to handle mapping problems in special cases.

[0137] In addition, considering the complexity of the knowledge graph, the mapping process may encounter the problem of multiple meanings of entities and relationships. To solve this problem, multi-view embedding technology can be used to create multiple embedding vectors for each entity and relationship. Each vector represents different semantics or contexts. This can more accurately capture the meaning of entities and relationships in different scenarios and improve the performance of the model in complex tasks.

[0138] Finally, when mapping the knowledge graph to a high-dimensional vector space, the dimensionality optimization of the vector space should be considered. Too high a dimension will lead to increased computational costs, while too low a dimension may lead to information loss. Therefore, the vector dimension can be optimized through dimensionality reduction techniques such as PCA (Principal Component Analysis) and t-SNE (t-distributed Stochastic Neighbor Embedding) to reduce computational complexity while maintaining information integrity.

[0139] In the embodiment of the present invention, through these extension measures, a more complete knowledge graph can be constructed, and through a sophisticated mapping strategy, the information in the knowledge graph can be effectively integrated into the CV large model, thereby improving the recognition and generalization capabilities of the model.

[0140] Step 302: Obtain sample image data, and determine attention scores of multiple regions in the sample image data based on an attention mechanism.

[0141] Step 303: adjust the weight of the relevant information in the knowledge graph according to the attention score to fuse the sample image data and the knowledge graph.

[0142] Step 304, using the fused data to train a preset computer vision large model.

[0143] In practical applications, by designing a knowledge fusion strategy based on the attention mechanism, for each input image, the importance score of each region, namely the attention score (Attention Weights), is calculated, and then the relevant information in the knowledge graph is weighted and fused according to these scores.

[0144] In order to better utilize the information in the knowledge graph, a fusion algorithm based on the attention mechanism is used to process the input image; specifically, the importance scores of each region in the input image are first calculated, and then the weights of the knowledge graph information are adjusted according to these scores to be integrated into the feature representation of the model. In the embodiment of the present invention, this process not only enhances the model's understanding of the global context, but also helps the model to accurately identify the target object in a complex background.

[0145] In the specific implementation, a multi-head attention mechanism can be used, which can capture information from different representation subspaces, thereby improving the expressiveness of the model. Assume that the input image is divided into multiple regions, each region corresponds to a feature vector; for each region, calculate the attention score between it and all other regions, and the attention score reflects the correlation between the region and other regions; the attention score can be calculated using the dot product attention function:

[0146]

[0147] Where Q and K represent the query vector and key vector respectively. k is the dimension of the key vector, and the attention score matrix is ​​then normalized to a probability distribution through a softmax function and used as a weighting factor to combine the value vectors:

[0148] attention(Q,K,V)=softmax(score(Q,K))V

[0149] Finally, the weights calculated by the attention mechanism are applied to the relevant information in the knowledge graph to adjust the contribution of entities and relationships in the knowledge graph. For example, if the importance score of a certain area is high, then the entities and relationships related to that area in the knowledge graph will be given higher weights accordingly. In this way, the model can dynamically decide which pieces of knowledge are more important and adjust its decision-making process accordingly.

[0150] Step 305, obtain target image data, call the trained computer vision model, identify the target image data, and obtain a recognition result.

[0151] In practical applications, by deploying the trained model to practical applications, the newly input target image data is recognized and analyzed in real time, and the recognition results are output.

[0152] In an embodiment of the present invention, by constructing a knowledge graph related to the recognition task, sample image data is obtained, the sample image data and the knowledge graph are fused, and the fused data is used to train a preset computer vision big model, obtain target image data, call the trained computer vision big model, recognize the target image data, and obtain a recognition result. This realizes the fusion of the computer vision big model and the knowledge graph to perform image recognition, thereby being able to meet the needs of processing complex scenes or multimodal information and improving the ability of image recognition.

[0153] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0154] Reference Figure 4 , shows a schematic diagram of the structure of a processing device based on a computer vision large model provided by some embodiments of the present invention, which may specifically include the following modules:

[0155] A knowledge graph construction module 401 is used to construct a knowledge graph related to the recognition task;

[0156] The model fusion knowledge graph module 402 is used to obtain sample image data, fuse the sample image data with the knowledge graph, and use the fused data to train a preset computer vision large model;

[0157] The model image recognition module 403 is used to obtain target image data, call the trained computer vision large model, recognize the target image data, and obtain a recognition result.

[0158] In some embodiments of the present invention, the fusing of the sample image data and the knowledge graph includes:

[0159] Determining attention scores of multiple regions in the sample image data based on an attention mechanism;

[0160] The weight of the relevant information in the knowledge graph is adjusted according to the attention score to fuse the sample image data and the knowledge graph.

[0161] In some embodiments of the present invention, the sample image data includes unlabeled data, and before acquiring the target image data, calling the trained computer vision model, and identifying the target image data to obtain the identification result, the following steps are further included:

[0162] Acquire unlabeled data from the sample image data;

[0163] Generate pseudo labels for the unlabeled data according to the knowledge graph;

[0164] The computer vision large model is controlled to perform self-learning through the pseudo-labels.

[0165] In some embodiments of the present invention, the sample image data includes annotated data, and before acquiring the target image data, calling the trained computer vision model, and identifying the target image data to obtain the identification result, the following steps are further included:

[0166] Acquire labeled data from the sample image data;

[0167] The preset large computer vision model is fine-tuned and trained according to the labeled data.

[0168] In some embodiments of the present invention, before acquiring the target image data, calling the trained computer vision model, and identifying the target image data to obtain the identification result, the following steps are further included:

[0169] In the process of training the large computer vision model, a multi-task learning framework is used to control the large computer vision model to learn multiple recognition tasks.

[0170] In some embodiments of the present invention, after constructing the knowledge graph related to the recognition task, the method further includes:

[0171] Cross-domain migration of data in knowledge graphs in multiple fields.

[0172] In some embodiments of the present invention, after acquiring the sample image data, the method further includes: preprocessing the sample image data; wherein the preprocessing includes any one or more of the following: standardization processing and normalization processing.

[0173] In an embodiment of the present invention, by constructing a knowledge graph related to the recognition task, sample image data is obtained, the sample image data and the knowledge graph are fused, and the fused data is used to train a preset computer vision big model, obtain target image data, call the trained computer vision big model, recognize the target image data, and obtain a recognition result. This realizes the fusion of the computer vision big model and the knowledge graph to perform image recognition, thereby being able to meet the needs of processing complex scenes or multimodal information and improving the ability of image recognition.

[0174] Some embodiments of the present invention further provide an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the above method is implemented when the computer program is executed by the processor.

[0175] Some embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and the computer program implements the above method when executed by a processor.

[0176] Some embodiments of the present invention further provide a computer program product, including a computer program, which implements the above method when executed by a processor.

[0177] A processing method based on a large computer vision model, the method comprising:

[0178] Build a knowledge graph related to the recognition task;

[0179] Acquire sample image data, fuse the sample image data with the knowledge graph, and use the fused data to train a preset computer vision large model;

[0180] The target image data is acquired, and the trained computer vision model is called to recognize the target image data to obtain a recognition result.

[0181] Optionally, the fusing the sample image data and the knowledge graph includes:

[0182] Determining attention scores of multiple regions in the sample image data based on an attention mechanism;

[0183] The weight of the relevant information in the knowledge graph is adjusted according to the attention score to fuse the sample image data and the knowledge graph.

[0184] Optionally, the sample image data includes unlabeled data, and before acquiring the target image data, calling the trained computer vision large model, identifying the target image data, and obtaining the identification result, the following is further included:

[0185] Acquire unlabeled data from the sample image data;

[0186] Generate pseudo labels for the unlabeled data according to the knowledge graph;

[0187] The computer vision large model is controlled to perform self-learning through the pseudo-labels.

[0188] Optionally, the sample image data includes annotated data, and before acquiring the target image data, calling the trained computer vision large model, identifying the target image data, and obtaining the identification result, the following is further included:

[0189] Acquire labeled data from the sample image data;

[0190] The preset large computer vision model is fine-tuned and trained according to the labeled data.

[0191] Optionally, before acquiring the target image data, calling the trained computer vision model, recognizing the target image data, and obtaining the recognition result, the method further includes:

[0192] In the process of training the large computer vision model, a multi-task learning framework is used to control the large computer vision model to learn multiple recognition tasks.

[0193] Optionally, after constructing the knowledge graph related to the recognition task, the method further includes:

[0194] Cross-domain migration of data in knowledge graphs in multiple fields.

[0195] Optionally, after acquiring the sample image data, the method further includes: preprocessing the sample image data; wherein the preprocessing includes any one or more of the following: standardization processing and normalization processing.

[0196] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0197] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0198] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0199] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0200] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0203] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0204] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0205] The processing method, device, equipment and medium based on the computer vision large model are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A processing method based on a large computer vision model, characterized in that: The method comprises: Build a knowledge graph related to the recognition task; Acquire sample image data, fuse the sample image data with the knowledge graph, and use the fused data to train a preset computer vision large model; The target image data is acquired, and the trained computer vision model is called to recognize the target image data to obtain a recognition result.

2. The method according to claim 1, characterized in that The fusing the sample image data and the knowledge graph includes: Determining attention scores of multiple regions in the sample image data based on an attention mechanism; The weight of the relevant information in the knowledge graph is adjusted according to the attention score to fuse the sample image data and the knowledge graph.

3. The method according to claim 1 or 2, characterized in that: The sample image data includes unlabeled data. Before acquiring the target image data, calling the trained computer vision model, and identifying the target image data to obtain the identification result, the following steps are also included: Acquire unlabeled data from the sample image data; Generate pseudo labels for the unlabeled data according to the knowledge graph; The computer vision large model is controlled to perform self-learning through the pseudo-labels.

4. The method according to claim 1 or 2, characterized in that: The sample image data includes annotated data. Before acquiring the target image data, calling the trained computer vision model, and identifying the target image data to obtain the identification result, the following steps are also included: Acquire labeled data from the sample image data; The preset large computer vision model is fine-tuned and trained according to the labeled data.

5. The method according to claim 1 or 2, characterized in that: Before acquiring the target image data, calling the trained computer vision model, and identifying the target image data to obtain the identification result, the method further includes: In the process of training the large computer vision model, a multi-task learning framework is used to control the large computer vision model to learn multiple recognition tasks.

6. The method according to claim 1 or 2, characterized in that: After constructing the knowledge graph related to the recognition task, it also includes: Cross-domain migration of data in knowledge graphs in multiple fields.

7. The method according to claim 1, characterized in that After acquiring the sample image data, the method further includes: preprocessing the sample image data; wherein the preprocessing includes any one or more of the following: standardization processing and normalization processing.

8. A processing device based on a large computer vision model, characterized in that: The device comprises: Knowledge graph construction module, used to construct knowledge graphs related to recognition tasks; A model fusion knowledge graph module is used to obtain sample image data, fuse the sample image data with the knowledge graph, and use the fused data to train a preset computer vision large model; The model image recognition module is used to obtain target image data, call the trained computer vision large model, recognize the target image data, and obtain a recognition result.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images