Image analysis method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an image analysis method and device, computer equipment, a readable storage medium and a program product. The method comprises the steps that in a preset semantic segmentation model, images in a to-be-analyzed image set are processed according to a preset knowledge graph, structural feature vectors of the images are obtained, and the structural feature vectors represent structures of target objects in the images; dividing based on the time features of each image to obtain a plurality of structural feature vector groups, and performing semantic similarity calculation on structural feature vectors in each structural feature vector group to obtain similarity data of each structural feature vector group; and determining an evolution path of the target object according to a dynamic path planning strategy and the similarity data of each structural feature vector group. By adopting the method, the utilization rate of the image data can be improved.

Description

Technical Field

[0001] The present application relates to the fields of computers and cultural heritage, and in particular to an image analysis method, apparatus, computer equipment, readable storage medium and program product. Background Art

[0002] As an important information carrier, images carry a wealth of historical and cultural information. In the fields of archaeology, history, art history, and classical literature, images include not only on-site photos of archaeological sites and digital images of artworks of historical events, but also various forms of visual materials such as ancient manuscripts, inscriptions, and bronzes. Images not only provide researchers with visual materials, but also contain a lot of potential value to be interpreted and analyzed.

[0003] The commonly used methods for analyzing images in related technologies are: (1) manual analysis, analyzing images manually; (2) analyzing images by combining manual analysis with conventional algorithms; (3) analyzing images by deep learning. However, in related technologies, image data is obtained by analyzing images manually or by deep learning, and the image data is merely annotated. Therefore, the utilization rate of image data in related technologies is low. Summary of the invention

[0004] Based on this, it is necessary to provide an image analysis method, apparatus, computer equipment, readable storage medium and program product that can improve the utilization rate of image data in response to the above technical problems.

[0005] In a first aspect, the present application provides an image analysis method, comprising:

[0006] In a preset semantic segmentation model, each image in the image set to be analyzed is processed according to a preset knowledge graph to obtain a structural feature vector of each image, wherein the structural feature vector represents each structure of the target object in the image;

[0007] Dividing the images based on their temporal features to obtain a plurality of structural feature vector groups, and calculating semantic similarity of the structural feature vectors in each of the structural feature vector groups to obtain similarity data of each of the structural feature vector groups;

[0008] The evolution path of the target object is determined according to a dynamic path planning strategy and the similarity data of each of the structural feature vector groups.

[0009] In one of the embodiments, in the preset semantic segmentation model, each image in the image set to be analyzed is processed according to a preset knowledge graph to obtain a structural feature vector of each image, including:

[0010] In the preset semantic segmentation model, each image in the image set to be analyzed is segmented to obtain the structural features of each image;

[0011] The structural features are converted based on a preset knowledge graph to obtain a structural feature vector of each of the images.

[0012] In one embodiment, the method further comprises:

[0013] Acquire sample data, wherein the sample data includes each sample image and sample structure features corresponding to each sample image;

[0014] Performing segmentation processing on each of the sample images based on the initial semantic segmentation model to obtain predicted structural features corresponding to each of the sample images;

[0015] Based on the predicted structural features of each of the sample images and the sample structural features, a mask loss function is calculated, and based on the mask loss function, the initial semantic segmentation model is optimized to obtain the preset semantic segmentation model.

[0016] In one embodiment, the method further comprises:

[0017] Each of the sample images is labeled according to the preset knowledge graph to obtain sample structural features of each of the sample images.

[0018] In one embodiment, the method further comprises:

[0019] Arrange the time features of each of the structural feature vector groups in order to obtain a plurality of the structural feature vector groups;

[0020] For the i-th structural feature vector group, the cosine values ​​of each structural feature vector and each structural feature vector in the i+1-th structural feature vector group are calculated in turn, and the cosine values ​​are determined as the similarity data between the i-th structural feature vector group and the i+1-th structural feature vector group, 1≤i<n, and n is the number of the structural feature vector groups.

[0021] In one embodiment, obtaining the evolution path of the target object according to the dynamic path planning strategy and the similarity data of each of the structural feature vector groups includes:

[0022] For each structural feature vector in the first structural feature vector group, determine the maximum similarity data of the structural feature vector; determine the target structural feature vector of the second structural feature vector group according to the maximum similarity data, and determine the path between the structural feature vector in the first structural feature vector group and the target structural feature vector as the first evolution path;

[0023] For each target structural feature vector in the i-th structural feature vector group, confirm the maximum similarity data of the target structural feature vector; determine the target structural feature vector of the i+1-th structural feature vector group according to the maximum similarity data, and determine the path between the target structural feature vector in the i-th structural feature vector group and the target structural feature vector of the i+1-th structural feature vector group as the second evolution path;

[0024] Based on the first evolution path and the second evolution path, an evolution path of the target object is determined.

[0025] In one embodiment, the method further comprises:

[0026] The evolution path is displayed.

[0027] In a second aspect, the present application also provides an image analysis device, comprising:

[0028] A processing module, used to process each image in the image set to be analyzed according to a preset knowledge graph in a preset semantic segmentation model to obtain a structural feature vector of each image, wherein the structural feature vector represents each structure of the target object in the image;

[0029] A calculation module, used for dividing the images based on the time features of each image to obtain a plurality of structural feature vector groups, and performing semantic similarity calculation on the structural feature vectors in each structural feature vector group to obtain similarity data of each structural feature vector group;

[0030] The determination module is used to determine the evolution path of the target object according to the dynamic path planning strategy and the similarity data of each of the structural feature vector groups.

[0031] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0032] In a preset semantic segmentation model, each image in the image set to be analyzed is processed according to a preset knowledge graph to obtain a structural feature vector of each image, wherein the structural feature vector represents each structure of the target object in the image;

[0033] Dividing the images based on their temporal features to obtain a plurality of structural feature vector groups, and calculating semantic similarity of the structural feature vectors in each of the structural feature vector groups to obtain similarity data of each of the structural feature vector groups;

[0034] The evolution path of the target object is determined according to a dynamic path planning strategy and the similarity data of each of the structural feature vector groups.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0036] In a preset semantic segmentation model, each image in the image set to be analyzed is processed according to a preset knowledge graph to obtain a structural feature vector of each image, wherein the structural feature vector represents each structure of the target object in the image;

[0037] Dividing the images based on their temporal features to obtain a plurality of structural feature vector groups, and calculating semantic similarity of the structural feature vectors in each of the structural feature vector groups to obtain similarity data of each of the structural feature vector groups;

[0038] The evolution path of the target object is determined according to a dynamic path planning strategy and the similarity data of each of the structural feature vector groups.

[0039] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0040] In a preset semantic segmentation model, each image in the image set to be analyzed is processed according to a preset knowledge graph to obtain a structural feature vector of each image, wherein the structural feature vector represents each structure of the target object in the image;

[0041] Dividing the images based on their temporal features to obtain a plurality of structural feature vector groups, and calculating semantic similarity of the structural feature vectors in each of the structural feature vector groups to obtain similarity data of each of the structural feature vector groups;

[0042] The evolution path of the target object is determined according to a dynamic path planning strategy and the similarity data of each of the structural feature vector groups.

[0043] The above-mentioned image analysis method, device, computer equipment, readable storage medium and program product use a preset semantic segmentation model and a preset knowledge graph to annotate the fine-grained knowledge of the image, obtain the structural feature vector of each image, and divide the structural feature vector to obtain multiple structural feature vector groups, calculate the similarity data of each structural feature vector group, confirm the evolution path of the target object based on the dynamic path planning strategy and the similarity data, realize efficient processing of large-scale images, and analyze the structural feature vector to obtain the evolution path of the target object, thereby improving the utilization rate of image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is a schematic flow chart of an image analysis method in one embodiment;

[0046] Figure 2 is a schematic diagram of the structure of a semantic segmentation model in an embodiment;

[0047] Figure 3 A schematic diagram of components of a bronze tripod according to an embodiment;

[0048] Figure 4 is a schematic diagram of the structure of an evolution path in an embodiment;

[0049] Figure 5 A schematic diagram of an image analysis method in one embodiment

[0050] Figure 6 is a structural block diagram of an image analysis device in one embodiment;

[0051] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] In an exemplary embodiment, Figure 1 As shown, an image analysis method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0054] Step 101: In a preset semantic segmentation model, each image in the image set to be analyzed is processed according to a preset knowledge graph to obtain a structural feature vector of each image.

[0055] The preset semantic segmentation model may be a semantic segmentation model pre-trained by sample data, for example, the semantic segmentation model may be YOLO or a decoupled semantic segmentation model (MMSegmentation). The preset knowledge graph is used to represent and organize the structural information of the target object in the image in a structured form. The image set to be analyzed includes multiple images, each of which may be obtained by directly photographing the target object or scanning a book containing the target object. The structural feature vector represents each structure of the target object in the image, and the structure may be a key part in the evolution process of the target object in the image. For example, the key part may be determined by the preset knowledge graph.

[0056] Specifically, the terminal can perform semantic segmentation on each image in the image set to be analyzed in a preset semantic segmentation model to obtain the structural features of the target object in the image. The terminal converts the structural features of each image according to the preset knowledge graph to obtain the conversion result of the structural features of each image, and the conversion result of each image is the structural feature vector of the image. The terminal can store the structural feature vector corresponding to each image in a vector database. The terminal can deduplicate the structural feature vectors in the vector database, that is, delete the duplicate structural feature vectors, and store the deduplicated structural feature vectors in the standard vector database.

[0057] Step 102: divide each image based on its temporal features to obtain a plurality of structural feature vector groups, and calculate semantic similarity of the structural feature vectors in each structural feature vector group to obtain similarity data of each structural feature vector group.

[0058] The time feature may be the period in which the target object in the image is located.

[0059] Specifically, the terminal may store the corresponding relationship between each image and the time feature of each image in a database in advance, or each image may carry a time feature tag, and the terminal may identify the time feature tag of each image to obtain the time feature corresponding to each image. The terminal may obtain the time feature of the image from the database, and divide the structural feature vectors in the same time feature into a group according to the time feature. The specific grouping method may be: the terminal clusters the structural feature vectors of the same time feature to obtain a vector group, or the terminal may divide the structural feature vectors of the same time tag into a group according to the time feature tag, etc. It should be understood that the grouping method is only used for example and is not specifically limited here. The terminal sorts the grouped structural feature vectors based on the order of the time features to obtain multiple structural feature vector groups. The terminal may perform semantic similarity calculation on the structural feature vectors between two adjacent structural feature vector groups to obtain similarity data of the structural feature vectors in each structural feature vector group.

[0060] Step 103: Determine the evolution path of the target object according to the dynamic path planning strategy and the similarity data of each structural feature vector group.

[0061] The similarity data may represent the similarity of the structural feature vectors in each structural feature vector group.

[0062] Specifically, the terminal can process two adjacent structural feature vector groups in sequence to obtain the entire evolution path of the target object in the image. For two adjacent structural feature vector groups, the terminal can select the maximum similarity data in each structural feature vector group according to the dynamic path planning strategy, and confirm the path between the structural feature vectors corresponding to the similarity data as the evolution path between the two adjacent structural feature vector groups.

[0063] The above-mentioned image analysis method uses a preset semantic segmentation model and a preset knowledge graph to annotate the fine-grained knowledge of the image, obtains the structural feature vector of each image, and divides the structural feature vector to obtain multiple structural feature vector groups, calculates the similarity data of each structural feature vector group, and confirms the evolution path of the target object based on the dynamic path planning strategy and similarity data, thereby realizing efficient processing of large-scale images, and analyzing the structural feature vectors to obtain the evolution path of the target object, thereby improving the utilization rate of image data.

[0064] In an exemplary embodiment, in a preset semantic segmentation model, each image in the image set to be analyzed is processed according to a preset knowledge graph to obtain a structural feature vector of each image, including:

[0065] In the preset semantic segmentation model, each image in the image set to be analyzed is segmented to obtain the structural features of each image; the structural features are transformed based on the preset knowledge graph to obtain the structural feature vector of each image.

[0066] Specifically, the terminal can perform segmentation mask processing on each image in the image set to be analyzed through a preset semantic segmentation model, that is, the terminal can assign a corresponding category label to each pixel in the image according to the preset semantic segmentation model, determine the same type of category labels as the same structural feature, and the terminal can obtain the structural features of each image. The terminal can convert the structural features into a multi-dimensional structural feature vector represented by "0" and "1" according to the preset knowledge graph, and each dimensional structural feature vector can represent a structural feature. "0" indicates that the structural feature marked in the preset knowledge graph does not exist in the image, and "1" indicates that the structural feature marked in the preset knowledge graph exists in the image. The terminal can store the structural feature vector corresponding to each image in a vector database. The terminal can deduplicate the structural feature vectors in the vector database and store different structural feature vectors in the standardizer vector database. The image corresponding to the structural feature vector in the standardizer vector database can be considered as a standardizer.

[0067] Exemplarily, the image set to be analyzed may be a bronze tripod image set, and the preset knowledge graph may include 52 structural features of bronze tripods. The terminal may represent the structural features of each segmented bronze tripod image as a 52-dimensional structural feature vector represented by "0" and "1", where each dimension represents a structural feature of a bronze tripod, "1" indicates that the bronze tripod in the image has structural features, and "0" indicates that the bronze tripod in the image does not have structural features. The terminal may store the structural feature vector of each bronze tripod image in a vector database. It should be understood that the examples given here are for explanation only and do not constitute specific limitations.

[0068] In this embodiment, ontology technology (preset knowledge graph) and semantic segmentation technology are combined to realize the automatic annotation and production of fine-grained knowledge of large-scale image resources, thereby improving the efficiency and accuracy of image annotation.

[0069] In an exemplary embodiment, the image analysis method further includes:

[0070] Obtain sample data; perform segmentation processing on each sample image based on the initial semantic segmentation model to obtain the predicted structural features corresponding to each sample image; calculate the mask loss function based on the predicted structural features of each sample image and the sample structural features, and optimize the initial semantic segmentation model based on the mask loss function to obtain a preset semantic segmentation model.

[0071] The sample data includes sample images and sample structural features corresponding to the sample images. The sample images may be scanned images of photos or books. The initial semantic segmentation model may include a backbone feature extractor, a pixel decoder, and a transformer decoder. The backbone feature extractor may be a transformer model, such as a vision transformer, a window segmentation vision transformer, etc., and the pixel decoder may be a deconvolution network.

[0072] Specifically, the terminal can obtain a sample image and a sample structural feature of the sample image. Figure 2 As shown, Figure 2 It is a structural diagram of a semantic segmentation model in an embodiment. The terminal can add the sample image and the sample structural features corresponding to the sample image to the semantic segmentation model, extract the low-resolution features of the sample image in the backbone feature extractor, and send the low-resolution features to the pixel decoder; in the pixel decoder, the low-resolution features are gradually upsampled to generate high-resolution pixel feature embeddings to obtain image features, and the image features are processed in the Transformer decoder to generate a segmentation mask, that is, assign a category label to each pixel in the image to obtain the predicted structural features corresponding to each sample image. Mask2Former also uses multi-scale high-resolution features. Through the feature pyramid, features of different scales are input into different layers of the Transformer decoder respectively. At the same time, sinusoidal position encoding and learnable scale-level embedding are added to each scale to enhance the model's perception of features of different scales. Mask2Former uses cross attention (Masked Attention) in the Transformer decoder. Mask2Former limits the attention calculation to the predicted mask area, and the calculation formula of the mask attention mechanism is as follows:

[0073]

[0074] Among them, X l is the mask attention matrix; Q l is the query vector in the self-attention mechanism; is the key vector in the self-attention mechanism; V l is the value vector in the self-attention mechanism. l-1 is the attention mask value of the (l-1)th layer at position (x, y), and the specific calculation formula is as follows:

[0075]

[0076] The terminal can calculate the mask loss function based on the predicted structural features and sample structural features of each sample image, and optimize the initial semantic segmentation model based on the mask loss function to obtain a preset semantic segmentation model. In Mask2Former, the mask loss function is calculated on K random sampling points, and the parameters of Mask2Former are optimized through the mask loss function to obtain a preset semantic segmentation model.

[0077] In this embodiment, the generalization ability of the semantic segmentation model is improved by expanding the range of sample images. At the same time, the mask loss function is calculated by using the predicted structural features and the sample structural features, and the initial semantic segmentation model is optimized to obtain a preset semantic segmentation model. By calculating the mask loss at K random sampling points instead of the entire mask, the training memory requirement is reduced by 3 times, from 18GB per image to 6GB.

[0078] In an exemplary embodiment, the image analysis method further includes:

[0079] Each sample image is labeled according to the preset knowledge graph to obtain the sample structure features of each sample image.

[0080] Specifically, the terminal can collect image data sets, analyze the images in the image data sets, and obtain the structural features that affect the evolution of the vessel type. For example, the images in the image data sets are bronze tripods, such as Figure 3 As shown, Figure 3 The schematic diagram of the components of a bronze tripod in one embodiment. The structures that affect the evolution of the bronze tripod may include the shape of the mouth, spout, handle, ears, feet, neck, abdomen, shoulders, waist, lid, top, door, bottom, handle, spout and other parts. The terminal can construct a preset knowledge graph based on the structures that affect the evolution of the bronze tripod. The terminal can obtain the preset knowledge graph corresponding to the sample image. The terminal can annotate the structure of the sample image according to the preset knowledge graph to obtain the sample structure features of each sample image.

[0081] In this embodiment, the sample structural features of the sample image are obtained from the preset knowledge graph, and the semantic segmentation model is optimized, thereby improving the accuracy of the semantic segmentation model in segmenting the image.

[0082] In an exemplary embodiment, the image analysis method further includes:

[0083] Arrange the structural feature vector groups in the order of their time features to obtain multiple structural feature vector groups; for the i-th structural feature vector group, calculate the cosine value of each structural feature vector and each structural feature vector in the i+1-th structural feature vector group in turn, and determine the cosine value as the similarity data between the i-th structural feature vector group and the i+1-th structural feature vector group.

[0084] Wherein, 1≤i<n, n is the number of structural feature vector groups. Each structural feature vector group includes structural feature vectors with consistent time features. The closer the cosine value is to 1, the more similar the two structural feature vectors are.

[0085] Specifically, the terminal can arrange the time features of each structure feature vector group in the order of their time features to obtain multiple structure feature vector groups. Exemplarily, the time feature is the era, and the structure feature vectors in the standard device vector database can be divided into three groups according to the era, namely, early, middle and late. In the early stage, there are 2 standards, named A1 and A1, in the middle stage, there are 2 standards, named B1 and B2, and in the late stage, there are 3 standards, named C1, C2 and C3. The terminal can calculate the similarity data of the early and mid-term standard devices: cos (A1, B1) = 0.8, cos (A1, B2) = 0.6, cos (A2, B1) = 0.5, cos (A2, B2) = 0.7; the terminal calculates the similarity data of the mid-term and late standard devices: cos (B1, C1) = 0.6, cos (B1, C2) = 0.7, cos (B1, C3) = 0.75, cos (B2, C1) = 0.5, cos (B2, C2) = 0.4, cos (B2, C3) = 0.8. It should be understood that the examples here are only for illustration and do not constitute specific limitations. The terminal can store the semantic similarity data in a semantic similarity database.

[0086] In this embodiment, by calculating the cosine value between each structural feature vector in two structural feature vector groups, similarity data is confirmed, which provides data support for the subsequent acquisition of the evolution path.

[0087] In an exemplary embodiment, the evolution path of the target object is obtained according to the dynamic path planning strategy and the similarity data of each structural feature vector group, including:

[0088] For each structural feature vector in the first structural feature vector group, determine the maximum similarity data of the structural feature vector; based on the maximum similarity data, determine the target structural feature vector of the second structural feature vector group, and determine the path between the structural feature vector in the first structural feature vector group and the target structural feature vector as the first evolutionary path; for each target structural feature vector in the i-th structural feature vector group, confirm the maximum similarity data of the target structural feature vector; based on the maximum similarity data, determine the target structural feature vector of the i+1-th structural feature vector group, and determine the path between the target structural feature vector in the i-th structural feature vector group and the target structural feature vector in the i+1-th structural feature vector group as the second evolutionary path; the terminal determines the evolutionary path of the target object based on the first evolutionary path and multiple second evolutionary paths.

[0089] Specifically, for each structural feature vector in the first structural feature vector group, the terminal sorts the similarity data between the structural feature vector and each structural feature vector in the second structural feature vector group, finds the maximum value of the similarity data between the structural feature vector and the structural feature vectors in the second structural feature vector group, obtains the maximum similarity data, determines the structural feature vector in the second structural feature vector group corresponding to the maximum similarity data as the target feature vector, and determines the path between the structural feature vector and the target structural feature vector as the first evolution path. The terminal sorts the similarity data between the target structural feature vector in the i-th structural feature vector group and each structural feature vector in the i+1-th structural feature vector group, finds the maximum value of the similarity data between the target structural feature vector and the structural feature vector in the i+1-th structural feature vector group, obtains the maximum similarity data, determines the structural feature vector in the i+1-th structural feature vector group corresponding to the maximum similarity data as the structural feature vector most similar to the target structural feature vector, and confirms the structural feature vector most similar to the target structural feature vector as the target feature vector in the i+1-th structural feature vector group, and determines the path between the target structural feature vector and the target structural feature vector in the i+1-th structural feature vector group as the second evolution path. 1<i<n, the terminal can generate multiple second evolution paths. The terminal can determine the evolution path of the target object based on the first evolution path and multiple second evolution paths.

[0090] For example, Figure 4 As stated, Figure 4This is a structural diagram of the evolutionary path in an embodiment. Each structural feature vector can correspond to the structural feature of a standard. The terminal can use the similarity data of two standards (structural feature vectors) between adjacent ages (structural feature vector groups) as the evolutionary path between the two standards, find the standard with the largest distance from each standard to the next age, and obtain the evolutionary path. There are two paths from the early bronze A1 to the middle bronze. The similarity data between A1 and B1 and A1 to B2 are sorted. The similarity data from A1 to B1 is 0.8, and the similarity data from A1 to B2 is 0.6. 0.8>0.6. 0.8 is confirmed as the maximum similarity data. B1 in the middle bronze is determined as the target structural feature vector, and the path from A1 to B1 is selected as the first evolutionary path of A1. Similarly, the first evolutionary path from A2 in the early stage to B2 in the middle stage can be found. The terminal considers from the middle stage to the late stage, and for B1 in the middle stage , sort the similarity data between B1 and C1, B1 and C1, B1 and C3, and obtain the largest similarity data between B1 and C3, confirm the similarity data between B1 and C3 as the maximum similarity data, confirm C3 as the target structure feature vector corresponding to B1, and determine the path between B1 and C1 as the second evolutionary path. Similarly, in the late stage, the second evolutionary path with B2 is determined to be from B2 to C3, that is, there should be two evolutionary paths from the early stage to the middle stage and then to the late stage: (A1, B1, C3) and (A2, B2, C3). It should be understood that the examples here are only for illustration and cannot constitute a specific limitation.

[0091] In an exemplary embodiment, the image analysis method further includes: displaying the evolution path.

[0092] Specifically, the terminal visualizes the evolution path, obtains a visualization result, and displays the visualization result. The terminal can also count the number of occurrences of the structural features in each structural feature vector group, and perform a visualization operation to display the visualization result. The terminal can visualize the constructed preset knowledge graph to display the original image and the cut image corresponding to the structural features of the image after semantic segmentation.

[0093] In this embodiment, data analysis results and the visualization between the original image and the cut image are realized through data visualization.

[0094] In an exemplary embodiment, Figure 5 As shown, Figure 5 This is a schematic diagram of an image analysis method in an embodiment, which can be divided into four stages: knowledge engineering stage, data processing stage, data analysis stage and data visualization stage. Specifically, the following steps are included:

[0095] Step 501: The terminal collects a large-scale image dataset.

[0096] Step 502: Determine the device type annotation features, select some images from the large-scale image data set, and determine the annotation features of the research object in the image with the participation of domain experts.

[0097] Step 503: The terminal constructs a preset knowledge graph based on the annotation features in the evolution law of the device type.

[0098] Step 504: The terminal may select some images as sample images, and perform feature annotation on the sample images based on a preset knowledge graph to obtain sample structural features.

[0099] Step 505: The terminal processes the sample image based on the initial semantic segmentation model to obtain preset structural features, calculates the preset structural features and the sample structural features to obtain a mask loss function, optimizes the initial segmentation model based on the mask loss function, and obtains a preset semantic segmentation model.

[0100] Step 506: semantically segment the images in the image set to be analyzed based on the preset semantic segmentation model to obtain structural features, and transform the structural features according to the preset knowledge concept map to obtain a structural feature vector. The structural feature vector is stored in the vector database.

[0101] Step 507: The terminal may perform vector deduplication processing on the structural feature vectors in the vector database to form a standard device database.

[0102] Step 508: The terminal can perform statistical analysis on the standard instrument database, etc. For example, in the field of cultural heritage, semantic similarity calculations can be performed between two standard instruments of different instrument types to obtain similarity data, and the similarity data is stored in the semantic similarity database.

[0103] Step 509: The terminal determines the evolution path of the target object according to the dynamic path planning strategy and the similarity data of each of the structural feature vector groups.

[0104] Step 510: The data results can be visualized, mainly consisting of two parts: the first part is the statistical analysis results based on image features, and the second part is the knowledge graph visualization between the original image and the cut image.

[0105] In this embodiment, structured data is produced by annotating image resources with fine-grained knowledge, and the structured data generated in the first stage is analyzed using statistical analysis methods of data science. The analysis results are visualized using data visualization methods, which can achieve efficient processing of large-scale images and realize an integrated image annotation system based on image data production, data analysis, and data visualization.

[0106] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0107] Based on the same inventive concept, the embodiment of the present application also provides an image analysis device for implementing the above-mentioned image analysis method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image analysis device embodiments provided below can refer to the limitations on the image analysis method above, and will not be repeated here.

[0108] In an exemplary embodiment, Figure 6 As shown, an image analysis device 60 is provided, comprising: a processing module 61, a calculation module 62 and a determination module 63, wherein:

[0109] A processing module 61 is used to process each image in the image set to be analyzed according to a preset knowledge graph in a preset semantic segmentation model to obtain a structural feature vector of each image, where the structural feature vector represents each structure of the target object in the image;

[0110] A calculation module 62, used for dividing the images based on the time features of each image to obtain a plurality of structural feature vector groups, and performing semantic similarity calculation on the structural feature vectors in each structural feature vector group to obtain similarity data of each structural feature vector group;

[0111] The determination module 63 is used to determine the evolution path of the target object according to the dynamic path planning strategy and the similarity data of each structure feature vector group.

[0112] In one embodiment, the processing module 61 is specifically used to perform segmentation processing on each image in the image set to be analyzed in a preset semantic segmentation model to obtain the structural features of each image;

[0113] The structural features are transformed based on the preset knowledge graph to obtain the structural feature vectors of each image.

[0114] In one embodiment, the processing module 61 is further used to obtain sample data, where the sample data includes each sample image and sample structure features corresponding to each sample image;

[0115] Segment each sample image based on the initial semantic segmentation model to obtain the predicted structural features corresponding to each sample image;

[0116] Based on the predicted structural features and sample structural features of each sample image, the mask loss function is calculated, and the initial semantic segmentation model is optimized based on the mask loss function to obtain a preset semantic segmentation model.

[0117] In one embodiment, the processing module 61 is further used to label each sample image according to a preset knowledge graph to obtain sample structure features of each sample image.

[0118] In one embodiment, the calculation module 62 is further used to arrange the time features of each structure feature vector group in order to obtain multiple structure feature vector groups;

[0119] For the i-th structural feature vector group, the cosine value of each structural feature vector and each structural feature vector in the i+1-th structural feature vector group is calculated in turn, and the cosine value is determined as the similarity data between the i-th structural feature vector group and the i+1-th structural feature vector group, 1≤i<n, n is the number of structural feature vector groups.

[0120] In one embodiment, the determination module 63 is used to determine the maximum similarity data of the structural feature vector for each structural feature vector in the first structural feature vector group; determine the target structural feature vector of the second structural feature vector group according to the maximum similarity data, and determine the path between the structural feature vector in the first structural feature vector group and the target structural feature vector as the first evolution path;

[0121] For each target structural feature vector in the i-th structural feature vector group, confirm the maximum similarity data of the target structural feature vector; determine the target structural feature vector of the i+1-th structural feature vector group according to the maximum similarity data, and determine the path between the target structural feature vector in the i-th structural feature vector group and the target structural feature vector in the i+1-th structural feature vector group as the second evolution path;

[0122] Based on the first evolution path and the second evolution path, an evolution path of the target object is determined.

[0123] In one embodiment, the determination module 63 is further configured to display the evolution path.

[0124] Each module in the above-mentioned image analysis device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0125] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, an image analysis method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0126] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0128] In the preset semantic segmentation model, each image in the image set to be analyzed is processed according to the preset knowledge graph to obtain a structural feature vector of each image, where the structural feature vector represents each structure of the target object in the image;

[0129] Based on the time features of each image, the image is divided to obtain a plurality of structural feature vector groups, and the semantic similarity of the structural feature vectors in each structural feature vector group is calculated to obtain similarity data of each structural feature vector group;

[0130] According to the dynamic path planning strategy and the similarity data of each structural feature vector group, the evolution path of the target object is determined.

[0131] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0132] In the preset semantic segmentation model, each image in the image set to be analyzed is processed according to the preset knowledge graph to obtain a structural feature vector of each image, where the structural feature vector represents each structure of the target object in the image;

[0133] Based on the time features of each image, the image is divided to obtain a plurality of structural feature vector groups, and the semantic similarity of the structural feature vectors in each structural feature vector group is calculated to obtain similarity data of each structural feature vector group;

[0134] According to the dynamic path planning strategy and the similarity data of each structural feature vector group, the evolution path of the target object is determined.

[0135] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0136] In the preset semantic segmentation model, each image in the image set to be analyzed is processed according to the preset knowledge graph to obtain a structural feature vector of each image, where the structural feature vector represents each structure of the target object in the image;

[0137] Based on the time features of each image, the image is divided to obtain a plurality of structural feature vector groups, and the semantic similarity of the structural feature vectors in each structural feature vector group is calculated to obtain similarity data of each structural feature vector group;

[0138] According to the dynamic path planning strategy and the similarity data of each structural feature vector group, the evolution path of the target object is determined.

[0139] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0140] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0141] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image analysis method, characterized in that: The method comprises: In a preset semantic segmentation model, each image in the image set to be analyzed is processed according to a preset knowledge graph to obtain a structural feature vector of each image, wherein the structural feature vector represents each structure of the target object in the image; Dividing the images based on their temporal features to obtain a plurality of structural feature vector groups, and calculating semantic similarity of the structural feature vectors in each of the structural feature vector groups to obtain similarity data of each of the structural feature vector groups; The evolution path of the target object is determined according to a dynamic path planning strategy and the similarity data of each of the structural feature vector groups.

2. The method according to claim 1, characterized in that In the preset semantic segmentation model, each image in the image set to be analyzed is processed according to the preset knowledge graph to obtain a structural feature vector of each image, including: In the preset semantic segmentation model, each image in the image set to be analyzed is segmented to obtain the structural features of each image; The structural features are converted based on a preset knowledge graph to obtain a structural feature vector of each of the images.

3. The method according to claim 1, characterized in that The method further comprises: Acquire sample data, wherein the sample data includes each sample image and sample structure features corresponding to each sample image; Performing segmentation processing on each of the sample images based on the initial semantic segmentation model to obtain predicted structural features corresponding to each of the sample images; Based on the predicted structural features of each of the sample images and the sample structural features, a mask loss function is calculated, and based on the mask loss function, the initial semantic segmentation model is optimized to obtain the preset semantic segmentation model.

4. The method according to claim 3, characterized in that The method further comprises: Each of the sample images is labeled according to the preset knowledge graph to obtain sample structural features of each of the sample images.

5. The method according to claim 1, characterized in that The method further comprises: Arrange the time features of each of the structural feature vector groups in order to obtain a plurality of the structural feature vector groups; For the i-th structural feature vector group, the cosine values ​​of each structural feature vector and each structural feature vector in the i+1-th structural feature vector group are calculated in turn, and the cosine values ​​are determined as the similarity data between the i-th structural feature vector group and the i+1-th structural feature vector group, 1≤i<n, and n is the number of the structural feature vector groups.

6. The method according to claim 5, characterized in that The step of obtaining the evolution path of the target object according to the dynamic path planning strategy and the similarity data of each of the structural feature vector groups includes: For each structural feature vector in the first structural feature vector group, determine the maximum similarity data of the structural feature vector; determine the target structural feature vector of the second structural feature vector group according to the maximum similarity data, and determine the path between the structural feature vector in the first structural feature vector group and the target structural feature vector as the first evolution path; For each target structural feature vector in the i-th structural feature vector group, confirm the maximum similarity data of the target structural feature vector; determine the target structural feature vector of the i+1-th structural feature vector group according to the maximum similarity data, and determine the path between the target structural feature vector in the i-th structural feature vector group and the target structural feature vector of the i+1-th structural feature vector group as the second evolution path; Based on the first evolution path and the second evolution path, an evolution path of the target object is determined.

7. The method according to claim 1, characterized in that The method further comprises: The evolution path is displayed.

8. An image analysis device, characterized in that: The device comprises: A processing module, used to process each image in the image set to be analyzed according to a preset knowledge graph in a preset semantic segmentation model to obtain a structural feature vector of each image, wherein the structural feature vector represents each structure of the target object in the image; A calculation module, used for dividing the images based on the time features of each image to obtain a plurality of structural feature vector groups, and performing semantic similarity calculation on the structural feature vectors in each structural feature vector group to obtain similarity data of each structural feature vector group; The determination module is used to determine the evolution path of the target object according to the dynamic path planning strategy and the similarity data of each of the structural feature vector groups.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images