Visual feature descriptor generation method and system fusing semantic topological information

Abstract

The invention belongs to the technical field of computer vision and image processing, and relates to a visual feature descriptor generation method and system fusing semantic topological information, and the method comprises the following steps: generating an original visual vector based on each key feature point; for each key feature point, determining scene semantic affiliation of the key feature point to generate a structured semantic vector; a semantic modulation weight is generated by the gating generation network; generating a semantic enhanced visual vector; generating an initial joint descriptor by the dimension reduction fusion network; generating a semantic confidence scalar based on a deterministic score of an image segmentation model on which the generation of the structured semantic vector depends; and performing dynamic weighted summation on the initial joint descriptor and the simplified visual vector according to a semantic confidence scalar to generate an adaptive feature descriptor. According to the method, the problem that two feature points which are visually similar but belong to different object instances cannot be distinguished in a traditional mode is solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and image processing, and relates to a visual feature descriptor generation method and system fusing semantic topological information. BACKGROUND

[0002] In robot navigation, augmented reality and three-dimensional reconstruction applications, extracting and matching stable and high-discrimination visual features from images is the key to realizing environment perception and positioning. The real world is full of visually repetitive structures and similar textures, such as rows of identical doors in office buildings and identical boxes stacked in warehouses, which makes traditional feature points relying on local pixel appearance prone to confusion and false matching, which is known as perceptual aliasing. The current core challenge is how to make visual features get rid of pure pixel appearance dependence and give them the ability to understand scene structure and object identity, so as to realize robust and accurate feature matching in complex and large-scale environments.

[0003] In order to solve the feature extraction and matching problem, the industry generally adopts the method of local feature point detection and description algorithm based on image gradient or intensity change, such as classic algorithms such as SIFT, SURF and ORB, which obtain feature descriptors that are invariant to illumination, scale and rotation through carefully designed mathematical transformations. In recent years, with the development of deep learning, methods using convolutional neural networks to directly learn feature descriptors from image blocks have also become mainstream. These methods have shown better performance on standard datasets and can extract more abstract and robust visual patterns. Whether traditional methods or deep learning methods, the core idea is still around how to more effectively encode the visual appearance information of the local neighborhood.

[0004] Based on the above problems, the disadvantage of the traditional method is the semantic blindness in its essence, which cannot distinguish two visually similar but belonging to different object instances. The traditional method is difficult to distinguish, which will cause serious problems when object-level understanding or long-term stable tracking is needed. SUMMARY

[0005] In a first aspect, the application provides a visual feature descriptor generation method fusing semantic topological information, which adopts the following technical scheme: The visual feature descriptor generation method fusing semantic topological information comprises the following steps: S1, acquiring a to-be-processed image, identifying key feature points in the to-be-processed image, and generating an original visual vector based on each key feature point; S2, determining the scene semantic attribution of each key feature point to generate a structured semantic vector; S3, input the structured semantic vector to the gating generation network, and generate a semantic modulation weight by the gating generation network; S4, element-wise multiply the original visual vector and the semantic modulation weight to generate a semantic enhanced visual vector; S5, splice the semantic enhanced visual vector and the structured semantic vector, and input the spliced vector to the dimension reduction fusion network to generate an initial joint descriptor by the dimension reduction fusion network; S6, generate a semantic confidence scalar based on the certainty score of the image segmentation model relied on by the generated structured semantic vector; S7, perform dimension alignment processing on the original visual vector to generate a simplified visual vector, and dynamically weight and sum the initial joint descriptor and the simplified visual vector according to the semantic confidence scalar to generate an adaptive feature descriptor.

[0006] In a further aspect of the present application, the original visual vector is generated based on each key feature point, including the following steps: An image neighborhood block is cut from the image to be processed with the position of each key feature point as the center; Each image neighborhood block is input into a preset visual feature extraction network, and an original visual vector is output by the preset visual feature extraction network.

[0007] In a further aspect of the present application, the structured semantic vector is generated by determining the scene semantic attribution, including the following steps: An image segmentation model is run to process the image to be processed to determine the object category attribute and the unique object instance number to which each key feature point belongs; A three-dimensional bounding box of the object to which the key feature point belongs is obtained, and the normalized coordinates of the key feature point in the three-dimensional bounding box are calculated to obtain the local geometric position attribute; The object category attribute, the unique object instance number and the local geometric position attribute are combined and encoded to generate a structured semantic vector.

[0008] In a further aspect of the present application, the normalized coordinates of the key feature point in the three-dimensional bounding box are calculated to obtain the local geometric position attribute, including the following steps: The coordinates of the key feature point in the three-dimensional space are obtained; The minimum coordinate vertex and the maximum coordinate vertex of the three-dimensional bounding box are obtained; The relative position of the key feature point in the three-dimensional bounding box is calculated based on the three-dimensional space coordinates of the key feature point, the minimum coordinate vertex and the maximum coordinate vertex of the three-dimensional bounding box, as the local geometric position attribute.

[0009] In a further aspect of the present application, the semantic modulation weight is generated by the gating generation network, including the following steps: The gating generative network learns and outputs semantic modulation weights with the same dimension as the original visual vector based on the object category attributes, unique object instance numbers, and local geometric location attributes contained in the input structured semantic vector.

[0010] A further aspect of this invention generates semantically enhanced visual vectors, comprising the following steps: Each weight value in the semantic modulation weights is multiplied by the corresponding value in the original visual vector. Based on semantic identity, the original visual features are selectively enhanced or suppressed to generate semantically enhanced visual vectors.

[0011] A further aspect of this invention involves generating an initial joint descriptor using a dimensionality reduction fusion network, comprising the following steps: The semantically enhanced visual vector and the structured semantic vector are concatenated and spliced ​​along the vector dimension to form an intermediate vector; The intermediate vectors are processed by a dimensionality reduction and fusion network to generate the initial joint descriptor.

[0012] A further aspect of this invention involves generating a semantic confidence scalar, comprising the following steps: Evaluate the deterministic score of the image segmentation model in determining the object category attribute and unique object instance number of key feature points; The deterministic score is mapped to a preset continuous interval to generate a semantic confidence scalar.

[0013] A further aspect of the present invention generates adaptive feature descriptors, comprising the following steps: The semantic confidence scalar is used as the first weight, and the difference between 1 and the semantic confidence scalar is used as the second weight. Multiply the first weight by the initial joint descriptor, and multiply the second weight by the simplified visual vector; The two product vectors are added element by element to generate an adaptive feature descriptor.

[0014] Secondly, this invention provides a visual feature descriptor generation system that integrates semantic topological information, employing the following technical solution: A visual feature descriptor generation system that integrates semantic topological information includes the following modules: The original visual vector generation module acquires the image to be processed, identifies key feature points in the image, and generates an original visual vector based on each key feature point. The structured semantic vector generation module determines the scene semantic attribution for each key feature point in order to generate a structured semantic vector. The semantic modulation weight generation module takes structured semantic vectors as input and feeds them into a gated generation network, which then generates semantic modulation weights. The semantic enhanced visual vector generation module element-wise multiplies the original visual vector and the semantic modulation weight to generate a semantic enhanced visual vector; The initial joint descriptor generation module splices the semantic enhanced visual vector and the structured semantic vector, and inputs the spliced vector into the dimension reduction fusion network to generate an initial joint descriptor; The semantic confidence scalar generation module generates a semantic confidence scalar based on the certainty score of the image segmentation model relied on by the generated structured semantic vector; The adaptive feature descriptor generation module performs dimension alignment processing on the original visual vector to generate a simplified visual vector, and dynamically weights and sums the initial joint descriptor and the simplified visual vector according to the semantic confidence scalar to generate an adaptive feature descriptor.

[0015] In summary, the present application includes the following beneficial technical effects: 1. By deeply fusing the visual appearance information of the feature points with the semantic topological information such as the object category to which the feature points belong, the instance number and the relative position on the object, the discriminability of the feature descriptor is enhanced. The feature points that may be blurred or similar in vision become easy to distinguish because they have unique semantic identities. For example, two similar feature points, one on the handle of "door A" and the other on the handle of "door B", will generate completely different descriptors after processing by the method, thus fundamentally solving the perception confusion problem caused by repeated scene structure.

[0016] 2. An adaptive fusion mechanism based on semantic confidence is proposed, which enables the feature descriptor generation process to dynamically cope with changes in semantic information quality. When the certainty of image segmentation and instance recognition is high, the system will rely on the joint descriptor containing rich semantic information. Conversely, in the case of poor lighting, severe occlusion and other conditions that lead to unreliable semantic analysis, the system will automatically increase the proportion of original visual features. This intelligent adjustment mechanism ensures the robustness of the method, making it not only fully utilize the advantages of semantic information, but also effectively avoid performance degradation caused by false semantics, enhancing the ability to work stably in various complex and variable environments.

[0017] 3. The generated feature descriptors are embedded with rich semantics of the physical world, going beyond the traditional feature points as pure geometric markers, laying the foundation for building truly semantic maps and enabling higher-level human-robot interactions. Since each feature point knows which part of which object it is, this enables robots or intelligent systems to not only build a geometric model of the environment, but also to understand the classes and instances of various objects in the environment, thereby supporting higher-level instructions such as "go to the conference room door" or "pick up the cup on the table", pushing the boundaries from simple "where" localization to a combination of "what" and "where" scene understanding. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below, and the drawings are used to provide further understanding of the present application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0019] Fig. 1 The flowchart of the embodiments of the present application is disclosed.

[0020] Fig. 2 The structural schematic diagram of the embodiments of the present application is disclosed. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are a part of the embodiments of the present application, rather than all the embodiments of the present application. All other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0022] The following will be described in combination with the drawings of the embodiments of the present application. Figs. 1-2 The preferred detailed description of the present application is made.

[0023] The drawings of the embodiments of the present application will be described in combination with the drawings of the embodiments of the present application. Fig. 1 The present application proposes a visual feature descriptor generation method fusing semantic topological information, including the following steps: S1, acquiring a to-be-processed image, identifying key feature points in the to-be-processed image, and generating an original visual vector based on each key feature point; S2, for each key feature point, determining its scene semantic attribution to generate a structured semantic vector; S3, inputting the structured semantic vector to a gate generation network to generate a semantic modulation weight by the gate generation network; S4, element-wise multiplication is performed on the original visual vector and the semantic modulation weight to generate a semantic enhanced visual vector; S5, the semantic enhanced visual vector is spliced with the structured semantic vector, and the spliced vector is input into the dimension reduction fusion network to generate an initial joint descriptor; S6, a semantic confidence scalar is generated based on the certainty score of the image segmentation model relied on by the generated structured semantic vector; S7, the original visual vector is subjected to dimension alignment processing to generate a simplified visual vector, and the initial joint descriptor and the simplified visual vector are dynamically weighted and summed according to the semantic confidence scalar to generate an adaptive feature descriptor.

[0024] In one of the embodiments of the present application, step S1 comprises the following steps: Specifically, the key feature points are acquired and recognized, and the original visual vector containing visual appearance information is generated based on the local neighborhood of each key feature point in the to-be-processed image. A digital image file is generated by capturing a scene containing a to-be-detected target through a camera device, such as an industrial camera installed on an automated inspection robot, which is the to-be-processed image. A preset feature point detection algorithm program acts on the to-be-processed image, and the preset feature point detection algorithm program analyzes the brightness change gradient of the image pixel by pixel. When a significant and stable change in the gradient of a point in multiple directions is found, such as the corner of an object in the image or the center of a unique texture, the point is marked as a key feature point, and its two-dimensional coordinate position in the image is recorded.

[0025] For each identified key feature point, a square image region of a preset size is cropped from the to-be-processed image with the coordinates as the center, and this square region is the image neighborhood block. Each cropped image neighborhood block is transmitted to a preset visual feature extraction network as input data, and the network is a deep learning model trained by a large amount of general image data, which contains multiple calculation layers capable of analyzing image features layer by layer. After the forward calculation of the network, the network finally outputs a one-dimensional array of fixed length, and each value in the array represents a specific visual element extracted from the image neighborhood block, such as a quantized expression of texture, edge or color distribution. This one-dimensional array is the original visual vector that can represent the local visual appearance information of the image.

[0026] The to-be-processed image is a digital image file, such as a JPEG or PNG file, recording the color and brightness information of each pixel in the scene. The key feature point is a pixel in the image with high distinguishability, usually containing the x and y coordinate values of the point in the image coordinate system, and sometimes also including its main direction and scale information to ensure stability under rotation and scaling. The image neighborhood block is a small size rectangular image piece cut from the to-be-processed image, with a preset size, such as 32x32 pixels or 64x64 pixels. The size is set based on the need to balance between including sufficient context information and avoiding the introduction of too much irrelevant background, usually based on statistical analysis of the size of objects in the target scene. The preset visual feature extraction network is a pre-trained convolutional neural network without the final classification layer, such as the first half of the VGG16 or ResNet50 model, which functions to convert the input image neighborhood block into a numerical vector.

[0027] The original visual vector is a one-dimensional floating-point array output by the preset visual feature extraction network, such as 512 dimensions, determined by the structure of the selected network. The vector is designed to have compact and robust encoding ability for the visual content of the image neighborhood block.

[0028] For example, assuming that the to-be-processed image taken by the camera device is a 1920x1080 pixel photo of a factory workshop, the feature point detection algorithm identifies a key feature point on the screw of the conveyor belt in the image, with coordinates (850, 520). In implementation, a 64x64 pixel image neighborhood block is cut around the coordinates as the center, and this 64x64 pixel image neighborhood block containing the screw and the surrounding conveyor belt texture is input into the preset visual feature extraction network. After multiple layers of feature extraction and calculation inside the network, a 512-dimensional original visual vector is finally output, such as an array like [0.23, -0.61, 1.05,..., 0.88]. The numerical distribution of this array uniquely represents the visual appearance information of the location of the specific screw. Repeat this process for all other key feature points identified in the image, and each key feature point generates a corresponding original visual vector.

[0029] In one embodiment of the present application, step S2 includes the following steps: Specifically, for each previously determined key feature point, an image segmentation model is run, which is a computer program capable of analyzing the image to be processed and classifying each pixel into a specific object category. The program outputs a segmentation map after processing the image to be processed, in which different colors or labels are used to mark each independent object in the image, such as "conveyor belt", "screw" and the like. By querying the label corresponding to the key feature point coordinates in the segmentation map, the object category attribute to which the key feature point belongs can be determined, and a unique object instance number for distinguishing multiple objects of the same category in the scene can be obtained. The normalized coordinates of the key feature point within the three-dimensional bounding box of the object to which it belongs are calculated. The three-dimensional bounding box is the smallest virtual cuboid that can exactly enclose the entire object instance, which is usually provided by an image segmentation model or a three-dimensional reconstruction system.

[0030] By operating the three-dimensional coordinates of the key feature point in the scene with the minimum and maximum vertex coordinates of the three-dimensional bounding box of the object to which it belongs, a relative position coordinate ranging from 0 to 1 is obtained, which is the local geometric position attribute. The three groups of information obtained, namely the object category attribute, the object instance number and the local geometric position attribute, are combined and encoded. The encoding process converts the non-numerical object category attribute into a digital representation, such as using one-hot encoding, and then concatenates all the digital information into a fixed-length one-dimensional array. The final one-dimensional array is a structured semantic vector that digitally describes the physical identity of the key feature point.

[0031] The calculation of the output local geometric position attribute satisfies the formula: P_norm represents the output local geometric position attribute, which is a vector containing three components, and each component has a value between 0 and 1, representing the relative x, y, z position within the three-dimensional bounding box. P represents the coordinate vector of the key feature point in three-dimensional space, which is usually measured by a depth camera or a stereo matching algorithm. B_min represents the minimum coordinate vertex of the three-dimensional bounding box of the object to which the key feature point belongs, i.e. the coordinate vector of the corner point closest to the coordinate origin. B_max represents the maximum coordinate vertex of the same three-dimensional bounding box of the object, i.e. the coordinate vector of the corner point farthest from the coordinate origin. The setting of B_min and B_max is based on the definition of the range of the identified object instance by the image segmentation model.

[0032] wherein the scene semantic attribution refers to the identity information of the key feature point in the physical scene, which is determined by image segmentation and geometric positioning. The structured semantic vector is a one-dimensional numerical array that structurally encodes the discrete identity information of the key feature point, including which object category it belongs to, which specific instance it is, and its local position on the instance.

[0033] The image segmentation model is a deep learning network, and its function is to divide the pixels of an input image into different regions and assign a semantic label to each region, such as "table" or "chair". The object class attribute is a semantic label output from the image segmentation model, such as "conveyor belt" or "screw", which is text or enumeration type data. The object instance number is a unique numerical identifier, such as integer 1 or 2, used to distinguish multiple objects of the same class appearing in a scene, and the setting is based on the counting order of the image segmentation model when identifying objects.

[0034] The three-dimensional bounding box is a virtual cuboid composed of six faces, determined by two three-dimensional points of minimum coordinates and maximum coordinates, used to enclose the complete object instance in three-dimensional space. The local geometric position attribute is a vector composed of three floating-point numbers, representing the relative position of the key feature point within the three-dimensional bounding box of the object it belongs to, calculated according to the above normalization formula.

[0035] For example, for a key feature point with coordinates (850, 520) and the original visual vector generated for it, first run the image segmentation model, which identifies that the key feature point belongs to the object class attribute "conveyor belt" and assigns it a unique object instance number "1" because there is only one conveyor belt in the scene. At the same time, the model determines the three-dimensional bounding box of the conveyor belt, with the minimum coordinate vertex B_min being (700, 500, 100) and the maximum coordinate vertex B_max being (1200, 550, 110). Assuming that the three-dimensional coordinates P of the key feature point are measured by depth information as (850, 520, 102). According to the formula, the local geometric position attribute P_norm is calculated, with the x component being (850-700) / (1200-700)=0.3, the y component being (520-500) / (550-500)=0.4, and the z component being (102-100) / (110-100)=0.2.

[0036] The object class attribute "conveyor belt" is encoded as a vector [0, 1, 0,...], and the object instance number "1" is kept as a numerical value 1 and combined with the local geometric position attribute [0.3, 0.4, 0.2] to generate a structured semantic vector, such as a one-dimensional array of [0, 1, 0,..., 1, 0.3, 0.4, 0.2].

[0037] In one embodiment of the present application, step S3 includes the following steps: Specifically, the structured semantic vector generated for the key feature points is taken as input data and transmitted to a gating generation network, which is a neural network specially used for information conversion. After the neural network receives the structured semantic vector containing physical identity information such as object category, instance number and local position, it processes the information through the pre-trained calculation layers inside the network. This processing process is essentially learning the mapping relationship from physical identity to importance weight. For example, the neural network will determine which visual features are most critical for identifying a specific object according to the information that the input is a "screw on a conveyor belt".

[0038] The network calculates and outputs a one-dimensional weight array with the same dimension as the original visual vector. Each value in the array corresponds to the importance of the feature at the corresponding position in the original visual vector. The output weight array is the semantic modulation weight, which satisfies the formula: The formula represents the functional relationship of the gating generation network. W_sem represents the output semantic modulation weight, which is a one-dimensional vector. f_gate represents the nonlinear transformation function performed by the gating generation network, and the calculation method is determined by the structure and parameters inside the network. V_sem represents the input structured semantic vector. θ represents all learnable parameters inside the gating generation network, such as connection weights and bias terms between neurons. These parameters are obtained by training on a large amount of paired data, and the training is based on minimizing the prediction error of the final task.

[0039] Among them, the physical identity information is the descriptive information about object category, instance and position contained in the structured semantic vector, which is the basis for the gating generation network to make judgments. The semantic modulation weight is a one-dimensional floating-point vector with the same dimension as the original visual vector, such as 512 dimensions. Its function is to serve as a set of multipliers to adjust the strength of each feature in the original visual vector in subsequent steps. The gating generation network is a small fully connected neural network, which functions similarly to a programmable controller. It receives the structured semantic vector representing identity and generates weight signals for regulating other vectors accordingly. The structure of the network can be set to contain two hidden layers with 64 neurons each, and uses ReLU as the activation function. Finally, a linear output layer is used to generate semantic modulation weights with the same dimension as the original visual vector.

[0040] For example, the structured semantic vector generated for the screw on the conveyor belt, which is a one-dimensional array of [0, 1, 0,..., 1, 0.3, 0.4, 0.2], is input into the preset gating generation network. The network performs forward calculation based on the data containing the category of "conveyor belt", "instance 1", "local position (0.3, 0.4, 0.2)", and other physical identity information. Since the original visual vector is 512-dimensional, the gating generation network outputs a semantic modulation weight with the same dimension of 512. For example, the output semantic modulation weight is [1.2, 0.9, 1.5,..., 0.8]. The first value 1.2 in this vector means that the first feature of the original visual vector is important for identifying "screws on the conveyor belt" and should be enhanced, while the second value 0.9 means that the contribution of the second feature is relatively small and can be appropriately suppressed.

[0041] In one embodiment of the present application, step S4 includes the following steps: Specifically, the generated original visual vector and the semantic modulation weight are input, and the original visual appearance representation is reshaped by element-wise multiplication to generate a semantic enhanced visual vector. The system takes the two vectors generated for the same key feature point as input, i.e., the original visual vector representing the visual appearance and the semantic modulation weight representing the importance adjustment. Element-wise multiplication is performed, which pairs and multiplies the corresponding values in the two vectors. Specifically, the first value of the original visual vector is multiplied by the first value of the semantic modulation weight to obtain the first value of the new vector; the second value of the original visual vector is multiplied by the second value of the semantic modulation weight to obtain the second value of the new vector; and so on, until all elements of the two vectors are multiplied.

[0042] Since the values in the semantic modulation weight reflect the semantic importance of the corresponding visual features, this multiplication process actually selectively enhances or suppresses the original visual features, thereby generating a vector with the same dimension as the input vector, i.e., a semantic enhanced visual vector.

[0043] The calculation of the semantic enhanced visual vector satisfies the formula: ; Where V_enh represents the output semantic enhanced visual vector. V_orig represents the input original visual vector. W_sem represents the input semantic modulation weight. ⊙ represents element-wise multiplication, i.e., the corresponding elements of the two vectors are multiplied to generate a new vector. For example, if V_orig = [a, b] and W_sem = [c, d], then .

[0044] The semantic enhanced visual vector is a one-dimensional numerical vector, which has the same dimension as the original visual vector, and each element value of the vector is the result of multiplying the original visual feature value by the corresponding semantic modulation weight value, so it not only contains the visual appearance information of the object, but also integrates the judgment of the importance of the appearance information under the current semantics, so that it has stronger discriminability when used for identification and matching tasks.

[0045] For example, the original visual vector of the conveyor belt screw [0.23, -0.61, 1.05,..., 0.88] and the semantic modulation weight [1.2, 0.9, 1.5,..., 0.8] generated therefor are taken as inputs, an element-by-element multiplication operation is performed, the first element of the new vector is calculated to be 0.276, the second element is calculated to be -0.549, and the third element is calculated to be 1.575. The same operation is performed for all 512 elements until the last element is calculated to be 0.704. Finally, a 512-dimensional semantic enhanced visual vector [0.276, -0.549, 1.575,..., 0.704] is generated.

[0046] In one embodiment of the present application, step S5 includes the following steps: Specifically, the semantic enhanced visual vector generated for the same key feature point and the structured semantic vector are inputted and fused by concatenation to generate an initial joint descriptor. The system takes the semantic enhanced visual vector representing visual appearance and the structured semantic vector representing physical identity, performs a concatenation operation in the vector dimension, and the concatenation operation completely appends a one-dimensional array representing the structured semantic vector to the end of a one-dimensional array representing the semantic enhanced visual vector, thereby forming a new one-dimensional array containing two kinds of information and having a higher dimension. This higher-dimensional intermediate vector is taken as input and sent to a pre-set dimension reduction fusion network, which is a computer program composed of multiple calculation layers and is specially used for converting high-dimensional data into low-dimensional data and learning how to effectively combine information from different sources during the conversion process.

[0047] After the network processes the intermediate vector, a compact one-dimensional array with significantly reduced output dimension is outputted, which not only retains the remodeled visual appearance features, but also embeds the discrete identity logic of the key feature points. This output array is the initial joint descriptor, which satisfies the following formula:

[0048] where D init represents the initial joint descriptor of the output. f fusion represents the transformation function performed by the dimension reduction fusion network. V enh represents the input semantic enhanced visual vector. V sem represents the input structured semantic vector. The symbol [;] represents the concatenation operation of vectors, i.e. connecting the two vectors head to tail to form a long vector. φ represents all the learnable parameters inside the dimension reduction fusion network, which are adjusted during the training process to minimize the error of the final task.

[0049] where the initial joint descriptor is a one-dimensional numerical vector with compact dimension, which is obtained by fusing the semantic enhanced visual information and the structured semantic information. The intermediate vector is a temporary high-dimensional vector generated after the concatenation operation, and its data structure is to arrange the numerical values of the semantic enhanced visual vector and the structured semantic vector in order. The dimension reduction fusion network is a neural network aiming to reduce the data dimension and fuse multiple information sources, for example, a multilayer perceptron, which can include two fully connected hidden layers and an output layer. The number of neurons in the output layer determines the final dimension of the initial joint descriptor, for example, 256 dimensions. The setting of this dimension is based on the trade-off between the simplification and discrimination of feature representation in the application scenario.

[0050] For example, the 512-dimensional semantic enhanced visual vector [0.276, -0.549, 1.575,..., 0.704] of the conveyor screw and the structured semantic vector (assuming 24 dimensions) [0, 1, 0,..., 1, 0.3, 0.4, 0.2] are input. First, perform concatenation to generate a 512 plus 24 equal to 536-dimensional intermediate vector, the content of which is [0.276,..., 0.704, 0, 1, 0,..., 1, 0.3, 0.4, 0.2], and input this 536-dimensional intermediate vector into the preset dimension reduction fusion network. After calculation, the output dimension is compressed to a 256-dimensional initial joint descriptor, for example, an array such as [-0.98, 0.15,..., 0.44]. This 256-dimensional initial joint descriptor now carries the appearance information of the screw and its identity information in the scene in a highly condensed and fused form.

[0051] In one embodiment of the present application, step S6 includes the following steps: Specifically, the system traces back the original information relied on by the structured semantic vector, i.e. the judgment made by the image segmentation model and the instance recognition process in step S2. When performing pixel classification and object recognition, these models not only output the final class label and instance number, but usually also attach a score quantifying the certainty of their prediction. First, the certainty score is retrieved, which directly reflects the credibility of the model's judgment that "this key feature point belongs to a certain object", for example, a higher score indicates that the model is very confident about its prediction.

[0052] The system maps the obtained certainty score to the standard continuous interval of 0 to 1. If the certainty score itself is already a probability value between 0 and 1, then the value is directly used. The final value in the range of 0 to 1, obtained through mapping or direct adoption, is the semantic confidence scalar used to represent the reliability of the current physical identity information.

[0053] The semantic confidence scalar is a weight factor between 0 and 1, used to measure how much semantic information should affect the final feature descriptor. A value close to 1 indicates that the semantic information is highly reliable, while a value close to 0 indicates that the semantic information is unreliable. The certainty score is a value generated by the image segmentation model or instance recognition process when outputting the prediction result, quantifying the model's confidence in its own prediction. The value of this score is based on the model's internal probability calculation logic, for example, in multi-class classification, the activation value of the output layer corresponding to the selected class.

[0054] For example, the image segmentation model identifies the key feature point with coordinates (850, 520) as part of a "conveyor belt", and in making this judgment, the model internally calculates a certainty score of 0.95 for its prediction. First, the certainty score of 0.95 is obtained, and since the value 0.95 is already within the continuous interval of 0 to 1, the mapping process can adopt this value. The system generates a semantic confidence scalar of 0.95 for this key feature point, indicating that the physical identity information of this key feature point is highly reliable.

[0055] In one embodiment of the present application, step S7 includes the following steps: Specifically, the system acquires the initial joint descriptor containing the fused information and the semantic confidence scalar representing the reliability of the semantic information generated previously, and also retrieves the original visual vector generated at the beginning of the key feature point, and performs dimension alignment processing on it. Since the dimension of the original visual vector may be different from that of the initial joint descriptor, the dimension alignment processing adjusts the dimension of the original visual vector to be consistent with the initial joint descriptor through a linear transformation layer to form a simplified visual vector. The system takes the semantic confidence scalar as the first weight, and calculates the difference between 1 and the scalar, and takes this difference as the second weight.

[0056] A weighted sum operation is performed, which multiplies the first weight with each value in the initial joint descriptor, and multiplies the second weight with each value in the simplified visual vector. The two multiplied vectors are added by position, i.e. the first element of the new vector is the sum of the first elements of the two product vectors, the second element is the sum of the second elements, and so on, to generate a vector with the same dimension as the initial joint descriptor. This vector is the adaptive feature descriptor that automatically adjusts its discriminative ability according to the reliability of the semantic information, which satisfies the following formula:

[0057] where D_adapt represents the final output adaptive feature descriptor. S_sem represents the input semantic confidence scalar, which takes a value between 0 and 1 and acts as a weight. D_init represents the input initial joint descriptor. V_simp represents the simplified visual vector after dimension alignment processing.

[0058] where the adaptive feature descriptor is a one-dimensional numerical vector, and its content can dynamically switch between pure visual features and semantic-enhanced joint features according to the reliability of the semantic information. The simplified visual vector is a vector obtained by processing the original visual vector through a linear transformation layer, and its dimension is consistent with that of the initial joint descriptor, which is convenient for subsequent weighted sum operation. The parameters of the linear transformation layer are learned during the model training stage, and the setting basis is to make the reduced vector still retain the original visual information to the greatest extent. The first weight is the semantic confidence scalar, and the second weight is obtained by calculating 1 minus the value of the semantic confidence scalar. The sum of the two is always 1, which ensures the robustness of the weighted fusion.

[0059] For example, the system obtains the 256-dimensional initial joint descriptor [-0.98, 0.15, …, 0.44] generated for the conveyor screw, and the semantic confidence scalar 0.95 calculated therefor, retrieves the 512-dimensional original visual vector [0.23, -0.61, 1.05, …, 0.88] thereof, and processes it into a 256-dimensional simplified visual vector, for example, [0.18, -0.45, …, 0.62], through a linear transformation layer. The semantic confidence scalar 0.95 is taken as the first weight, and the second weight is calculated as 1-0.95, i.e. 0.05. The first weight 0.95 is multiplied with the initial joint descriptor to obtain [-0.931, 0.1425, …, 0.418]. The second weight 0.05 is multiplied with the simplified visual vector to obtain [0.009, -0.0225, …, 0.031]. The two resultant vectors are added element by element to generate the final 256-dimensional adaptive feature descriptor. Since the semantic confidence scalar is close to 1, the final adaptive feature descriptor will be mainly determined by the initial joint descriptor, and highly dependent on reliable semantic information; if the semantic confidence scalar is close to 0, the final result will be more biased towards the simplified visual vector containing only visual information.

[0060] Referring to the accompanying drawings Fig. 2 The present application also proposes a visual feature descriptor generation system fusing semantic topological information, comprising the following modules: An original visual vector generation module obtains a to-be-processed image, identifies key feature points in the to-be-processed image, and generates an original visual vector based on each key feature point; A structured semantic vector generation module determines the scene semantic attribution of each key feature point to generate a structured semantic vector; A semantic modulation weight generation module inputs the structured semantic vector into a gate generation network to generate a semantic modulation weight by the gate generation network; A semantic-enhanced visual vector generation module element-wise multiplies the original visual vector and the semantic modulation weight to generate a semantic-enhanced visual vector; An initial joint descriptor generation module splices the semantic-enhanced visual vector and the structured semantic vector, and inputs the spliced vector into a dimension reduction fusion network to generate an initial joint descriptor by the dimension reduction fusion network; A semantic confidence scalar generation module generates a semantic confidence scalar based on the certainty score of an image segmentation model relied on by the generated structured semantic vector; An adaptive feature descriptor generation module performs dimension alignment processing on the original visual vector to generate a simplified visual vector, and dynamically weights and sums the initial joint descriptor and the simplified visual vector according to the semantic confidence scalar to generate an adaptive feature descriptor The various modules can be implemented by software, hardware and a combination thereof in whole or in part, and the hardware is supported in the form of being embedded in or independent of a processor in a computer device, and the software is supported in the form of being stored in a memory in the computer device, so as to facilitate the processor to call and execute the operations corresponding to the various modules.

[0061] It should be noted that the human information (including but not limited to human device information and personal information, etc.) and data (including but not limited to data for analysis, stored data and displayed data, etc.) involved in the present application are information and data authorized by the human body or fully authorized by all parties, and the collection, use and processing of the relevant data require relevant legal standards.

[0062] The above examples are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application is described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for generating visual feature descriptors that integrates semantic topological information, characterized in that, Includes the following steps: S1. Obtain the image to be processed, identify key feature points in the image to be processed, and generate an original visual vector based on each key feature point. S2. For each key feature point, determine its scene semantic attribution to generate a structured semantic vector; S3. Input the structured semantic vector into the gated generation network, and the gated generation network generates semantic modulation weights; S4. Multiply the original visual vector and the semantic modulation weights element by element to generate a semantically enhanced visual vector. S5. Concatenate the semantically enhanced visual vector with the structured semantic vector, and input the concatenated vector into the dimensionality reduction fusion network to generate the initial joint descriptor. S6. Based on the deterministic score of the image segmentation model on which the generated structured semantic vector depends, generate a semantic confidence scalar; S7. Perform dimension alignment on the original visual vector to generate a simplified visual vector, and dynamically weight and sum the initial joint descriptor and the simplified visual vector according to the semantic confidence scalar to generate an adaptive feature descriptor.

2. The visual feature descriptor generation method integrating semantic topological information according to claim 1, characterized in that, Generating the original visual vector based on each key feature point includes the following steps: Using the location of each key feature point as the center, extract the image neighborhood block from the image to be processed; Each image neighborhood block is input into a preset visual feature extraction network, which then outputs the original visual vector.

3. The visual feature descriptor generation method incorporating semantic topological information according to claim 1, characterized in that, Determining the scene semantic attribution to generate structured semantic vectors includes the following steps: Run the image segmentation model to process the image to determine the object category attribute and unique object instance number of each key feature point; Obtain the 3D bounding box of the object to which the key feature points belong, and calculate the normalized coordinates of the key feature points within the 3D bounding box to obtain their local geometric position attributes. The object category attribute, unique object instance number, and local geometric location attribute are combined and encoded to generate a structured semantic vector.

4. The visual feature descriptor generation method integrating semantic topological information according to claim 3, characterized in that, Calculating the normalized coordinates of key feature points within the 3D bounding box to obtain their local geometric position attributes includes the following steps: Obtain the coordinates of key feature points in three-dimensional space; Obtain the minimum and maximum coordinate vertices of the 3D bounding box; Based on the 3D spatial coordinates of the key feature points and the minimum and maximum coordinate vertices of the 3D bounding box, the relative positions of the key feature points within the 3D bounding box are calculated and used as local geometric position attributes.

5. The method for generating visual feature descriptors that integrates semantic topological information according to claim 1, characterized in that, The semantic modulation weights are generated by a gating generative network, including the following steps: The gating generative network learns and outputs semantic modulation weights with the same dimension as the original visual vector based on the object category attributes, unique object instance numbers, and local geometric location attributes contained in the input structured semantic vector.

6. The method for generating visual feature descriptors that integrates semantic topological information according to claim 1, characterized in that, Generating semantically enhanced visual vectors includes the following steps: Each weight value in the semantic modulation weights is multiplied by the corresponding value in the original visual vector. Based on semantic identity, the original visual features are selectively enhanced or suppressed to generate semantically enhanced visual vectors.

7. The method for generating visual feature descriptors that integrates semantic topological information according to claim 1, characterized in that, The initial joint descriptor is generated from the dimensionality reduction fusion network, including the following steps: The semantically enhanced visual vector and the structured semantic vector are concatenated and spliced ​​along the vector dimension to form an intermediate vector; The intermediate vectors are processed by a dimensionality reduction and fusion network to generate the initial joint descriptor.

8. The method for generating visual feature descriptors that integrates semantic topological information according to claim 1, characterized in that, Generating a semantic confidence scalar includes the following steps: Evaluate the deterministic score of the image segmentation model in determining the object category attribute and unique object instance number of key feature points; The deterministic score is mapped to a preset continuous interval to generate a semantic confidence scalar.

9. The method for generating visual feature descriptors that integrates semantic topological information according to claim 1, characterized in that, Generating adaptive feature descriptors includes the following steps: The semantic confidence scalar is used as the first weight, and the difference between 1 and the semantic confidence scalar is used as the second weight. Multiply the first weight by the initial joint descriptor, and multiply the second weight by the simplified visual vector; The two product vectors are added element by element to generate an adaptive feature descriptor.

10. A visual feature descriptor generation system that integrates semantic topological information, characterized in that, Includes the following modules: The original visual vector generation module acquires the image to be processed, identifies key feature points in the image, and generates an original visual vector based on each key feature point. The structured semantic vector generation module determines the scene semantic attribution for each key feature point in order to generate a structured semantic vector. The semantic modulation weight generation module takes structured semantic vectors as input and feeds them into a gated generation network, which then generates semantic modulation weights. The semantically enhanced visual vector generation module multiplies the original visual vector with the semantic modulation weights element by element to generate a semantically enhanced visual vector. The initial joint descriptor generation module concatenates the semantically enhanced visual vector with the structured semantic vector, and inputs the concatenated vector into the dimensionality reduction fusion network, which generates the initial joint descriptor. The semantic confidence scalar generation module generates a semantic confidence scalar based on the deterministic score of the image segmentation model on which the generated structured semantic vector depends. The adaptive feature descriptor generation module performs dimension alignment on the original visual vector to generate a simplified visual vector, and dynamically weights and sums the initial joint descriptor and the simplified visual vector according to the semantic confidence scalar to generate an adaptive feature descriptor.

Images