Sole gluing area point cloud generation method based on RGBD camouflage instance segmentation

Abstract

The invention discloses a sole gluing area point cloud generation method based on RGBD camouflage instance segmentation. The method comprises the steps of obtaining a target sole image; constructing a four-mode tensor matrix corresponding to each pixel point on the target sole image; inputting the four-mode tensor matrix into a pre-trained sole segmentation model, and generating a binary classification probability of each pixel point through forward propagation; determining J target pixel points according to the dichotomy probability of each pixel point; according to the J target pixel points, generating a gluing area point cloud of the target shoe sole; according to the method, the orthogonalization pixel grid with direction consistency and structural symmetry is generated, so that the spatial relation around each pixel point is accurately captured; meanwhile, on the basis of completing pixel-level segmentation, three-dimensional reconstruction is only carried out on the pixel points which are judged to be the to-be-glued area, it is ensured that the actual gluing area is covered by point cloud, and introduction of non-target area points is avoided.

Description

Technical Field

[0001] The present invention relates to a method for generating a glue coating point cloud, in particular to a method for generating a point cloud of a shoe sole glue coating area based on RGBD camouflage instance segmentation. Background Art

[0002] During the sole manufacturing process, the automatic gluing system relies on accurate three-dimensional path information to achieve the process requirements of uniform glue lines and clear boundaries. The existing technology usually uses RGBD cameras to obtain color images and depth images of the sole, first back-projects the entire depth map to generate a complete three-dimensional point cloud of the sole, and then extracts the outline of the gluing area or fits the gluing path based on this point cloud. The document with publication number CN110226806B discloses a method and device for generating a gluing trajectory for soles, which can solve the problems in the existing technology of being unable to accurately extract the edge of the sole or the poor spraying effect and easy production of defective products when spraying the sole using a CAD model.

[0003] However, the prior art disclosed in the above-mentioned literature obtains a relatively complete geometric model during the point cloud acquisition process, but has obvious defects in practical applications: First, the generated point cloud contains information about the entire sole surface, resulting in large amounts of data and high redundancy. However, the actual area used for gluing only accounts for a small portion of the total surface area, resulting in heavy computational overhead and slow response times. Second, in scenes with similar textures, uneven lighting, or the presence of visual camouflage (e.g., blurred boundaries between dark gluing lines and dark soles), traditional segmentation methods struggle to accurately extract the gluing areas. Back-projecting the entire image introduces numerous erroneous or uncertain regions into the three-dimensional space, causing distortion. To address this issue, the present invention provides a method for generating point clouds of gluing areas on soles based on RGBD camouflage instance segmentation. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a method for generating a point cloud of the sole glued area based on RGBD camouflage instance segmentation. The method solves the technical problems raised by the background technology by first segmenting the target pixel points and then back-projecting the target pixel points to generate a point cloud of the glued area.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: The method for generating point cloud of the sole glued area based on RGBD camouflage instance segmentation includes: S1. Obtain a target sole image; wherein the target sole image includes RGB features and depth features of each pixel; S2, constructing a four-modal tensor matrix corresponding to each pixel point on the target sole image; Among them, the shape of the four-modal tensor matrix is ​​H×W, H and W are the number of pixels in the height direction and width direction of the four-modal tensor matrix respectively; S3. Input the four-modal tensor matrix into a pre-trained sole segmentation model, and generate a binary classification probability for each pixel through forward propagation; S4. Determine J target pixels based on the binary classification probability of each pixel; S5. Generate a point cloud of the glued area of ​​the target sole based on the J target pixel points.

[0006] In some specific embodiments, constructing a four-modal tensor matrix corresponding to each pixel point on the target sole image includes: S2-1, constructing UV coordinates for each pixel point on the target sole image; S2-2, defining a pixel grid for each pixel based on the UV coordinates constructed for each pixel; S2-3, extracting RGB features and depth features corresponding to the pixel grid on the target sole image; S2-4, concatenate the RGB features and depth features located in the same pixel grid to generate a four-modal tensor corresponding to the pixel point; S2-5, traverse each pixel point of the target sole image and repeat S2-1 to S2-4 until a four-modal tensor corresponding to H×W pixels is generated; S2-6. Arrange the H×W quadrimodal tensors based on the spatial positions of the UV coordinates to construct a quadrimodal tensor matrix.

[0007] In some specific embodiments, the UV coordinates constructed based on each pixel point define a pixel grid for each pixel point, including: S2-2-1. Anchor the UV coordinates of any pixel point as the center coordinates; S2-2-2. Taking the center coordinate as the starting point, determine the coordinates of its four neighboring regions in orthogonal directions; S2-2-3. Connect the coordinates of the four neighborhoods and the center coordinate to form four spatial lines; S2-2-4, locate the four midpoints of the four spatial lines; S2-2-5. Connect the four midpoints in a clockwise or counterclockwise order to form an initial grid centered on the central coordinate; S2-2-6. Rotate the initial grid 90 degrees around the center coordinate to generate an orthogonalized pixel grid; S2-2-7. Traverse the UV coordinates of each pixel point until the pixel grid of each pixel point is defined.

[0008] In some specific embodiments, the pre-training step of the sole segmentation model includes: A1. Obtain a set of camouflaged segmented images; A2. Input the camouflage segmentation image set into a convolutional neural network model for iterative supervised training to generate the sole segmentation model.

[0009] In some specific embodiments, obtaining the disguised segmented image set includes: A1-1. Obtain RGBD images of several sole samples; A1-2. Label each pixel on the RGBD images of several sole samples with a binary classification label; the binary classification label includes: points to be glued and points not to be glued; A1-3. Perform image enhancement on the RGBD images of several shoe sole samples annotated with binary classification labels to generate several RGBD enhanced images. A1-4. Mix the RGBD images of several shoe sole samples annotated with binary classification labels with several RGBD enhanced images to generate a camouflaged segmentation image set.

[0010] In some specific embodiments, generating the sole segmentation model includes: A2-1. Inputting the disguised segmented image set into a convolutional neural network model; A2-2. Extract the current batch of RGBD images from the camouflaged segmentation image set and generate the pixel prediction mask map of the current batch through forward propagation; A2-3. Calculate the cross entropy loss between the pixel prediction mask of the current batch and the binary classification labels corresponding to the RGBD image; A2-4. With the goal of minimizing the cross entropy loss, update the parameters of the convolutional neural network model and iteratively calculate the cross entropy loss of the next batch until the convolutional neural network model converges to the sole segmentation model.

[0011] In some specific embodiments, generating the binary classification probability of each pixel point through forward propagation includes: S3-1, using the UV coordinate as the index, anchor the target pixel in the four-modal tensor matrix; S3-2. Perform convolution extraction on the four-modal tensor of the target pixel and its neighboring pixels based on the pre-trained K convolution kernels to generate a convolution feature vector of the target pixel; S3-3, decoding the convolution feature vector of the target pixel point to obtain the binary classification probability that the target pixel point belongs to the glue-coated area; S3-4. Traverse the UV coordinates until the binary classification probability of each pixel point in the four-modal tensor matrix is ​​generated.

[0012] In some specific embodiments, determining J target pixels according to the binary classification probability of each pixel includes: S4-1, comparing the binary classification probability of each pixel in the four-modal tensor matrix with the set threshold; S4-2, if the binary classification probability is not less than the set threshold, then mark the pixel as a point to be glued, otherwise, mark it as a non-glue point; S4-3, traverse the four-modal tensor matrix and repeat S4-1 to S4-2 until a pixel mask corresponding to H×W pixels is obtained; S4-4, in the pixel mask image, marking the pixel corresponding to the point to be glued as the target pixel; S4-5. Traverse the pixel mask map until J target pixels are determined.

[0013] In some specific embodiments, generating a point cloud of the glued area of ​​the target sole according to J target pixel points includes: S5-1, extracting the UV coordinates of the target pixel point, and determining the pixel grid where the target pixel point is located based on the UV coordinates; S5-2, extracting the depth feature of the target pixel point based on the pixel grid where the target pixel point is located; S5-3, back-projecting the UV coordinates and depth features of the target pixel point to calculate the point cloud coordinates of the target pixel point; S5-4, traverse J target pixel points and repeat S5-1 to S5-3 until a glue-coated area point cloud consisting of J point cloud coordinates is generated.

[0014] The present invention provides a method for generating point clouds of the sole glue coating area based on RGBD camouflage instance segmentation, which has the following beneficial effects: The present invention determines the center coordinates and the coordinates of its four neighbors, connects these coordinates to form a spatial line, further locates the midpoint of the line to construct an initial grid, and then generates an orthogonalized pixel grid with directional consistency and structural symmetry through rotation, thereby ensuring that the spatial relationship around each pixel is accurately captured. Furthermore, the present invention extracts the UV coordinates of the target pixel points and combines them with their corresponding depth features, uses the camera intrinsic parameter matrix for back projection, calculates the three-dimensional coordinates of each target pixel point in the camera coordinate system, and generates a point cloud of the glued area. On the basis of completing pixel-level segmentation, three-dimensional reconstruction is performed only on the pixel points determined to be the area to be glued, ensuring that the point cloud covers the actual glued area and avoiding the introduction of non-target area points. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the process of generating point cloud of the sole glue coating area based on RGBD camouflage instance segmentation of the present invention; Figure 2 Schematic diagram of the construction process of the four-modal tensor matrix of the present invention; Figure 3 Schematic diagram of the process of defining the pixel grid according to the present invention; Figure 4 Schematic diagram of the target pixel determination process of the present invention; Figure 5 Schematic diagram of the process of generating the point cloud of the glue-coated area according to the present invention. DETAILED DESCRIPTION

[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0017] Example 1: Please refer to Figures 1 to 5 The present invention provides a method for generating a point cloud of a sole glue coating area based on RGBD camouflage instance segmentation, comprising the following steps: S1. Obtain a target sole image; wherein the target sole image includes RGB features and depth features of each pixel; Specifically, the target sole image is an RGBD image of the target sole captured by an RGBD camera. That is, for each pixel in the target sole image, in addition to recording the intensity features of the red (R), green (G), and blue (B) color channels, it also records the actual distance of the pixel relative to the camera, i.e., the depth feature.

[0018] S2, constructing a four-modal tensor matrix corresponding to each pixel point on the target sole image; Among them, the shape of the four-modal tensor matrix is ​​H×W, H and W are the number of pixels in the height direction and width direction of the four-modal tensor matrix respectively; S3. Input the four-modal tensor matrix into a pre-trained sole segmentation model, and generate a binary classification probability for each pixel through forward propagation; S4. Determine J target pixels based on the binary classification probability of each pixel; S5. Generate a point cloud of the glued area of ​​the target sole based on the J target pixel points.

[0019] In this embodiment, a method for generating a point cloud of the sole glued area based on RGBD camouflage instance segmentation is used. By acquiring a target sole image containing RGB features and depth features, a four-modal tensor matrix corresponding to each pixel is constructed, and the matrix is ​​input into a pre-trained sole segmentation model to generate a pixel-level binary classification probability. After J target pixels are screened out according to a set threshold, the three-dimensional point cloud coordinates are calculated, and finally the corresponding glued area point cloud is generated. This embodiment is based on the pixel-level classification results and only performs three-dimensional reconstruction on the pixel points determined to be the area to be glued, ensuring that the generated glued area point cloud is consistent with the actual glued area of ​​the target sole. At the same time, since the RGB features and depth features in the four-modal tensor maintain spatial alignment at the pixel level, the generated point cloud coordinates can truly restore the geometric distribution of the glue-coated area in three-dimensional space, ensuring the geometric shape of the point cloud in the glue-coated area, providing a clearly structured geometric coordinate input for glue-coating based on the sole point cloud, and improving the efficiency of automated glue-coating.

[0020] Exemplarily, step S2 of the point cloud generation method specifically includes: S2-1, constructing UV coordinates for each pixel point on the target sole image; Specifically, the UV coordinates consist of the U coordinate and the V coordinate. The U coordinate represents the position of the pixel in the width direction of the image, that is, the horizontal distance from the left edge of the image to the corresponding pixel. The V coordinate represents the position of the pixel in the height direction of the image, that is, the vertical distance from the top edge of the image to the corresponding pixel.

[0021] Therefore, the UV coordinates represent the position of each pixel in the two-dimensional image coordinate system. The U coordinate represents the horizontal position of the pixel (i.e., the column number), and the V coordinate represents the vertical position of the pixel (i.e., the row number). Through the UV coordinates, the position of each pixel in the target sole image can be uniquely determined.

[0022] S2-2, defining a pixel grid for each pixel based on the UV coordinates constructed for each pixel; S2-3, extracting RGB features and depth features corresponding to the pixel grid on the target sole image; Specifically, for each pixel grid, the RGB feature represents the intensity values ​​of the red, green, and blue channels obtained from the RGB data of the target sole image. These represent the intensity of the corresponding pixel on the red, green, and blue spectra, respectively. Typically, the value of each color ranges from 0 to 255. At the same time, based on the pixel grid, the depth of the corresponding pixel is obtained from the depth data of the target sole image, that is, the actual distance of the corresponding pixel relative to the camera.

[0023] S2-4, concatenate the RGB features and depth features located in the same pixel grid to generate a four-modal tensor corresponding to the pixel point; S2-5, traverse each pixel point of the target sole image and repeat S2-1 to S2-4 until a four-modal tensor corresponding to H×W pixels is generated; S2-6. Arrange the H×W quadrimodal tensors based on the spatial positions of the UV coordinates to construct a quadrimodal tensor matrix.

[0024] In this embodiment, a structured expression of the spatial position of the image is achieved by establishing UV coordinates for each pixel and constructing a pixel grid. On this basis, the RGB features and depth features of each pixel are simultaneously extracted and spliced ​​into a four-modal tensor. Finally, a complete four-modal tensor matrix is ​​constructed according to the spatial arrangement relationship of the UV coordinates.

[0025] This matrix aligns and splices the RGB features (color information) and the corresponding depth features at the pixel level, and organizes them in an orderly manner according to their spatial position in the image to form a multimodal image data structure with spatial consistency, which is suitable for the segmentation of the sole glue area described in this embodiment.

[0026] Furthermore, the step S2-2 further includes: S2-2-1. Anchor the UV coordinates of any pixel point as the center coordinates; S2-2-2. Taking the center coordinate as the starting point, determine the coordinates of its four neighboring regions in orthogonal directions; Specifically, the four neighborhood coordinates represent the UV coordinates of four neighboring pixel points in orthogonal directions (horizontal and vertical directions) of the center coordinate; and the neighboring pixel points represent the directly adjacent pixel points above, below, left and right of any pixel point.

[0027] S2-2-3. Connect the coordinates of the four neighborhoods and the center coordinate to form four spatial lines; S2-2-4, locate the four midpoints of the four spatial lines; S2-2-5. Connect the four midpoints in a clockwise or counterclockwise order to form an initial grid centered on the central coordinate; S2-2-6. Rotate the initial grid 90 degrees around the center coordinate to generate an orthogonalized pixel grid; S2-2-7. Traverse the UV coordinates of each pixel point until the pixel grid of each pixel point is defined.

[0028] In this embodiment, by taking the UV coordinates of each pixel point as the center coordinate, the four neighborhood coordinates in the horizontal and vertical directions are determined, and the center coordinates and the neighborhood coordinates are connected to form four spatial lines; the midpoints of each line are further located and connected in sequence to generate an initial grid, which is then rotated 90 degrees to obtain an orthogonalized pixel grid. This embodiment constructs a pixel grid with directional consistency and structural symmetry for each pixel point. By traversing all pixel points, a systematic definition of the pixel grid in the entire image is achieved, providing a unified spatial reference for the alignment and splicing of RGB features and depth features.

[0029] Exemplarily, step S3 of the point cloud generation method specifically includes: S3-1, using the UV coordinate as the index, anchor the target pixel in the four-modal tensor matrix; S3-2. Perform convolution extraction on the four-modal tensor of the target pixel and its neighboring pixels based on the pre-trained K convolution kernels to generate a convolution feature vector of the target pixel; Specifically, the convolution feature vector represents a K-dimensional feature vector composed of K features generated at the UV coordinate position by performing local weighted summation on the four-modal tensor of the target pixel and its neighborhood through K pre-trained convolution kernels; wherein each feature corresponds to the pattern response intensity of a convolution kernel to the local area, and the feature vector integrates the semantic information of the target pixel in RGB intensity and depth.

[0030] S3-3, decoding the convolution feature vector of the target pixel point to obtain the binary classification probability that the target pixel point belongs to the glue-coated area; Specifically, the decoding means mapping the K-dimensional convolution feature vector to a two-dimensional space through a fully connected layer, and then normalizing it through a Sigmoid activation function to output the probability value of the target pixel point belonging to the category of the area to be glued, with a value range of [0,1].

[0031] S3-4. Traverse the UV coordinates until the binary classification probability of each pixel point in the four-modal tensor matrix is ​​generated.

[0032] Specifically, the traversal of the UV coordinates refers to taking the upper left corner of the image as the starting point, and executing steps S3-1 to S3-3 for each UV coordinate position in sequence in row priority order, that is, anchoring, extracting convolution features and vector decoding pixel by pixel, and finally generating pixel-level binary classification probability.

[0033] Exemplarily, step S4 of the point cloud generation method specifically includes: S4-1, comparing the binary classification probability of each pixel in the four-modal tensor matrix with the set threshold; S4-2, if the binary classification probability is not less than the set threshold, then mark the pixel as a point to be glued, otherwise, mark it as a non-glue point; S4-3, traverse the four-modal tensor matrix and repeat S4-1 to S4-2 until a pixel mask corresponding to H×W pixels is obtained; Specifically, the pixel mask image is an H×W matrix, in which each element corresponds to a pixel position in the original image, and the value in the matrix is ​​usually 0 or 1.

[0034] S4-4, in the pixel mask image, marking the pixel corresponding to the point to be glued as the target pixel; S4-5. Traverse the pixel mask map until J target pixels are determined.

[0035] In this embodiment, a pixel mask is generated by comparing the binary classification probability with a set threshold, and J target pixels are then identified. These target pixels accurately identify the areas in the sole image that require gluing. Specifically, the selection of target pixels ensures that only those pixels identified as areas to be glued are retained, eliminating interference from non-glue-coated areas and preventing accidental or missed gluing.

[0036] Because this method relies on pixel-level classification, the resulting target pixels closely match the actual gluing boundaries, accurately reflecting gluing requirements even under complex textures or depth variations. This is particularly true for sole surfaces with similar colors, uneven lighting, or visual camouflage, allowing for stable extraction of valid gluing areas.

[0037] Exemplarily, step S5 of the point cloud generation method specifically includes: S5-1, extracting the UV coordinates of the target pixel point, and determining the pixel grid where the target pixel point is located based on the UV coordinates; S5-2, extracting the depth feature of the target pixel point based on the pixel grid where the target pixel point is located; S5-3, back-projecting the UV coordinates and depth features of the target pixel point to calculate the point cloud coordinates of the target pixel point; Specifically, back projection means mapping the target pixel point (with depth features) in the two-dimensional image plane back to the position in the three-dimensional space, thereby realizing the conversion from the UV coordinate of the target pixel point to the point cloud coordinate.

[0038] First, determine the UV coordinates of the target pixel point and read the depth feature value (Depth) corresponding to the target pixel point from the depth image.

[0039] Then, the camera's intrinsic matrix is ​​used to convert the pixel coordinates (UV coordinates) and depth eigenvalues ​​into three-dimensional coordinates in the camera coordinate system. The camera's intrinsic matrix contains information about the camera's internal parameters, such as focal length, principal point position, etc., and is used to convert two-dimensional image coordinates into three-dimensional space coordinates. For example, the conversion formula is as follows: , , , Among them, X, Y, Z represent the three-dimensional coordinates of the target pixel in the camera coordinate system, where Z is the value of the depth feature in the camera coordinate system; U, V are the UV coordinates of the target pixel. and Represents the focal length in the horizontal and vertical directions respectively; and They represent the horizontal and vertical offsets of the image center point respectively, and Depth represents the depth feature of the target pixel point.

[0040] S5-4, traverse J target pixel points and repeat S5-1 to S5-3 until a glue-coated area point cloud consisting of J point cloud coordinates is generated.

[0041] In this embodiment, by extracting the UV coordinates of the target pixel point and determining the pixel grid where it is located, combined with the depth features corresponding to the position, the camera intrinsic parameter matrix is ​​used to back-project the two-dimensional image coordinates and the depth value, and the three-dimensional spatial coordinates of each target pixel point in the camera coordinate system are calculated. Finally, all J target pixel points are traversed to generate a point cloud of the glued area composed of their three-dimensional coordinates; this method performs three-dimensional reconstruction based on the pixel points of the segmented area to be glued, ensuring that the point cloud data covers the actual area to be glued and avoiding the introduction of invalid points; at the same time, since the back-projection process directly depends on the pixel-level aligned RGB features and depth features, the generated three-dimensional point cloud is closely fitted to the geometric shape of the sole glued boundary in spatial distribution, and can truly reflect the three-dimensional structure of complex surfaces, small gaps or edge turning points, providing a clear and directly drivable geometric coordinate input for the robot's automated gluing, thereby improving the efficiency of automated gluing of the sole.

[0042] Example 2: The technical solution of Example 2 differs from that of Example 1 in that it discloses a pre-training step for the sole segmentation model in Example 1, which includes: A1. Obtain a set of camouflaged segmented images; A2. Input the camouflage segmentation image set into a convolutional neural network model for iterative supervised training to generate the sole segmentation model.

[0043] Furthermore, step A1 of Example 2 specifically includes: A1-1. Obtain RGBD images of several sole samples; A1-2. Label each pixel on the RGBD images of several sole samples with a binary classification label; the binary classification label includes: points to be glued and points not to be glued; Exemplarily, in this embodiment, the labeling is completed by professional image labeling software.

[0044] A1-3. Perform image enhancement on the RGBD images of several shoe sole samples annotated with binary classification labels to generate several RGBD enhanced images. It's important to note that image enhancement, as described in the examples, refers to operations performed without damaging or interfering with the accuracy of the original RGBD image and its corresponding binary classification labels (the points to be glued and the points not to be glued). These operations include color adjustment, noise addition, cropping, and padding. In short, during image enhancement, it's crucial to ensure that the enhancement operation doesn't increase the difficulty of object recognition in the original RGBD image or invalidate the original binary classification labels.

[0045] A1-4. Mix the RGBD images of several shoe sole samples annotated with binary classification labels with several RGBD enhanced images to generate a camouflaged segmentation image set.

[0046] In this embodiment, by obtaining RGBD images of several sole samples, labeling each pixel with a binary label of a point to be glued or a point not to be glued, and then enhancing the image, a semantically clear camouflage segmentation image set is constructed.

[0047] Furthermore, step A2 of Example 2 specifically includes: A2-1. Inputting the disguised segmented image set into a convolutional neural network model; A2-2. Extract the current batch of RGBD images from the camouflaged segmentation image set and generate the pixel prediction mask map of the current batch through forward propagation; Specifically, the pixel prediction mask map is the result obtained by performing feature extraction and classification on the input RGBD image through a convolutional neural network model, wherein each pixel corresponds to a prediction probability value, indicating the probability that the pixel belongs to the point to be glued.

[0048] A2-3. Calculate the cross entropy loss between the pixel prediction mask of the current batch and the binary classification labels corresponding to the RGBD image; Specifically, the cross entropy loss is used to quantify the difference between the pixel prediction mask of the current batch and the true binary classification label; the cross entropy loss formula is as follows: ; Where L represents the cross entropy loss of the current batch, P represents the total number of pixels in the current batch, including all pixels in the height and width directions, that is, P = H × W; i represents the index of the pixel point, from 1 to P; Indicates the true binary classification label of the i-th pixel in the current batch of RGBD images (0 or 1); where 1 represents the point to be glued and 0 represents the point not to be glued; Indicates the predicted probability value of the i-th pixel in the pixel prediction mask image belonging to the point to be glued, ranging from [0,1].

[0049] A2-4. With the goal of minimizing the cross entropy loss, update the parameters of the convolutional neural network model and iteratively calculate the cross entropy loss of the next batch until the convolutional neural network model converges to the sole segmentation model.

[0050] In this example, a convolutional neural network model is used to feed a set of disguised segmented images into a dataset. A forward propagation is then performed to generate a predicted pixel mask for the current batch. A cross-entropy loss is then calculated based on the difference between the predicted mask and the true binary classification labels. To minimize this loss, the model parameters are iteratively updated using a backpropagation algorithm, gradually optimizing the network's ability to distinguish between glue-applied and non-glue-applied points in the RGBD image. Ultimately, a shoe sole segmentation model with stable output is generated.

[0051] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (e.g., infrared, wireless, microwave, etc.).

[0052] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. The method for generating point cloud of sole glue area based on RGBD camouflage instance segmentation is characterized by: include: S1. Obtain a target sole image; wherein the target sole image includes RGB features and depth features of each pixel; S2, constructing a four-modal tensor matrix corresponding to each pixel point on the target sole image; Among them, the shape of the four-modal tensor matrix is ​​H×W, H and W are the number of pixels in the height direction and width direction of the four-modal tensor matrix respectively; S3. Input the four-modal tensor matrix into a pre-trained sole segmentation model, and generate a binary classification probability for each pixel through forward propagation; S4. Determine J target pixels based on the binary classification probability of each pixel; S5. Generate a point cloud of the glued area of ​​the target sole based on the J target pixel points.

2. The method for generating point cloud of sole glue area based on RGBD camouflage instance segmentation according to claim 1 is characterized in that: The constructing of a four-modal tensor matrix corresponding to each pixel point on the target sole image includes: S2-1, constructing UV coordinates for each pixel point on the target sole image; S2-2, defining a pixel grid for each pixel based on the UV coordinates constructed for each pixel; S2-3, extracting RGB features and depth features corresponding to the pixel grid on the target sole image; S2-4, concatenate the RGB features and depth features located in the same pixel grid to generate a four-modal tensor corresponding to the pixel point; S2-5, traverse each pixel point of the target sole image and repeat S2-1 to S2-4 until a four-modal tensor corresponding to H×W pixels is generated; S2-6. Arrange the H×W quadrimodal tensors based on the spatial positions of the UV coordinates to construct a quadrimodal tensor matrix.

3. The method for generating point cloud of sole glue area based on RGBD camouflage instance segmentation according to claim 2 is characterized in that: The UV coordinates constructed based on each pixel point define a pixel grid for each pixel point, including: S2-2-1. Anchor the UV coordinates of any pixel point as the center coordinates; S2-2-2. Taking the center coordinate as the starting point, determine the coordinates of its four neighboring regions in orthogonal directions; S2-2-3. Connect the coordinates of the four neighborhoods and the center coordinate to form four spatial lines; S2-2-4, locate the four midpoints of the four spatial lines; S2-2-5. Connect the four midpoints in a clockwise or counterclockwise order to form an initial grid centered on the central coordinate; S2-2-6. Rotate the initial grid 90 degrees around the center coordinate to generate an orthogonalized pixel grid; S2-2-7. Traverse the UV coordinates of each pixel point until the pixel grid of each pixel point is defined.

4. The method for generating point cloud of sole glue area based on RGBD camouflage instance segmentation according to claim 1 is characterized in that: The pre-training steps of the sole segmentation model include: A1. Obtain a set of camouflaged segmented images; A2. Input the camouflage segmentation image set into a convolutional neural network model for iterative supervised training to generate the sole segmentation model.

5. The method for generating point cloud of sole glue area based on RGBD camouflage instance segmentation according to claim 4 is characterized in that: The step of obtaining the disguised segmented image set includes: A1-1. Obtain RGBD images of several sole samples; A1-2. Label each pixel on the RGBD images of several sole samples with a binary classification label; the binary classification label includes: points to be glued and points not to be glued; A1-3. Perform image enhancement on the RGBD images of several shoe sole samples annotated with binary classification labels to generate several RGBD enhanced images. A1-4. Mix the RGBD images of several shoe sole samples annotated with binary classification labels with several RGBD enhanced images to generate a camouflaged segmentation image set.

6. The method for generating point cloud of sole glue coating area based on RGBD camouflage instance segmentation according to claim 5 is characterized in that: The generating of the sole segmentation model comprises: A2-1. Inputting the disguised segmented image set into a convolutional neural network model; A2-2. Extract the current batch of RGBD images from the camouflaged segmentation image set and generate the pixel prediction mask map of the current batch through forward propagation; A2-3. Calculate the cross entropy loss between the pixel prediction mask of the current batch and the binary classification labels corresponding to the RGBD image; A2-4. With the goal of minimizing the cross entropy loss, update the parameters of the convolutional neural network model and iteratively calculate the cross entropy loss of the next batch until the convolutional neural network model converges to the sole segmentation model.

7. The method for generating point cloud of sole glue area based on RGBD camouflage instance segmentation according to claim 1 is characterized in that: The method of generating the binary classification probability of each pixel point through forward propagation includes: S3-1, using the UV coordinate as the index, anchor the target pixel in the four-modal tensor matrix; S3-2. Perform convolution extraction on the four-modal tensor of the target pixel and its neighboring pixels based on the pre-trained K convolution kernels to generate a convolution feature vector of the target pixel; S3-3, decoding the convolution feature vector of the target pixel point to obtain the binary classification probability that the target pixel point belongs to the glue-coated area; S3-4. Traverse the UV coordinates until the binary classification probability of each pixel point in the four-modal tensor matrix is ​​generated.

8. The method for generating point cloud of sole glue area based on RGBD camouflage instance segmentation according to claim 2 is characterized in that: The step of determining J target pixels based on the binary classification probability of each pixel includes: S4-1, comparing the binary classification probability of each pixel in the four-modal tensor matrix with the set threshold; S4-2, if the binary classification probability is not less than the set threshold, then mark the pixel as a point to be glued, otherwise, mark it as a non-glue point; S4-3, traverse the four-modal tensor matrix and repeat S4-1 to S4-2 until a pixel mask corresponding to H×W pixels is obtained; S4-4, in the pixel mask image, marking the pixel corresponding to the point to be glued as the target pixel; S4-5. Traverse the pixel mask map until J target pixels are determined.

9. The method for generating point cloud of sole glue coating area based on RGBD camouflage instance segmentation according to claim 8, characterized in that: Generating a point cloud of the glued area of ​​the target sole according to the J target pixel points includes: S5-1, extracting the UV coordinates of the target pixel point, and determining the pixel grid where the target pixel point is located based on the UV coordinates; S5-2, extracting the depth feature of the target pixel point based on the pixel grid where the target pixel point is located; S5-3, back-projecting the UV coordinates and depth features of the target pixel point to calculate the point cloud coordinates of the target pixel point; S5-4, traverse J target pixel points and repeat S5-1 to S5-3 until a glue-coated area point cloud consisting of J point cloud coordinates is generated.

Images