Tomato detection method and system based on multi-view parallel three-dimensional modeling

Abstract

The invention provides a tomato detection method and system based on multi-view parallel three-dimensional modeling, and the method comprises the steps: synchronously obtaining original images under N view angles on an assembly line, extracting a view field image, and distributing a unique view angle code; dividing each visual angle image with the unique visual angle code into a plurality of image blocks distributed with unique block position codes, extracting features of each visual angle image layer by layer by adopting a hierarchical multi-visual-field visual converter, and then performing fusion to obtain a fused multi-visual-angle joint feature sequence; outputting a global three-dimensional point diagram by the global solution head according to the multi-view joint features, and calculating to obtain a tomato size detection result; and outputting a local three-dimensional point diagram by the local solution head according to the multi-view joint features, and obtaining a tomato defect detection result according to the local three-dimensional point diagram and the local semantic information so as to generate a tomato detection result. The method has the beneficial effects that the real-time performance is high, the detection accuracy is high, and the assembly line tomato detection with continuous treatment can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural product detection, and in particular to a tomato detection method and system based on multi-view parallel three-dimensional modeling. Background Art

[0002] In agricultural factory production lines, tomato appearance and size inspection are key components of quality grading and packaging. Surface defects, rot, or substandard tomatoes can reduce consumer purchasing interest and consumption experience, weakening market competitiveness. Therefore, efficient and accurate online inspection is crucial for improving tomato product quality and optimizing production line efficiency.

[0003] Current pipeline visual inspection technology has bottlenecks: single-view vision only obtains two-dimensional projections, lacks depth perception, relies on geometric assumptions to infer three-dimensional structures, and is difficult to adapt to dynamic detection; although binocular vision can calculate depth, the smooth surface easily leads to feature point mismatching and three-dimensional reconstruction distortion, and the three-dimensional parameters need to be indirectly derived, affecting accuracy. In addition, the traditional single / binocular vision frame-by-frame independent processing mode is inefficient and cannot match the high-speed beat of the pipeline. In terms of three-dimensional reconstruction, traditional methods rely on multi-stage pipelines, which are computationally complex and have accumulated errors, making it difficult to meet real-time requirements; mainstream methods such as DUSt3R only support paired image input, and the amount of calculation increases quadratically when processing dense multi-view sequences, resulting in high memory consumption and difficulty adapting to continuous operations.

[0004] Therefore, there is an urgent need to develop an assembly line tomato detection method with high real-time performance, high detection accuracy and continuous processing. Summary of the Invention

[0005] In order to solve the above-mentioned problems in the prior art, the present invention provides a tomato detection method and system based on multi-view parallel 3D modeling, which can achieve high real-time performance, high detection accuracy and continuous processing of pipeline tomato detection.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: In a first aspect, the present invention provides a tomato detection method based on multi-view parallel 3D modeling, comprising the steps of: S1. Synchronously acquire original images from N viewing angles on the pipeline, extract the field of view image corresponding to a single tomato from each original image, and assign a unique viewing angle code to each field of view image; S2. Divide each view image with a unique view code into multiple image blocks, assign a unique block position code to each image block, and use a hierarchical multi-view visual transformer to extract the features of each view image layer by layer to obtain an image block feature sequence for each view image; S3. Fusing the unique perspective code with the image block feature sequence to obtain a fused perspective feature sequence, fusing the fused perspective features of all perspective images through global attention to obtain a fused multi-perspective joint feature sequence, wherein the features of each image block in the multi-perspective joint feature sequence include local semantic information of the view image in which it is located and collaborative geometric information with other view images; S4. The global decoding head outputs a global 3D point map of all view pixels in a reference coordinate system based on the multi-view joint features, and the local decoding head outputs a local 3D point map in a camera coordinate system corresponding to each view image based on the multi-view joint features; S5. Calculate a size detection result of the tomato based on the global three-dimensional point graph, and obtain a defect detection result of the tomato based on the local three-dimensional point graph and the local semantic information; S6. Generate a tomato detection result according to the size detection result and the defect detection result.

[0007] The beneficial effects of the present invention are as follows: A multi-view parallel reconstruction method, which extracts features from multiple tomato images from different perspectives and then fuses them, enables single forward propagation processing of multiple tomato images from multiple perspectives to complete 3D reconstruction of the tomato. This method avoids the quadratic increase in computational complexity, high memory consumption, and large error accumulation associated with traditional methods for processing dense multi-view sequences, making it suitable for continuous production pipelines and improving the real-time and stability of 3D reconstruction. Furthermore, dual-branch point cloud decoding using a global and local decoding head decouples size measurement from defect location, improving both tomato size measurement accuracy and defect detection accuracy. This enables pipeline tomato detection with high real-time performance, high detection accuracy, and continuous processing.

[0008] Optionally, step S2 includes the following steps: S21, dividing each perspective image with a unique perspective code into multiple image blocks; S22, after generating initial features for each image block through linear projection, assign a unique block position code, and then perform feature splicing on the unique viewing angle code of the field image to generate an initial stage feature map including the unique viewing angle code and the unique block position code; S23, performing multiple downsampling and transformer layer operations on each initial feature map in sequence, and obtaining multiple stage feature maps of different sizes in sequence; S24, fusing high-level global features and low-level local features of the initial stage feature map and multiple stage feature maps of different sizes to generate a multi-scale enhanced feature map, and globally pooling the multi-scale enhanced feature map to obtain a feature vector for each image block; S25 , combining the feature vectors of each image block in each viewing angle image to obtain an image block feature sequence of each viewing angle image.

[0009] Optionally, the feature extraction of each image block in step S2 may be processed in parallel by a GPU.

[0010] Optionally, step S3 includes the following steps: S31, fusing the unique viewing angle code with the image block feature sequence to obtain a fused viewing angle feature sequence; S32, splitting the fused view feature sequence into multiple heads, and performing attention calculation on each head to obtain a single-head attention output for each head, and then performing multi-head fusion on the single-head attention outputs of all heads to obtain a multi-head attention output; S33, performing residual connection and normalization, convolutional feedforward neural network processing, and residual connection and normalization on the multi-head attention output in sequence to obtain the current layer output feature; S34. After stacking multiple layers of interaction of step S32 and step S33, a fused multi-view joint feature sequence is obtained, wherein the features of each image block in the multi-view joint feature sequence include the local semantic information of the field of view image and the collaborative geometric information of other field of view images.

[0011] Optionally, step S33 includes the following steps: S331. Perform a residual connection on the multi-head attention output and the current layer input feature to obtain a residual output, and perform layer normalization on the residual output to obtain a normalized output. The calculation formulas for the residual connection and the layer normalization are as follows: ResOut=AttnOut+H l-1 ; NormAttn = BatchNorm(ResOut); Among them, ResOut is the residual output, AttnOut is the multi-head attention output, H l-1 The input feature of the current layer, NormAttn is the normalized output, and BatchNorm is the normalization layer; S332, performing a convolution operation on the normalized output through a convolution kernel, applying an activation function to introduce a nonlinear transformation, and then performing a pooling operation to obtain a convolution feedforward output; S333. Apply residual connection and normalization to the convolution feedforward output again to obtain the current layer output feature. The calculation formula of the residual connection and normalization is as follows: H l = BatchNorm(Pooled+ResOut); Among them, H l is the output feature of the current layer, NormAttn is the normalized output, BatchNorm is the normalized layer, and Pooled is the convolution feedforward output.

[0012] Optionally, the reference coordinate system in step S4 is a camera coordinate system corresponding to a perspective image presenting the most tomato features among all perspective images.

[0013] Optionally, obtaining the defect detection result of the tomato according to the local three-dimensional point map and the local semantic information in step S5 includes the following steps: Defect detection is performed based on the local three-dimensional point map and the local semantic information to obtain the current defect and the defect area of ​​the current defect in the corresponding perspective image. Then, the local three-dimensional point map is aligned to the reference coordinate system through a normal distribution transformation algorithm to obtain the spatial position information of the current defect, so as to obtain a defect detection result including the current defect and the defect area and spatial position information of the current defect.

[0014] Optionally, step S1 further includes the following steps: The original images from N perspectives on the pipeline are acquired synchronously. Each original image is corrected for geometric distortion and brightness normalized. Then, a tomato segmentation network based on a residual backbone combined with pyramid pooling is used to extract the field of view image corresponding to a single tomato. A unique perspective code is assigned to each field of view image to generate an input tensor that includes the field of view images from all perspectives.

[0015] Optionally, step S6 includes the following steps: Marking the current defect and defect area in the corresponding view image according to the defect detection result, and visually rendering the global three-dimensional point map to obtain a three-dimensional model of the tomato, and marking the defect on the three-dimensional model according to the spatial position information of the current defect in the defect detection result; Generate a tomato detection result based on the size detection result and the defect detection result, and generate a report in a preset format; Control information is generated according to the tomato detection result, and the control information is sent to a robotic arm on the assembly line for sorting.

[0016] In a second aspect, the present invention provides a tomato detection system based on multi-view parallel three-dimensional modeling, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the tomato detection method based on multi-view parallel three-dimensional modeling of the first aspect is implemented.

[0017] Among them, the technical effects corresponding to the tomato detection system based on multi-view parallel 3D modeling provided by the second aspect refer to the relevant description of the tomato detection method based on multi-view parallel 3D modeling provided by the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the main process of a tomato detection method based on multi-view parallel 3D modeling according to an embodiment of the present invention; Figure 2 Schematic diagram of a tomato accurate segmentation network based on a combination of residual backbone and pyramid pooling according to an embodiment of the present invention; Figure 3 A schematic diagram of a model of a hierarchical multi-view visual converter according to an embodiment of the present invention; Figure 4 Schematic diagram of multi-layer interaction between an encoder and a decoder involved in an embodiment of the present invention; Figure 5 Schematic diagram of a tomato detection system based on multi-view parallel 3D modeling according to an embodiment of the present invention.

[0019] Description of reference numerals: 1. Tomato detection system based on multi-view parallel 3D modeling; 2. Processor; 3. Memory. DETAILED DESCRIPTION

[0020] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0021] Example 1 Please refer to Figure 1 , a tomato detection method based on multi-view parallel 3D modeling, comprising the steps of: S1. Synchronously acquire original images from N viewing angles on the pipeline, extract the field of view image corresponding to a single tomato from each original image, and assign a unique viewing angle code to each field of view image.

[0022] Wherein, step S1 further includes the steps of: The original images from N perspectives on the pipeline are acquired synchronously. Each original image is corrected for geometric distortion and brightness normalized. Then, a tomato segmentation network based on a residual backbone combined with pyramid pooling is used to extract the field of view image corresponding to a single tomato. A unique perspective code is assigned to each field of view image to generate an input tensor that includes field of view images from all perspectives.

[0023] In this embodiment, multiple sets of industrial cameras, such as 5 to 10 cameras with a resolution greater than 512×512, are deployed on both sides and above the production line to simultaneously capture RGB images of tomatoes during transport. A uniform lighting system, such as a ring-shaped shadowless light source, is also integrated to minimize the impact of surface reflections on image quality.

[0024] At this time, refer to Figure 2 It can be seen that the pre-processing in step S1 of this embodiment includes the following steps: S11. For the original image captured by the pipeline from multiple perspectives, normalize the pixel coordinates to the camera coordinate system based on the camera calibration parameters. Apply the distortion correction formula to each pixel of the image to eliminate radial and tangential distortion, restore the image to the image coordinate system, and output an orthographic image with geometric distortion eliminated, restoring the true shape and spatial position of the tomato. The formula involved is: ; ; ; ; ; ; ; Among them, the camera calibration parameters include the intrinsic parameter matrix K, the first-order, second-order, and third-order radial distortion coefficients k1, k2, and k3, and the first-order and second-order tangential distortion coefficients p1 and p2.

[0025] Among them, x distorted and y distorted is the pixel coordinate before distortion, c x 、c y is the principal point coordinate, f x 、f y is the focal length, x undistorted and y undistorted The pixel coordinates after distortion correction.

[0026] S12. For the geometrically corrected tomato image, use zero-mean normalization to calculate the global mean μ and standard deviation σ of the image. Adjust the brightness pixel by pixel to reduce light interference and provide stable input for subsequent feature extraction. The formula is: ; Among them, I(i,j) represents the original brightness value of the pixel with coordinates (i,j) in the tomato image after geometric correction, I norm (i, j) represents the new pixel value obtained by performing zero-mean normalization (Z-Score) on the original pixel value I(i, j). =10 -5 , to prevent division by zero.

[0027] The above steps eliminate the interference of global illumination of the image, making the pixel brightness distribution satisfy the standard normal distribution with a mean of 0 and a standard deviation of 1, thereby providing a stable input for subsequent feature extraction.

[0028] S13, the image I with a size of 512×512×3 and normalized brightness norm , first perform initial convolution and pooling, including 7×7 convolution with stride 2, followed by 3×3 maximum pooling with stride 2. Then perform residual block stacking, including 3 residual blocks, 4 residual blocks, 6 residual blocks, and 3 residual blocks. Finally, we get a 1 / 8 size feature map, denoted as F backbone .

[0029] S14, F backbone Perform multi-scale pooling, where the global average pooling is for F backbone Calculate the global mean, 2×2 block pooling is to divide the feature map into 2×2 sub-blocks, and average pooling is performed block by block, 3×3 block pooling is to divide the feature map into 3×3 sub-blocks, and average pooling is performed block by block, and 6×6 block pooling is to divide the feature map into 6×6 sub-blocks, and average pooling is performed block by block.

[0030] S15. For each pooled output, restore it to the backbone network feature map size through bilinear interpolation upsampling, and then pass through a 1×1 convolution compression channel to obtain a multi-scale feature map F pool,1 ,F pool,2 ,F pool,3 ,F pool,4 .

[0031] S16, splicing original backbone network feature map F backbone Combined with multi-scale pooling features, we get the fusion feature F concat .

[0032] S17, compress the features through 3×3 convolution with 512 channels, and then restore the feature map size to the input image level of 512×512 by upsampling ×8, and output F decode .

[0033] S18, decoded feature map F decode Perform 1×1 convolution to compress the channel to 1 and output a single-channel feature map F single, and then through Sigmoid activation, the feature value is mapped to [0,1] to distinguish the high-value tomato area and the low-value background. The formula is: ; Finally, we set the threshold τ and make pixel-by-pixel decisions, outputting a 512×512 binary mask where the area with a value of 1 is the tomato ROI: .

[0034] S2. Divide each perspective image with a unique perspective code into multiple image blocks, assign a unique block position code to each image block, and use a hierarchical multi-view visual transformer to extract the features of each perspective image layer by layer to obtain an image block feature sequence for each perspective image.

[0035] In this embodiment, the feature extraction of each image block in step S2 can be processed in parallel by the GPU. This, combined with the independent multi-head attention computation in subsequent steps, enables real-time processing of multi-view images of dozens of tomatoes per second in the pipeline, improving the real-time and scalability of tomato appearance and size detection.

[0036] Among them, Figure 3 As shown, step S2 includes the following steps: S21. Divide each view image with a unique view code into multiple image blocks.

[0037] In this embodiment, the number of image blocks is H*W / p 2 =512*512 / 16, the image block set is: ; Among them, I i,j Represented as the image block at row i and column j.

[0038] S22. After generating initial features for each image block through linear projection, a unique block position code is assigned, and then the unique viewing angle code of the field of view image is feature spliced ​​to generate an initial stage feature map including the unique viewing angle code and the unique block position code.

[0039] Among them, after the initial features are generated for each image block through linear projection, they are first integrated into its two-dimensional position coding in a single image, such as row and column coordinate coding, and then the unique perspective coding p corresponding to the perspective is obtained. i Perform feature concatenation or element-by-element addition to generate an initial feature token containing view information and local position information.

[0040] Among them, the unique perspective code p iUsed to identify the spatial orientation of the view, such as camera 1 corresponds to p1, camera 2 corresponds to p2. It is globally unique and identifies the spatial position of different cameras.

[0041] The unique block position encoding is unique within a single view and is used to identify the location of an image block within a single image, helping the model understand the local spatial relationships between image blocks. For example, the position encoding establishes a vertical relationship between the top and bottom blocks of a tomato. Combining these two capabilities enables dual positioning capabilities: cross-view spatial positioning and single-view local position awareness.

[0042] Then, through the transformer layer, local correlations are modeled within the image block and between adjacent image blocks, and the initial stage feature map is output, i.e. Figure 3 The feature map of the middle stage 1 has a size of H / 4*W / 4.

[0043] S23. Perform multiple downsampling and transformer layer operations on each initial feature map in sequence to obtain multiple stage feature maps of different sizes.

[0044] like Figure 3 As shown in Figure 1, the stage 1 feature map passes through the downsampling module to compress the feature map size to H / 8*W / 8 while retaining the key features. It then passes through the transformer layer again to expand the attention range and fuse features from a wider area, outputting the stage 2 feature map with a size of H / 8*W / 8.

[0045] Then, the stage 2 feature map is repeatedly downsampled and operated with the transformer layer to compress the feature map size to H / 16×W / 16, further aggregate the global context information, and output the stage 3 feature map with a size of H / 16×W / 16.

[0046] Finally, the stage 3 feature map is subjected to the final downsampling and transformer layer operation to compress the feature map size to H / 32×W / 32, completing the layer-by-layer extraction from local to global features and outputting the stage 4 feature map.

[0047] That is, the multiple stage feature maps of different sizes in this embodiment include a stage 2 feature map of size H / 8*W / 8, a stage 3 feature map of size H / 16×W / 16, and a stage 4 feature map of size H / 32×W / 32. In other embodiments, the number of stage feature map sizes can be appropriately increased or decreased.

[0048] S24. Fusing high-level global features and low-level local features of the initial stage feature map and multiple stage feature maps of different sizes to generate a multi-scale enhanced feature map, and globally pooling the multi-scale enhanced feature map to obtain a feature vector for each image block.

[0049] like Figure 3As shown in the figure, the multi-scale feature maps output by stages 1-4 are fused through the top-down path and lateral connections of the Feature Pyramid Network (FPN) algorithm, combining the high-level global features of stage 4 with the low-level local features of stages 1-3 to generate a multi-scale enhanced feature map. The multi-scale feature maps fused by the FPN algorithm are then globally pooled to obtain the feature vector for each image block.

[0050] Therefore, through hierarchical multi-view feature extraction, the surface texture and overall shape information of the tomato are retained, the integrity of the point cloud is improved, and the three-dimensional reconstruction of the tomato is more complete.

[0051] S25 , combining the feature vectors of each image block in each viewing angle image to obtain an image block feature sequence of each viewing angle image.

[0052] like Figure 3 As shown in Figure 1, the two-dimensional feature map is compressed into a one-dimensional vector; then it is mapped through a linear layer to finally output a 1024-dimensional feature vector.

[0053] S3. Fuse the unique perspective code with the image block feature sequence to obtain a fused perspective feature sequence. Through global attention, fuse the fused perspective features of all perspective images to obtain a fused multi-perspective joint feature sequence. The features of each image block in the multi-perspective joint feature sequence include the local semantic information of the field of view image and the collaborative geometric information of other field of view images.

[0054] Wherein, step S3 includes the following steps: S31. Fusing the unique viewing angle code with the image block feature sequence to obtain a fused viewing angle feature sequence.

[0055] Among them, the generated view identification encoding vector P=[p1,p2,…,p N ], p i and the feature sequence H of the i-th view i Each image block feature h i,j Add element by element to get the feature H of fusion view information i '=h i,j +p i , ensuring that the model can distinguish the spatial locations of different views.

[0056] Among them, suppose that the i-th view image is feature-encoded to generate the image block feature sequence H i ={h i,1 ,h i,2 ,…,h i,M}, where h i,j ∈R 1024represents the feature vector of the jth image block in the i-th view, and M is the total number of image blocks. In this embodiment, when the 512×512 image is divided into 16×16 blocks, M=1024.

[0057] S32. Split the fused view feature sequence into multiple heads, and perform attention calculation on each head separately to obtain the single-head attention output of each head. Then, perform multi-head fusion on the single-head attention outputs of all heads to obtain the multi-head attention output.

[0058] Among them, the input feature H l-1 Split into M heads, first for each head, H l-1 Mapped to query vector Q, key vector K and value vector V, that is, Q=HW Q ,K=HW K ,V=HW V Then calculate the dot product similarity between the query and the key Score=(Q*K T ) / , where d k is the key vector dimension; and the attention weight is obtained by Softmax normalization: Atten=Softmax(Score). Then the value vector is weighted and summed according to the attention weight to obtain the single head output Head m =Atten*V, and finally multi-head fusion, the output of M heads is spliced ​​and passed through the linear layer to obtain the attention output AttnOut=Concat(Head1,…,Head M )*W O .

[0059] S33. The multi-head attention output is sequentially subjected to residual connection and normalization, convolutional feedforward neural network processing, and residual connection and normalization to obtain the output features of the current layer.

[0060] Wherein, step S33 includes the following steps: S331. Perform a residual connection on the multi-head attention output and the current layer input feature to obtain a residual output, and perform layer normalization on the residual output to obtain a normalized output. The calculation formulas for residual connection and layer normalization are as follows: ResOut=AttnOut+H l-1 ; NormAttn = BatchNorm(ResOut); Among them, ResOut is the residual output, AttnOut is the multi-head attention output, H l-1 The input feature of the current layer, NormAttn is the normalized output, and BatchNorm is the normalization layer.

[0061] S332. Perform a convolution operation on the normalized output through the convolution kernel, apply an activation function to introduce a nonlinear transformation, and then perform a pooling operation to obtain a convolution feedforward output.

[0062] Among them, the calculation formulas for convolution operation, nonlinear transformation and pooling operation are as follows: ; ; .

[0063] The above formula is commonly used in existing models and will not be described in detail here.

[0064] S333. Apply residual connection and normalization to the convolution feedforward output again to obtain the output features of the current layer. The calculation formula of residual connection and normalization is as follows: H l = BatchNorm(Pooled+ResOut); Among them, H l is the output feature of the current layer, NormAttn is the normalized output, BatchNorm is the normalized layer, and Pooled is the convolution feedforward output.

[0065] S34. After stacking multiple layers of interaction of step S32 and step S33, a fused multi-view joint feature sequence is obtained. The features of each image block in the multi-view joint feature sequence include the local semantic information of the field of view image and the collaborative geometric information of other field of view images.

[0066] like Figure 4 As shown, by stacking multiple interaction layers in the encoder of steps S32 and S33, the semantic associations and geometric constraints of cross-view features are gradually enhanced. Lower interaction layers focus on capturing local feature correspondence, such as matching tomato surface textures, while higher interaction layers focus on global structural alignment, such as modeling the 3D shape of the entire fruit. Consequently, the subsequent decoder also adopts an 18-layer stacked interaction structure.

[0067] In this embodiment, block computing is used to divide long sequence features into small blocks for attention calculation, avoiding video memory overflow and supporting parallel processing of thousands of views.

[0068] Therefore, after 18 layers of interaction, the encoder outputs a multi-view joint feature sequence Z = H that integrates all view information. L , where each image block feature z i,jThis includes the local semantic information of the current view and the collaborative geometric information of other views. The former is the skin color, spots, etc., and the latter is the consistency of 3D coordinates across different viewing angles. This feature sequence is subsequently input into step S4 to generate a global and local 3D coordinate mapping.

[0069] S4. The global decoding head outputs a global 3D point map of all view pixels in the reference coordinate system based on the multi-view joint features, and the local decoding head outputs a local 3D point map in the camera coordinate system corresponding to each view image based on the multi-view joint features.

[0070] Specifically, step S4 includes the following steps: S41, the camera coordinate system corresponding to the view image showing the most tomato features among all view images is used as the reference coordinate system, and the global decoding head outputs the global three-dimensional point map X of all view pixels in the reference coordinate system. G , used to calculate the global size parameters such as the volume and horizontal diameter of the tomato. For each view, the local decoding head outputs the 3D point map X in the camera coordinate system i .

[0071] That is, the reference coordinate system in step S4 is the camera coordinate system corresponding to the perspective image that presents the most tomato features among all perspective images. Specifically, it can correspond to the image captured by one of the N industrial cameras deployed in the production line. Usually, the perspective that can fully present the main features of the tomato is selected as the first view. For example, if camera 1 is deployed directly above the production line and captures the top image of the tomato, then the camera coordinate system I1 is the image from the top perspective. The camera coordinate system of I1 is used as the global unified coordinate system, that is, the reference coordinate system; the other views (I2, I3, ..., I N ) needs to be aligned to the reference coordinate system to achieve unified expression of multi-view 3D information.

[0072] S42. Synchronously output a confidence map to characterize the reliability of each 3D coordinate prediction and suppress low-confidence outliers.

[0073] S43. Align the local point map to the global coordinate system through the normal distribution transformation algorithm to improve the detail accuracy of defect positioning.

[0074] S5. Calculate the size detection result of the tomato based on the global three-dimensional point graph, and obtain the defect detection result of the tomato based on the local three-dimensional point graph and local semantic information.

[0075] Wherein, obtaining the defect detection result of the tomato based on the local three-dimensional point map and the local semantic information in step S5 includes the following steps: Defect detection is performed based on the local three-dimensional point map and local semantic information to obtain the current defect and the defect area of ​​the current defect in the corresponding perspective image. Then, the local three-dimensional point map is aligned to the reference coordinate system through the normal distribution transformation algorithm to obtain the spatial position information of the current defect, so as to obtain a defect detection result including the current defect, the defect area of ​​the current defect and the spatial position information.

[0076] Specifically: According to the global three-dimensional point map X G Calculate the 3D bounding box of the tomato, extract geometric parameters such as major axis length, volume, and surface area, and use point cloud density to estimate the fruit fullness.

[0077] The local three-dimensional point map X i The coordinate information of the image, such as surface curvature and depth, is combined with the semantic information in the image feature H, such as color and texture, and input into the defect classification head. Through the anomaly detection algorithm, defect areas such as cracks and rot are marked on the two-dimensional image and mapped to the global three-dimensional point map X. G To determine the spatial location.

[0078] The test results are output based on the preset size threshold and defect confidence threshold, and the test results are divided into pass / fail and defect type.

[0079] S6. Generate a tomato detection result based on the size detection result and the defect detection result.

[0080] Wherein, step S6 includes the following steps: S61. Mark the current defect and defect area in the corresponding view image according to the defect detection result, and perform visual rendering on the global three-dimensional point map to obtain a three-dimensional model of the tomato. Mark the defect on the three-dimensional model according to the spatial position information of the current defect in the defect detection result.

[0081] That is, this embodiment can realize visual rendering.

[0082] S62: Generate a tomato detection result based on the size detection result and the defect detection result, and generate a report in a preset format.

[0083] That is, this embodiment can implement data formatting.

[0084] S63. Generate control information based on the tomato detection results, and send the control information to the robotic arm on the assembly line for sorting.

[0085] That is, this embodiment can process tomatoes through external interaction and is suitable for high-throughput real-time detection of tomato sorting, grading and quality control after picking.

[0086] Therefore, the advantages of this embodiment are: 1. A multi-view parallel reconstruction method uses feature extraction and subsequent fusion of tomato images from multiple perspectives to achieve 3D reconstruction of the tomato. This method avoids the quadratic growth of computational complexity, high memory consumption, and large error accumulation associated with traditional methods for processing dense multi-view sequences. It is suitable for continuous production line operations and improves the real-time and stability of 3D reconstruction.

[0087] 2. A hierarchical multi-view visual transformer preserves the surface texture and overall shape information of the tomato, making the 3D reconstruction of the tomato more complete. Dual-branch point cloud decoding using a global decoding head and a local decoding head decouples size measurement from defect localization, avoiding feature interference between size measurement and defect detection. This improves the stability of global geometric parameter calculation and the sensitivity of local defect detection, thereby enhancing the measurement accuracy of the tomato and the accuracy of defect detection.

[0088] 3. Through block-based attention calculation and GPU parallel scheduling, we optimize model parallelism and video memory usage, adapt to the hardware environment of industrial pipelines, and achieve real-time multi-view image processing of dozens of tomatoes per second in the pipeline, matching the high-speed production rhythm. This lowers the hardware threshold for large-scale multi-view parallel computing (5-10), supporting industrial-grade deployment and expansion of the system.

[0089] That is, this embodiment can achieve high real-time, high detection accuracy, and continuous processing of pipeline tomato detection.

[0090] Example 2 Please refer to Figure 5 The tomato detection system 1 based on multi-view parallel 3D modeling includes a memory 3, a processor 2, and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, the steps in the above-mentioned embodiment 1 are implemented.

[0091] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art will be able to understand the specific structures and variations of these systems / devices based on the methods described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the methods of the above embodiments of the present invention are within the scope of protection of the present invention.

[0092] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0093] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions.

[0094] It should be noted that, in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims enumerating several means, several of these means may be embodied by one and the same hardware. The use of the words first, second, third etc. is for convenience only and does not indicate any order. These words may be understood as part of the component name.

[0095] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0096] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments after learning the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0097] Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if such modifications and variations fall within the scope of the claims and their equivalents, the present invention shall also include such modifications and variations.

Claims

1. A tomato detection method based on multi-view parallel 3D modeling, characterized by: Including steps: S1. Synchronously acquire original images from N viewing angles on the pipeline, extract the field of view image corresponding to a single tomato from each original image, and assign a unique viewing angle code to each field of view image; S2. Divide each view image with a unique view code into multiple image blocks, assign a unique block position code to each image block, and use a hierarchical multi-view visual transformer to extract the features of each view image layer by layer to obtain an image block feature sequence for each view image; S3. Fusing the unique perspective code with the image block feature sequence to obtain a fused perspective feature sequence, fusing the fused perspective features of all perspective images through global attention to obtain a fused multi-perspective joint feature sequence, wherein the features of each image block in the multi-perspective joint feature sequence include local semantic information of the view image in which it is located and collaborative geometric information with other view images; S4. The global decoding head outputs a global 3D point map of all view pixels in a reference coordinate system based on the multi-view joint features, and the local decoding head outputs a local 3D point map in a camera coordinate system corresponding to each view image based on the multi-view joint features; S5. Calculate a size detection result of the tomato based on the global three-dimensional point graph, and obtain a defect detection result of the tomato based on the local three-dimensional point graph and the local semantic information; S6. Generate a tomato detection result according to the size detection result and the defect detection result.

2. The tomato detection method based on multi-view parallel 3D modeling according to claim 1, characterized in that: The step S2 comprises the following steps: S21, dividing each perspective image with a unique perspective code into multiple image blocks; S22, after generating initial features for each image block through linear projection, assign a unique block position code, and then perform feature splicing on the unique viewing angle code of the field image to generate an initial stage feature map including the unique viewing angle code and the unique block position code; S23, performing multiple downsampling and transformer layer operations on each initial feature map in sequence, and obtaining multiple stage feature maps of different sizes in sequence; S24, fusing high-level global features and low-level local features of the initial stage feature map and multiple stage feature maps of different sizes to generate a multi-scale enhanced feature map, and globally pooling the multi-scale enhanced feature map to obtain a feature vector for each image block; S25 , combining the feature vectors of each image block in each perspective image to obtain an image block feature sequence for each perspective image.

3. The tomato detection method based on multi-view parallel 3D modeling according to claim 1, characterized in that: The feature extraction of each image block in step S2 can be processed in parallel by the GPU.

4. The tomato detection method based on multi-view parallel 3D modeling according to claim 1, characterized in that: The step S3 comprises the following steps: S31, fusing the unique viewing angle code with the image block feature sequence to obtain a fused viewing angle feature sequence; S32, splitting the fused view feature sequence into multiple heads, and performing attention calculation on each head to obtain a single-head attention output for each head, and then performing multi-head fusion on the single-head attention outputs of all heads to obtain a multi-head attention output; S33, performing residual connection and normalization, convolutional feedforward neural network processing, and residual connection and normalization on the multi-head attention output in sequence to obtain the current layer output feature; S34. After stacking multiple layers of interaction of step S32 and step S33, a fused multi-view joint feature sequence is obtained, wherein the features of each image block in the multi-view joint feature sequence include the local semantic information of the field of view image and the collaborative geometric information of other field of view images.

5. The tomato detection method based on multi-view parallel 3D modeling according to claim 4, characterized in that: The step S33 includes the following steps: S331. Perform a residual connection on the multi-head attention output and the current layer input feature to obtain a residual output, and perform layer normalization on the residual output to obtain a normalized output. The calculation formulas for the residual connection and the layer normalization are as follows: ResOut=AttnOut+H l-1 ; NormAttn = BatchNorm(ResOut); Among them, ResOut is the residual output, AttnOut is the multi-head attention output, H l-1 The input feature of the current layer, NormAttn is the normalized output, and BatchNorm is the normalization layer; S332, performing a convolution operation on the normalized output through a convolution kernel, applying an activation function to introduce a nonlinear transformation, and then performing a pooling operation to obtain a convolution feedforward output; S333. Apply residual connection and normalization to the convolution feedforward output again to obtain the current layer output feature. The calculation formula of the residual connection and normalization is as follows: H l = BatchNorm(Pooled+ResOut); Among them, H l is the output feature of the current layer, NormAttn is the normalized output, BatchNorm is the normalized layer, and Pooled is the convolution feedforward output.

6. The tomato detection method based on multi-view parallel 3D modeling according to claim 1, characterized in that: The reference coordinate system in step S4 is the camera coordinate system corresponding to the perspective image that presents the most tomato features among all perspective images.

7. The tomato detection method based on multi-view parallel 3D modeling according to claim 1, characterized in that: Obtaining the defect detection result of the tomato according to the local three-dimensional point map and the local semantic information in step S5 includes the following steps: Defect detection is performed based on the local three-dimensional point map and the local semantic information to obtain the current defect and the defect area of ​​the current defect in the corresponding perspective image. Then, the local three-dimensional point map is aligned to the reference coordinate system through a normal distribution transformation algorithm to obtain the spatial position information of the current defect, so as to obtain a defect detection result including the current defect and the defect area and spatial position information of the current defect.

8. The tomato detection method based on multi-view parallel 3D modeling according to any one of claims 1 to 7, characterized in that: The step S1 further comprises the steps of: The original images from N perspectives on the pipeline are acquired synchronously. Each original image is corrected for geometric distortion and brightness normalized. Then, a tomato segmentation network based on a residual backbone combined with pyramid pooling is used to extract the field of view image corresponding to a single tomato. A unique perspective code is assigned to each field of view image to generate an input tensor that includes the field of view images from all perspectives.

9. The tomato detection method based on multi-view parallel 3D modeling according to claim 1, characterized in that: The step S6 comprises the following steps: Marking the current defect and defect area in the corresponding view image according to the defect detection result, and visually rendering the global three-dimensional point map to obtain a three-dimensional model of the tomato, and marking the defect on the three-dimensional model according to the spatial position information of the current defect in the defect detection result; Generate a tomato detection result based on the size detection result and the defect detection result, and generate a report in a preset format; Control information is generated according to the tomato detection result, and the control information is sent to a robotic arm on the assembly line for sorting.

10. A tomato detection system based on multi-view parallel 3D modeling, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the tomato detection method based on multi-view parallel three-dimensional modeling according to any one of claims 1 to 9 is implemented.

Images