Tomato Detection Method and System Based on Multi-View Parallel 3D Modeling

Abstract

This invention provides a tomato detection method and system based on multi-view parallel 3D modeling. The method includes: synchronously acquiring original images from N viewpoints on an assembly line, extracting field-of-view images, and assigning unique viewpoint codes; dividing each viewpoint image with a unique viewpoint code into multiple image blocks with unique block position codes, and using a hierarchical multi-view vision transformer to extract features from each viewpoint image layer by layer, followed by fusion to obtain a fused multi-view joint feature sequence; a global decoding head outputting a global 3D point map based on the multi-view joint features to calculate the tomato size detection result; a local decoding head outputting a local 3D point map based on the multi-view joint features, and obtaining the tomato defect detection result based on the local 3D point map and local semantic information to generate the tomato detection result. The beneficial effects of this invention are that it can achieve high real-time performance, high detection accuracy, and continuous processing for assembly line tomato detection.

Description

Technical Field

[0001] This invention relates to the field of agricultural product testing technology, and in particular to a tomato testing method and system based on multi-view parallel 3D modeling. Background Technology

[0002] In agricultural factory production lines, tomato appearance and size inspection are core steps in quality grading and packaging. Tomatoes with surface defects, rot, or that do not meet size standards will reduce consumers' willingness to buy and their eating experience, thus weakening market competitiveness. Therefore, efficient and accurate online inspection is crucial for improving the commercial quality of tomatoes and optimizing production line efficiency.

[0003] Current pipelined visual inspection technologies face bottlenecks: Single-view vision only obtains two-dimensional projections, lacks depth perception, and relies on geometric assumptions to infer three-dimensional structures, making it difficult to adapt to dynamic inspection; while binocular vision can calculate depth, smooth surfaces easily lead to mismatched feature points and distorted 3D reconstruction, and 3D parameters need to be indirectly derived, affecting accuracy. Furthermore, the traditional single / binocular vision frame-by-frame independent processing mode is inefficient and cannot match the high-speed cycle of pipelines. In terms of 3D reconstruction, traditional methods rely on multi-stage pipelines, resulting in computational complexity, error accumulation, and difficulty meeting real-time requirements; mainstream methods such as DUSt3R only support paired image inputs, and the computational load increases quadratically when processing dense multi-view sequences, consuming large amounts of memory and making it difficult to adapt to continuous operation.

[0004] Therefore, there is an urgent need to develop a streamlined tomato detection method that is highly real-time, has high detection accuracy, and can achieve continuous processing. Summary of the Invention

[0005] To address the aforementioned problems in the prior art, this invention provides a tomato detection method and system based on multi-view parallel 3D modeling, which can achieve high real-time performance, high detection accuracy, and continuous processing for pipeline tomato detection.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a tomato detection method based on multi-view parallel 3D modeling, comprising the following steps:

[0008] S1. Simultaneously acquire original images from N perspectives on the production line, extract the field of view image corresponding to a single tomato from each original image, and assign a unique perspective code to each field of view image.

[0009] S2. Divide each view image with a unique view encoding into multiple image blocks, assign a unique block position encoding to each image block, and use a hierarchical multi-view visual transformer to extract the features of each view image layer by layer to obtain the image block feature sequence of each view image.

[0010] S3. The unique viewpoint encoding is fused with the image block feature sequence to obtain a fused viewpoint feature sequence. The fused viewpoint features of all viewpoint images are fused by global attention to obtain a fused multi-view joint feature sequence. The features of each image block in the multi-view joint feature sequence include the local semantic information of the image in which it is located and the collaborative geometric information of other viewpoint images.

[0011] S4. The global decoding head outputs a global 3D point map of all view pixels in the reference coordinate system based on the multi-view joint features, and the local decoding head outputs a local 3D point map in the camera coordinate system corresponding to each view image based on the multi-view joint features.

[0012] S5. Calculate the tomato size detection result based on the global three-dimensional point map, and obtain the tomato defect detection result based on the local three-dimensional point map and the local semantic information;

[0013] S6. Generate tomato detection results based on the size detection results and the defect detection results.

[0014] The beneficial effects of this invention are as follows: A multi-view parallel reconstruction method, which extracts features from multiple perspective images of tomatoes and then fuses them, enables single forward propagation processing of multiple tomato images from various perspectives to complete 3D reconstruction of the tomato. This avoids the problems of quadratic increase in computational load, large memory consumption, and large error accumulation associated with traditional methods when processing dense multi-view sequences. It is suitable for continuous production line scenarios and improves the real-time performance and stability of 3D reconstruction. Furthermore, the decoupling of size measurement and defect localization through dual-branch point cloud decoding using a global and local decoding head improves the accuracy of tomato size measurement and defect detection. Therefore, it enables high real-time performance, high detection accuracy, and continuous processing for production line tomato detection.

[0015] Optionally, step S2 includes the following steps:

[0016] S21. Divide each view image with a unique view encoding into multiple image blocks;

[0017] S22. After generating initial features for each image block through linear projection, assign a unique block position code, and then stitch together the unique viewpoint code of the image in the field of view to generate an initial stage feature map containing the unique viewpoint code and the unique block position code.

[0018] S23. Perform multiple downsampling and transformer layer operations on each initial feature map in sequence to obtain multiple stage feature maps of different sizes.

[0019] S24. The initial stage feature map and multiple stage feature maps of different sizes are fused with high-level global features and low-level local features to generate a multi-scale enhanced feature map. The multi-scale enhanced feature map is then globally pooled to obtain the feature vector of each image block.

[0020] S25. Combine the feature vectors of each image patch in each viewpoint image to obtain the image patch feature sequence of each viewpoint image.

[0021] Optionally, the feature extraction for each image block in step S2 can be processed in parallel by the GPU.

[0022] Optionally, step S3 includes the following steps:

[0023] S31. The unique viewpoint code is fused with the image block feature sequence to obtain a fused viewpoint feature sequence;

[0024] S32. The fused viewpoint feature sequence is split into multiple heads, and attention is calculated for each head to obtain the single-head attention output of each head. Then, the single-head attention outputs of all heads are fused together to obtain the multi-head attention output.

[0025] S33. The multi-head attention output is sequentially processed by residual connection and normalization, convolutional feedforward neural network processing, and residual connection and normalization to obtain the output features of the current layer.

[0026] S34. After stacking and interacting multiple layers of steps S32 and S33, a fused multi-view joint feature sequence is obtained. The features of each image block in the multi-view joint feature sequence include the local semantic information of the image in which it is located and the cooperative geometric information of other images in the field of view.

[0027] Optionally, step S33 includes the following steps:

[0028] S331. The multi-head attention output is residually connected to the current layer input features to obtain a residual output, and the residual output is layer-normalized to obtain a normalized output. The calculation formulas for the residual connection and the layer normalization are as follows:

[0029] ResOut = AttnOut + H l-1 ;

[0030] NormAttn = BatchNorm(ResOut);

[0031] Where ResOut is the residual output, AttnOut is the multi-head attention output, and H... l-1The input features for the current layer are defined by NormAttn, the normalized output is defined by BatchNorm, and the normalization layer is defined by BatchNorm.

[0032] S332. Perform a convolution operation on the normalized output using a convolution kernel, and apply an activation function to introduce a nonlinear transformation, followed by a pooling operation to obtain the convolution feedforward output.

[0033] S333. Apply residual connection and normalization again to the convolutional feedforward output to obtain the output features of the current layer. The calculation formulas for the residual connection and normalization are as follows:

[0034] H l = BatchNorm(Pooled+ResOut);

[0035] Among them, H l The output features of the current layer are defined by NormAttn, the normalized output is defined by BatchNorm, the normalized layer is defined by Pooled, and the convolutional feedforward output is defined by Pooled.

[0036] Optionally, the reference coordinate system in step S4 is the camera coordinate system corresponding to the viewpoint image that presents the most tomato features among all viewpoint images.

[0037] Optionally, the step S5, which obtains the defect detection result of the tomato based on the local three-dimensional point map and the local semantic information, includes the following steps:

[0038] Defect detection is performed based on the local 3D point map and the local semantic information to obtain the current defect and the defect region of the current defect in the corresponding viewpoint image. Then, the local 3D point map is aligned to the reference coordinate system through the normal distribution transformation algorithm to obtain the spatial location information of the current defect, so as to obtain the defect detection result including the current defect and the defect region and spatial location information of the current defect.

[0039] Optionally, step S1 further includes the step:

[0040] The system simultaneously acquires original images from N viewpoints on the pipeline, performs geometric distortion correction and brightness normalization on each original image, and then uses a tomato-based precise segmentation network that combines residual backbone and pyramid pooling to extract the field of view image corresponding to a single tomato. A unique viewpoint code is assigned to each field of view image to generate an input tensor that includes field of view images from all viewpoints.

[0041] Optionally, step S6 includes the following steps:

[0042] Based on the defect detection results, the current defect and defect area are marked in the corresponding viewpoint image, and the global three-dimensional point map is visualized and rendered to obtain a three-dimensional model of the tomato. Based on the spatial location information of the current defect in the defect detection results, the three-dimensional model is labeled with defects.

[0043] The size detection results and the defect detection results are used to generate tomato detection results, and a report in a preset format is generated.

[0044] Based on the tomato detection results, control information is generated and sent to the robotic arm on the assembly line for sorting.

[0045] Secondly, the present invention provides a tomato detection system based on multi-view parallel 3D modeling, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the tomato detection method based on multi-view parallel 3D modeling of the first aspect.

[0046] The technical effects of the tomato detection system based on multi-view parallel 3D modeling provided in the second aspect are described in the relevant description of the tomato detection method based on multi-view parallel 3D modeling provided in the first aspect. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the main process of the tomato detection method based on multi-view parallel 3D modeling according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of a tomato-based precise segmentation network combining residual backbone and pyramid pooling, as described in an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram of a hierarchical multi-view visual transformer according to an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the multi-layer interaction of the encoder and decoder according to an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of a tomato detection system based on multi-view parallel 3D modeling, according to an embodiment of the present invention.

[0052] Explanation of reference numerals in the attached figures:

[0053] 1. Tomato detection system based on multi-view parallel 3D modeling;

[0054] 2. Processor;

[0055] 3. Memory. Detailed Implementation

[0056] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0057] Example 1

[0058] Please refer to Figure 1 A tomato detection method based on multi-view parallel 3D modeling includes the following steps:

[0059] S1. Simultaneously acquire the original images from N perspectives on the production line, extract the field of view image corresponding to a single tomato from each original image, and assign a unique perspective code to each field of view image.

[0060] Step S1 further includes the following steps:

[0061] Simultaneously acquire raw images from N viewpoints on the pipeline, perform geometric distortion correction and brightness normalization on each raw image, and then use a tomato-based precise segmentation network that combines residual backbone and pyramid pooling to extract the field of view image corresponding to a single tomato. Assign a unique viewpoint code to each field of view image to generate an input tensor that includes field of view images from all viewpoints.

[0062] In this embodiment, multiple sets of industrial cameras, such as 5 to 10 viewing angles, are deployed on both sides and above the production line to simultaneously capture RGB images of the tomatoes during transmission. The resolution of the industrial cameras is greater than 512×512. A uniform lighting system, such as a ring-shaped shadowless light source, is also integrated to reduce the impact of surface reflections on image quality.

[0063] At this time, refer to Figure 2 As can be seen, the preprocessing in step S1 of this embodiment includes the following steps:

[0064] S11. For the original images acquired from multiple perspectives on the production line, the pixel coordinates are normalized to the camera coordinate system according to the camera calibration parameters. Then, distortion correction formulas are applied pixel-by-pixel to eliminate radial and tangential distortions, restoring the image to the image coordinate system. The resulting orthophoto image after geometric distortion correction is output, restoring the true shape and spatial relationship of the tomato. The formulas involved are:

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] ;

[0070] ;

[0071] ;

[0072] The camera calibration parameters include the intrinsic parameter matrix K, first-order, second-order, and third-order radial distortion coefficients k1, k2, and k3, and first-order and second-order tangential distortion coefficients p1 and p2.

[0073] Where, x distorted and y distorted c represents the pixel coordinates before distortion. x c y Principal point coordinates, f x f y x is the focal length undistorted and y undistorted These are the pixel coordinates after distortion correction.

[0074] S12. For the geometrically corrected tomato image, zero-mean normalization is applied to calculate the global mean μ and standard deviation σ of the image. The brightness is adjusted pixel by pixel to reduce illumination interference and provide stable input for subsequent feature extraction. The formula is as follows:

[0075] ;

[0076] Where I(i,j) represents the original brightness value of the pixel at coordinates (i,j) in the tomato image after geometric correction, I norm (i,j) represents the new pixel value obtained after performing zero-mean normalization (Z-Score) on the original pixel value I(i,j). =10 -5 This is to prevent division by zero.

[0077] The above steps eliminate the interference of global illumination in the image, making the pixel brightness distribution conform to a standard normal distribution with a mean of 0 and a standard deviation of 1, thereby providing a stable input for subsequent feature extraction.

[0078] S13. The image I with dimensions of 512×512×3 and normalized brightness... norm First, initial convolution and pooling are performed, including 7×7 convolution with a stride of 2, followed by 3×3 max pooling with a stride of 2. Then, residual blocks are stacked, including 3 residual blocks, 4 residual blocks, 6 residual blocks, and 3 residual blocks. Finally, a 1 / 8 size feature map is obtained, denoted as F. backbone .

[0079] S14, F backbone Perform multi-scale pooling, where global average pooling is performed on F. backbone Calculate the global mean. 2×2 block pooling divides the feature map into 2×2 sub-blocks and performs average pooling on each block. 3×3 block pooling divides the feature map into 3×3 sub-blocks and performs average pooling on each block. 6×6 block pooling divides the feature map into 6×6 sub-blocks and performs average pooling on each block.

[0080] S15. For each pooling output, upsample it using bilinear interpolation to restore it to the backbone network feature map size, then compress the channels using a 1×1 convolution to obtain the multi-scale feature map F. pool,1 ,F pool,2 ,F pool,3 ,F pool,4 .

[0081] S16. Assemble the original backbone network feature map F backbone Combined with multi-scale pooling features, the fused feature F is obtained. concat .

[0082] S17. The features are compressed using a 3×3 convolution with 512 channels, and then upsampled by ×8 to restore the feature map size to the input image level of 512×512, outputting F. decode .

[0083] S18. Decode the feature map F decode Perform a 1×1 convolution to compress the channels to 1, and output a single-channel feature map F. single Then, after Sigmoid activation, the feature values ​​are mapped to [0,1] to distinguish the high-value tomato region from the low-value background. The formula is:

[0084] ;

[0085] Finally, a threshold τ is set, and the results are evaluated pixel by pixel. A 512×512 binary mask is output, where the regions with a value of 1 are the tomato ROIs.

[0086] .

[0087] S2. Divide each view image with a unique view encoding into multiple image blocks, assign a unique block position encoding to each image block, and use a hierarchical multi-view visual transformer to extract the features of each view image layer by layer to obtain the image block feature sequence of each view image.

[0088] In this embodiment, the feature extraction for each image block in step S2 can be processed in parallel by the GPU, and together with the independent calculation of multi-head attention in subsequent steps, it can be used to realize the real-time processing of dozens of tomatoes per second in the pipeline, which can improve the real-time performance and scalability of tomato appearance and size detection.

[0089] Among them, such as Figure 3 As shown, step S2 includes the following steps:

[0090] S21. Divide each view image with a unique view encoding into multiple image blocks.

[0091] In this embodiment, the number of image blocks is H*W / p 2 =512*512 / 16, the image patch set is:

[0092] ;

[0093] Among them, I i,j This is represented as the image patch in row i and column j.

[0094] S22. After generating initial features for each image block through linear projection, assign a unique block position code, and then stitch together the unique viewpoint code of the image in the field of view to generate an initial stage feature map containing the unique viewpoint code and the unique block position code.

[0095] In this process, after generating initial features for each image patch through linear projection, its two-dimensional positional encoding in a single image is first incorporated, such as row and column coordinate encoding, and then combined with the unique viewpoint encoding p corresponding to that viewpoint. i Perform feature concatenation or element-wise addition to generate an initial feature token containing viewpoint information and local location information.

[0096] Among them, the unique perspective encoding p i Used to identify the spatial orientation of the view, such as p1 for camera 1 and p2 for camera 2. It is globally unique and identifies the spatial position of different cameras.

[0097] Among them, the unique block location encoding is unique within a single viewpoint and is used to identify the position of an image block in a single image, helping the model understand the local spatial relationships of image blocks. For example, the top and bottom image blocks of a tomato can be associated vertically through location encoding. The combination of these two methods enables dual localization capabilities: cross-viewpoint spatial localization and single-viewpoint local location awareness.

[0098] Subsequently, through the transformer layer, local correlations are modeled within image patches and between adjacent image patches, outputting the initial stage feature map, i.e. Figure 3 The mid-stage 1 feature map has a size of H / 4*W / 4.

[0099] S23. Perform multiple downsampling and transformer layer operations on each initial feature map in sequence to obtain multiple stage feature maps of different sizes.

[0100] like Figure 3 As shown, the feature map of stage 1 is compressed to H / 8*W / 8 by a downsampling module while retaining key features. Then, a transformer layer is used to expand the attention range and fuse features from a wider region, outputting the feature map of stage 2 with a size of H / 8*W / 8.

[0101] Then, the feature map of stage 2 is repeatedly downsampled and transformed to compress the feature map size to H / 16×W / 16. The global context information is further aggregated to output the feature map of stage 3 with a size of H / 16×W / 16.

[0102] Finally, the feature map of stage 3 is downsampled and transformed at the last time to compress the feature map size to H / 32×W / 32, completing the layer-by-layer extraction of features from local to global, and outputting the feature map of stage 4.

[0103] In this embodiment, the multiple stage feature maps of different sizes include a stage 2 feature map with a size of H / 8*W / 8, a stage 3 feature map with a size of H / 16×W / 16, and a stage 4 feature map with a size of H / 32×W / 32. In other embodiments, the number of stage feature maps of different sizes can be appropriately increased or decreased.

[0104] S24. The initial stage feature map and multiple stage feature maps of different sizes are fused with high-level global features and low-level local features to generate a multi-scale enhanced feature map. The multi-scale enhanced feature map is then globally pooled to obtain the feature vector of each image block.

[0105] like Figure 3 As shown, the multi-scale feature maps output from stages 1-4 are fused with the high-level global features of stage 4 and the low-level local features of stages 1-3 through the top-down path and lateral connections of the Feature Pyramid Network (FPN) algorithm, generating a multi-scale enhanced feature map. Then, the multi-scale feature maps fused by the Feature Pyramid Network algorithm are globally pooled to obtain the feature vector for each image patch.

[0106] Therefore, by extracting features in a hierarchical multi-view domain, the surface texture and overall shape information of the tomato are preserved, the integrity of the point cloud is improved, and the 3D reconstruction of the tomato is more complete.

[0107] S25. Combine the feature vectors of each image patch in each viewpoint image to obtain the image patch feature sequence of each viewpoint image.

[0108] like Figure 3 As shown, the two-dimensional feature map is compressed into a one-dimensional vector; then, through linear layer mapping, a 1024-dimensional feature vector is finally output.

[0109] S3. The unique viewpoint encoding is fused with the image patch feature sequence to obtain the fused viewpoint feature sequence. The fused viewpoint features of all viewpoint images are fused through global attention to obtain the fused multi-view joint feature sequence. The features of each image patch in the multi-view joint feature sequence include the local semantic information of the image in which it is located and the collaborative geometric information of other viewpoint images.

[0110] Step S3 includes the following steps:

[0111] S31. The unique viewpoint code is fused with the image patch feature sequence to obtain the fused viewpoint feature sequence.

[0112] Wherein, the generated viewpoint identifier encoding vector P=[p1,p2,…,p N ], p i and the feature sequence H of the i-th view i Each image patch feature h i,j By adding element by element, we obtain the feature H of the fused perspective information. i '=h i,j +p i This ensures that the model can distinguish the spatial location of different views.

[0113] Wherein, let H be the image patch feature sequence generated after feature encoding of the i-th viewpoint image. i ={h i,1 ,h i,2 ,…,h i,M}, where h i,j ∈R 1024 Let M represent the feature vector of the j-th image patch in the i-th view, and M be the total number of image patches. In this embodiment, when the 512×512 image is divided into 16×16 segments, M=1024.

[0114] S32. The fused viewpoint feature sequence is split into multiple heads, and attention is calculated for each head to obtain the single-head attention output of each head. Then, the single-head attention outputs of all heads are fused together to obtain the multi-head attention output.

[0115] Among them, the input feature H l-1 Split into M headers. First, for each header, H... l-1 The mapping is to a query vector Q, a key vector K, and a value vector V, i.e., Q = HW. Q K=HW K V=HW VNext, calculate the dot product similarity between the query and the key: Score = (Q * K) T ) / In the formula d k The key vector dimension is used; and the attention weights are obtained through Softmax normalization: Atten = Softmax(Score). Then, the value vector is weighted and summed according to the attention weights to obtain the single-head output Head. m =Atten*V, finally multi-head fusion, concatenating the outputs of M heads and passing them through a linear layer to obtain the attention output AttnOut=Concat(Head1,…,Head M )*W O .

[0116] S33. The multi-head attention output is sequentially processed through residual connection and normalization, convolutional feedforward neural network processing, and residual connection and normalization to obtain the output features of the current layer.

[0117] Step S33 includes the following steps:

[0118] S331. Perform a residual connection between the multi-head attention output and the current layer input features to obtain a residual output. Then, perform layer normalization on the residual output to obtain a normalized output. The calculation formulas for residual connection and layer normalization are as follows:

[0119] ResOut = AttnOut + H l-1 ;

[0120] NormAttn = BatchNorm(ResOut);

[0121] Where ResOut is the residual output, AttnOut is the multi-head attention output, and H... l-1 The input features for the current layer are defined by NormAttn, the normalized output is defined by BatchNorm, and the normalization layer is defined by BatchNorm.

[0122] S332. Perform convolution operation on the normalized output using a convolution kernel, apply an activation function to introduce a nonlinear transformation, and then perform pooling operation to obtain the convolution feedforward output.

[0123] The calculation formulas for convolution, nonlinear transformation, and pooling operations are as follows:

[0124] ;

[0125] ;

[0126] .

[0127] The above formulas are commonly used in existing models and will not be elaborated here.

[0128] S333. Apply residual connection and normalization again to the convolutional feedforward output to obtain the output features of the current layer. The calculation formulas for residual connection and normalization are as follows:

[0129] H l = BatchNorm(Pooled+ResOut);

[0130] Among them, H l The output features of the current layer are defined by NormAttn, the normalized output is defined by BatchNorm, the normalized layer is defined by Pooled, and the convolutional feedforward output is defined by Pooled.

[0131] S34. After stacking and interacting multiple layers of steps S32 and S33, a fused multi-view joint feature sequence is obtained. The features of each image block in the multi-view joint feature sequence include the local semantic information of the image in which it is located and the collaborative geometric information of other images in the field of view.

[0132] like Figure 4 As shown, by stacking multiple interaction layers in the encoders of steps S32 and S33, the semantic association and geometric constraints of cross-view features are gradually enhanced. The lower-level interaction layers mainly capture local feature correspondences, such as tomato surface texture matching, while the higher-level interaction layers focus on global structural alignment, such as the 3D shape modeling of the entire fruit. Therefore, the 18-layer stacked interaction structure is also adopted in the subsequent decoder.

[0133] In this embodiment, block computing is used to divide long sequence features into small blocks for attention calculation, avoiding memory overflow and supporting parallel processing of thousands of views.

[0134] Thus, after 18 layers of interaction, the encoder outputs a multi-view joint feature sequence Z=H that fuses all view information. L Each image patch features z i,j This includes local semantic information from this view and collaborative geometric information from other views; the former includes peel color, spots, etc., while the latter includes cross-view 3D coordinate consistency. This feature sequence is subsequently input into step S4 to generate global and local 3D coordinate mappings.

[0135] S4. The global decoding head outputs a global 3D point map of all view pixels in the reference coordinate system based on the joint features of multiple views, and the local decoding head outputs a local 3D point map in the camera coordinate system corresponding to each view image based on the joint features of multiple views.

[0136] Specifically, step S4 includes the following steps:

[0137] S41. Using the camera coordinate system corresponding to the viewpoint image that exhibits the most tomato features among all viewpoint images as the reference coordinate system, output a global 3D point map X of all viewpoint pixels in the reference coordinate system through the global decoding head. G This is used to calculate global dimensional parameters such as tomato volume and diameter. For each view, a 3D point map X in its camera coordinate system is output through a local decoding head. i .

[0138] That is, the reference coordinate system in step S4 is the camera coordinate system corresponding to the viewpoint image that presents the most tomato features among all viewpoint images. Specifically, it can correspond to the image captured by one of the N industrial cameras deployed in the production line. Usually, the viewpoint that can fully present the main features of the tomato is selected as the first viewpoint. For example, if camera 1 is deployed directly above the production line and captures an image of the top of the tomato, then the camera coordinate system I1 is the image of that top viewpoint. The camera coordinate system I1 is then used as the global unified coordinate system, i.e., the reference coordinate system; other views (I2, I3, ..., I... N The 3D point cloud needs to be aligned to this reference coordinate system to achieve a unified expression of multi-view 3D information.

[0139] S42. Synchronously output a confidence map to characterize the reliability of each 3D coordinate prediction and suppress outliers with low confidence.

[0140] S43. By aligning the local point map to the global coordinate system through the normal distribution transformation algorithm, the detail accuracy of defect location is improved.

[0141] S5. Calculate the tomato size detection result based on the global 3D point map, and obtain the tomato defect detection result based on the local 3D point map and local semantic information.

[0142] Step S5, which involves obtaining the tomato defect detection results based on the local 3D point map and local semantic information, includes the following steps:

[0143] Defect detection is performed based on local 3D point maps and local semantic information to obtain the current defect and its defect region in the corresponding viewpoint image. Then, the local 3D point maps are aligned to the reference coordinate system using a normal distribution transformation algorithm to obtain the spatial location information of the current defect, thus obtaining a defect detection result that includes the current defect, its defect region, and its spatial location information.

[0144] Specifically:

[0145] Based on the global 3D point map X G Calculate the 3D bounding box of the tomato, extract geometric parameters such as major axis length, volume, and surface area, and estimate the fruit fullness using point cloud density.

[0146] X local 3D point mapi The coordinate information, such as surface curvature and depth, is combined with semantic information, such as color and texture, from image features H. This information is then input into a defect classification head. Through an anomaly detection algorithm, defect regions, such as cracks and decay, are labeled on the 2D image and mapped to a global 3D point map X. G To determine spatial location.

[0147] Based on the preset size threshold and defect confidence threshold, the test results are output, and the test results are divided into qualified / unqualified and defect type.

[0148] S6. Generate tomato inspection results based on size inspection results and defect inspection results.

[0149] Step S6 includes the following steps:

[0150] S61. Based on the defect detection results, mark the current defect and defect area in the corresponding viewpoint image, and perform visualization rendering of the global three-dimensional point map to obtain the three-dimensional model of the tomato. Based on the spatial location information of the current defect in the defect detection results, mark the defect in the three-dimensional model.

[0151] In other words, this embodiment can achieve visual rendering.

[0152] S62. Generate tomato inspection results from the size inspection results and defect inspection results, and generate a report in a preset format.

[0153] In other words, this embodiment can achieve data formatting.

[0154] S63. Generate control information based on the tomato detection results and send the control information to the robotic arm on the assembly line for sorting.

[0155] This embodiment enables the processing of tomatoes through external interaction, and is suitable for high-throughput real-time detection in the sorting, grading and quality control of tomatoes after harvesting.

[0156] Therefore, the advantages of this embodiment are:

[0157] 1. A multi-view parallel reconstruction method is proposed, which extracts features from multiple perspective images of tomatoes and then fuses them. This method enables the processing of multiple tomato images from multiple perspectives using a single forward propagation, thereby completing the 3D reconstruction of tomatoes. This avoids the problems of quadratic increase in computation, large memory consumption, and large error accumulation when the traditional method processes dense multi-view sequences. It is suitable for continuous operation scenarios on production lines and improves the real-time performance and stability of 3D reconstruction.

[0158] 2. By using a hierarchical multi-view vision transformer, the surface texture and overall shape information of the tomato are preserved, making the 3D reconstruction of the tomato more complete. By using a dual-branch point cloud decoding of a global decoding head and a local decoding head, the size measurement and defect localization are decoupled, avoiding feature interference between size measurement and defect detection, improving the stability of global geometric parameter calculation and the sensitivity of local defect detection, thereby improving the measurement accuracy of tomato size and the accuracy of defect detection.

[0159] 3. By using block attention computation and GPU parallel scheduling, the parallelism of the model and the usage of video memory are optimized, adapting to the hardware environment of industrial production lines. This enables real-time processing of multi-view images of dozens of tomatoes per second in the production line, matching the high-speed production cycle, reducing the hardware threshold for large-scale multi-view (5-10) parallel computing, and supporting the industrial-grade deployment and expansion of the system.

[0160] In other words, this embodiment can achieve a streamlined tomato detection process with high real-time performance, high detection accuracy, and continuous processing.

[0161] Example 2

[0162] Please refer to Figure 5 The tomato detection system 1 based on multi-view parallel 3D modeling includes a memory 3, a processor 2, and a computer program stored on the memory 3 and run on the processor 2. When the processor 2 executes the computer program, it implements the steps in the above embodiment 1.

[0163] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and modifications of the systems / devices based on the methods described in the above embodiments of the present invention, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0164] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0165] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0166] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0167] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0168] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0169] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A tomato detection method based on multi-view parallel 3D modeling, characterized in that, Including the following steps: S1. Simultaneously acquire original images from N perspectives on the production line, extract the field of view image corresponding to a single tomato from each original image, and assign a unique perspective code to each field of view image. S2. Divide each view image with a unique view encoding into multiple image blocks, assign a unique block position encoding to each image block, and use a hierarchical multi-view visual transformer to extract the features of each view image layer by layer to obtain the image block feature sequence of each view image. S3. The unique viewpoint encoding is fused with the image block feature sequence to obtain a fused viewpoint feature sequence. The fused viewpoint features of all viewpoint images are fused by global attention to obtain a fused multi-view joint feature sequence. The features of each image block in the multi-view joint feature sequence include the local semantic information of the image in which it is located and the collaborative geometric information of other viewpoint images. S4. The global decoding head outputs a global 3D point map of all view pixels in the reference coordinate system based on the multi-view joint features, and the local decoding head outputs a local 3D point map in the camera coordinate system corresponding to each view image based on the multi-view joint features. S5. Calculate the tomato size detection result based on the global three-dimensional point map, and obtain the tomato defect detection result based on the local three-dimensional point map and the local semantic information; S6. Generate tomato detection results based on the size detection results and the defect detection results; Step S2 includes the following steps: S21. Divide each view image with a unique view encoding into multiple image blocks; S22. After generating initial features for each image block through linear projection, assign a unique block position code, and then stitch together the unique viewpoint code of the image in the field of view to generate an initial stage feature map containing the unique viewpoint code and the unique block position code. S23. Perform multiple downsampling and transformer layer operations on each initial feature map in sequence to obtain multiple stage feature maps of different sizes. S24. The initial stage feature map and multiple stage feature maps of different sizes are fused with high-level global features and low-level local features to generate a multi-scale enhanced feature map. The multi-scale enhanced feature map is then globally pooled to obtain the feature vector of each image block. S25. Combine the feature vectors of each image patch in each viewpoint image to obtain the image patch feature sequence of each viewpoint image; Step S3 includes the following steps: S31. The unique viewpoint code is fused with the image block feature sequence to obtain a fused viewpoint feature sequence; S32. The fused viewpoint feature sequence is split into multiple heads, and attention is calculated for each head to obtain the single-head attention output of each head. Then, the single-head attention outputs of all heads are fused together to obtain the multi-head attention output. S33. The multi-head attention output is sequentially processed by residual connection and normalization, convolutional feedforward neural network processing, and residual connection and normalization to obtain the output features of the current layer. S34. After stacking and interacting multiple layers of steps S32 and S33, a fused multi-view joint feature sequence is obtained. The features of each image block in the multi-view joint feature sequence include the local semantic information of the image in which it is located and the cooperative geometric information of other images in the field of view.

2. The tomato detection method based on multi-view parallel 3D modeling according to claim 1, characterized in that, In step S2, feature extraction for each image block can be processed in parallel by the GPU.

3. The tomato detection method based on multi-view parallel 3D modeling according to claim 1, characterized in that, Step S33 includes the following steps: S331. The multi-head attention output is residually connected to the current layer input features to obtain a residual output, and the residual output is layer-normalized to obtain a normalized output. The calculation formulas for the residual connection and the layer normalization are as follows: ResOut=AttnOut+H l-1 ; NormAttn = BatchNorm(ResOut); Where ResOut is the residual output, AttnOut is the multi-head attention output, and H... l-1 The input features for the current layer are defined by NormAttn, the normalized output is defined by BatchNorm, and the normalization layer is defined by BatchNorm. S332. Perform a convolution operation on the normalized output using a convolution kernel, and apply an activation function to introduce a nonlinear transformation, followed by a pooling operation to obtain the convolution feedforward output. S333. Apply residual connection and normalization again to the convolutional feedforward output to obtain the output features of the current layer. The calculation formulas for the residual connection and normalization are as follows: H l = BatchNorm(Pooled+ResOut); Among them, H l The output features of the current layer are defined by NormAttn, the normalized output is defined by BatchNorm, the normalized layer is defined by Pooled, and the convolutional feedforward output is defined by Pooled.

4. The tomato detection method based on multi-view parallel 3D modeling according to claim 1, characterized in that, The reference coordinate system in step S4 is the camera coordinate system corresponding to the viewpoint image that presents the most tomato features among all viewpoint images.

5. The tomato detection method based on multi-view parallel 3D modeling according to claim 1, characterized in that, Step S5, which obtains the defect detection result of the tomato based on the local three-dimensional point map and the local semantic information, includes the following steps: Defect detection is performed based on the local 3D point map and the local semantic information to obtain the current defect and the defect region of the current defect in the corresponding viewpoint image. Then, the local 3D point map is aligned to the reference coordinate system through the normal distribution transformation algorithm to obtain the spatial location information of the current defect, so as to obtain the defect detection result including the current defect and the defect region and spatial location information of the current defect.

6. The tomato detection method based on multi-view parallel 3D modeling according to any one of claims 1 to 5, characterized in that, Step S1 further includes the following step: The system simultaneously acquires original images from N viewpoints on the pipeline, performs geometric distortion correction and brightness normalization on each original image, and then uses a tomato-based precise segmentation network that combines residual backbone and pyramid pooling to extract the field of view image corresponding to a single tomato. A unique viewpoint code is assigned to each field of view image to generate an input tensor that includes field of view images from all viewpoints.

7. The tomato detection method based on multi-view parallel 3D modeling according to claim 1, characterized in that, Step S6 includes the following steps: Based on the defect detection results, the current defect and defect area are marked in the corresponding viewpoint image, and the global three-dimensional point map is visualized and rendered to obtain a three-dimensional model of the tomato. Based on the spatial location information of the current defect in the defect detection results, the three-dimensional model is labeled with defects. The size detection results and the defect detection results are used to generate tomato detection results, and a report in a preset format is generated. Based on the tomato detection results, control information is generated and sent to the robotic arm on the assembly line for sorting.

8. A tomato detection system based on multi-view parallel 3D modeling, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the tomato detection method based on multi-view parallel 3D modeling as described in any one of claims 1 to 7.

Images