A crab quality full-automatic detection method and system based on multi-modal image processing

Abstract

The application discloses a kind of crab quality full-automatic detection method and system based on multi-modal image processing, it is related to crab quality detection technical field, comprising: obtaining the continuous top image sequence of target crab and the foot pressure distribution corresponding to each frame top image;The body coordinate system of crab is established to each frame top image;To each frame top image, the preset activity area of each leg of target crab is extracted in the reference point of corresponding body coordinate system, and the visual confidence sequence is extracted;Based on body coordinate system, foot pressure distribution and preset space calibration matrix, generate the ground contact strength sequence;Based on visual confidence sequence and ground contact strength sequence, whether the leg missing of target crab exists by preset gait rule representing expected ground contact window of each leg in gait cycle is judged;Its beneficial effect is: using the dual missing of visual and pressure signal in expected ground contact window as judging basis, improve the reliability of crab leg integrity detection.

Description

Technical Field

[0001] This invention relates to the field of crab quality inspection technology, and in particular to a fully automated method and system for crab quality inspection based on multimodal image processing. Background Technology

[0002] Crab quality grading is a key process in aquaculture, processing and supply chain. Among them, the integrity of the crab's legs, such as whether they are missing, broken or damaged, is one of the important factors in determining the grade of the crab.

[0003] With the improvement of automation in the aquatic product industry, more and more companies are trying to adopt automated crab quality inspection solutions based on image processing. This involves using cameras to capture images of the crab's appearance and identify the number of its legs and appearance defects. However, because crab shells are wide laterally and the legs are distributed in a complex manner, their legs are often obscured by the shell when they move naturally, retract their bodies, or are stimulated by external factors. This results in some leg segments not being visible at the moment of shooting. Even with multi-angle camera setups, it is difficult to ensure that all legs are clearly visible at any given time.

[0004] Furthermore, in production environments such as inside boxes or on assembly lines, crabs often do not maintain a standard posture. They frequently exhibit actions such as curling up, crossing, sticking to the shell, or raising their legs, resulting in incomplete and complex leg images. These factors can significantly interfere with traditional image-based algorithms, easily leading to missed detections or misjudgments of leg defects, making it difficult to meet the requirements for detection stability and reliability in the crab quality grading process. Summary of the Invention

[0005] This invention is made to solve the above-mentioned problems, and aims to provide a fully automatic crab quality detection method and system based on multimodal image processing. It can use the dual absence of visual and pressure signals within the expected ground contact window as a judgment criterion to improve the reliability of crab leg integrity detection.

[0006] This invention provides a fully automated crab quality detection method based on multimodal image processing, characterized by the following steps: Step 1, acquiring a continuous sequence of top images of the target crab within a preset time period and the foot pressure distribution corresponding to each frame of the top image; Step 2, establishing a body coordinate system for locating the spatial position of the legs for each frame of the top image; Step 3, extracting visual confidence scores for each frame of the top image within a preset activity area of ​​each leg of the target crab, with the origin of the corresponding body coordinate system as a reference, to obtain a visual confidence score sequence reflecting the visibility of each leg of the target crab at each time point; Step 4, generating... Step 5: Based on the visual confidence sequence and the ground contact intensity sequence, determine whether the target crab has missing legs by using a preset gait rule that represents the expected ground contact window of each leg in the crab's gait cycle. The determination of whether there is missing legs includes: for any leg, when it is within the expected ground contact time window, if the corresponding visual confidence is lower than the first preset threshold and the ground contact intensity is lower than the second preset threshold, then evidence of missing legs is generated; if the target crab has evidence of missing legs in all expected ground contact time windows within a preset time period, then it is determined that the target crab has missing legs.

[0007] Preferably, in step 2, the body coordinate system is a two-dimensional coordinate system. The establishment of the body coordinate system includes: identifying the shell width direction and geometric center of the target crab in the top image; establishing the body coordinate system based on the shell width direction, with the origin of the body coordinate system as the geometric center, and the horizontal axis of the body coordinate system parallel to the shell width direction.

[0008] Preferably, step 3, extracting visual confidence specifically includes: determining the preset activity area of ​​each leg in the body coordinate system based on the preset crab eight-leg topological distribution model; detecting the number of joints of the leg segments within the preset activity area based on the preset posture estimation model, and extracting the coordinate position and position confidence of each joint; calculating the skeletal matching degree between the leg skeleton shape formed by connecting each joint and the preset standard leg skeleton template based on the coordinate position of each joint; and weighting and fusing the skeletal matching degree and the position confidence of each joint according to the preset weight coefficient to obtain the visual confidence.

[0009] Preferably, in step 4, generating the ground contact intensity bound to the corresponding leg of the target crab in each frame of the top image includes: mapping the foot pressure distribution corresponding to each frame of the top image to the preset activity area of ​​each leg of the target crab according to the body coordinate system and the spatial calibration matrix, so as to obtain the ground contact intensity bound to the corresponding leg of the target crab.

[0010] Preferably, in step 5, determining whether a leg is missing further includes:

[0011] For any leg, when it is within the expected ground contact time window, if the difference between the ground contact strength of the leg and the ground contact strength of the symmetrical leg is always greater than the third preset threshold, then missing inference evidence is generated for the leg with the smaller ground contact strength.

[0012] Preferably, determining that the target crab has missing legs includes: counting the number of inferences about the missing legs of each leg of the target crab within a preset time period; when the number of inferences about the missing legs of any one leg is greater than a preset threshold, it is determined that the target crab has missing legs.

[0013] Preferably, determining that the target crab has missing legs also includes: counting the number of expected ground contact time windows for each leg within a preset time period; and adjusting a preset quantity threshold based on the ratio of the number of expected ground contact time windows to a preset standard quantity.

[0014] This invention provides a fully automated crab quality detection system based on multimodal image processing, characterized by: an acquisition module for acquiring a continuous sequence of top images of a target crab within a preset time period and the foot pressure distribution corresponding to each frame of the top image; a coordinate system construction module for establishing a body coordinate system for locating the spatial position of the legs in each frame of the top image; a visual processing module for extracting visual confidence scores from each frame of the top image within a preset activity area of ​​each leg of the target crab, with the origin of the corresponding body coordinate system as a reference, to obtain a visual confidence score sequence reflecting the visibility of each leg of the target crab at each time point; and a ground contact strength binding module for binding the system based on the body coordinate system, foot pressure distribution, and preset spatial position of the legs. An inter-calibration matrix is ​​used to generate the ground contact intensity associated with the corresponding leg of the target crab in the top image of each frame, resulting in a ground contact intensity sequence. A decision module is used to determine whether the target crab has missing legs based on the visual confidence sequence and the ground contact intensity sequence, using preset gait rules representing the expected ground contact window of each leg in the crab's gait cycle. Determining whether a leg is missing includes: for any leg, if the corresponding visual confidence is lower than a first preset threshold and the ground contact intensity is lower than a second preset threshold when it is within the expected ground contact time window, then evidence of the leg's absence is generated; if, within a preset time period, there is evidence of the leg's absence in all expected ground contact time windows, then the target crab is determined to have a missing leg.

[0015] The present invention provides an electronic device, including a memory and a processor. The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the method described above.

[0016] The present invention provides a computer storage medium storing computer-executable instructions thereon, which, when executed by a processor, implement the steps of the method described above.

[0017] Technical effect

[0018] Compared with existing technologies, the fully automatic crab quality detection method and system based on multimodal image processing according to the present invention improves the reliability of crab leg integrity detection by fusing top visual information and bottom pressure information, and combining the inherent rhythmic characteristics of crab's sideways gait, and using the dual absence of visual and pressure signals within the expected ground contact window as the judgment criterion. Compared with traditional visual solutions, it improves the detection accuracy. Attached Figure Description

[0019] The above and other objects, features, and advantages of this application will become more apparent from the following detailed description of the embodiments in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain the application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 This is a flowchart of the fully automated crab quality detection method based on multimodal image processing in Embodiment 1 of the present invention;

[0021] Figure 2 This is a block diagram of the fully automated crab quality detection system based on multimodal image processing in Embodiment 2 of the present invention;

[0022] Figure 3 This is a block diagram of an electronic device according to Embodiment 3 of the present invention; Detailed Implementation

[0023] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate a fully automated crab quality detection method and system based on multimodal image processing.

[0024] Example 1

[0025] This embodiment provides a fully automated method for detecting crab quality based on multimodal image processing.

[0026] Figure 1 This is a flowchart of a fully automated crab quality detection method based on multimodal image processing, as described in an embodiment of the present invention.

[0027] like Figure 1 As shown, the fully automated crab quality detection method based on multimodal image processing in this embodiment includes the following steps:

[0028] Step S1: Obtain a continuous sequence of top images of the target crab within a preset time period and the foot pressure distribution corresponding to each frame of the top image.

[0029] In this embodiment, a dual-camera synchronous acquisition scheme is adopted: a high-speed camera at the top records the visual information of the crab's movement at a frame rate of 60-240fps, while a pressure imaging camera at the bottom synchronously records the pressure distribution on the crab's feet. The two cameras achieve timestamp alignment through hardware trigger signals, ensuring that the synchronization error is less than 1 frame.

[0030] The preset time period T is usually set to 0.5-2 seconds, which is sufficient to cover 1-4 complete gait cycles of crabs. The typical sideways gait cycle of crabs is about 0.3-0.6 seconds, so multiple complete ground-swing cycles can be observed within the preset time period, providing sufficient time-series data support for subsequent gait rhythm analysis.

[0031] The above-mentioned dual-modal synchronous acquisition scheme obtained continuous multimodal time-series data during the crab's movement, laying a data foundation for subsequent leg loss detection.

[0032] Step S2: For each frame of the top image, establish a body coordinate system for locating the spatial position of the legs.

[0033] In this embodiment, the body coordinate system is a two-dimensional coordinate system, which is established by making full use of the morphological characteristics of crabs. Unlike other animals, the width of a crab's shell is significantly greater than its length, a feature that provides a reliable basis for identifying the crab's lateral orientation. Specifically, the establishment of the body coordinate system includes the following steps:

[0034] First, identify the shell width direction and geometric center of the target crab in the top image. The shell width direction can be determined through image segmentation and principal component analysis, while the geometric center is the centroid location of the shell region.

[0035] Establish a body coordinate system based on the width direction of the shell, with the origin of the body coordinate system as the geometric center, and the horizontal axis of the body coordinate system parallel to the width direction of the shell.

[0036] By establishing a body coordinate system, the observations in the image coordinate system are transformed into a local coordinate system with the crab's body as the reference, so that subsequent leg positioning and gait analysis are not affected by changes in the crab's overall position and orientation.

[0037] It should be noted that after establishing the body coordinate system, preprocessing filtering can be further performed using the angular constraints of the crab's legs. The joints of the crab, such as the coxae, ischia, and carpus, have strict biomechanical angular range limitations. If the detected leg segments do not meet these angular constraints, they can be identified as artifacts and discarded, thereby improving the reliability of subsequent processing.

[0038] Step S3: For each frame of the top image, extract the visual confidence score within the preset active area of ​​each leg of the target crab with the origin of the corresponding body coordinate system as the reference, and obtain the visual confidence score sequence reflecting the visibility of each leg of the target crab at each time point.

[0039] The purpose of this step is to quantify the visibility of each leg from the top viewpoint. Because crabs' legs may obscure each other or swing rapidly when they move sideways, it is difficult to accurately determine the leg status by simply relying on joint point detection. Therefore, it is necessary to construct a confidence index by integrating multiple visual features.

[0040] In this embodiment, the specific process for extracting visual confidence is as follows:

[0041] First, based on a pre-defined topological distribution model of the crab's eight legs, the pre-defined activity area of ​​each leg in the body coordinate system is determined. Crabs have eight walking legs, symmetrically distributed on both sides of the body, and each leg's range of motion during normal movement is subject to certain biomechanical constraints. Using the pre-defined topological model, a fan-shaped or rectangular pre-defined activity area can be defined for each leg.

[0042] Then, based on a pre-defined pose estimation model, the number of joints in the leg segment is detected within a pre-defined activity area, and the coordinates and confidence scores of each joint are extracted. The pose estimation model can employ a deep learning-based keypoint detection network, outputting two-dimensional coordinates of the joints and their corresponding confidence scores.

[0043] Next, the skeletal matching degree between the leg skeleton formed by connecting each joint and the preset standard leg skeleton template is calculated based on the coordinate positions of each joint. The skeletal matching degree can be obtained by calculating the shape similarity between the detected skeleton and the standard template, for example, by using Procrustes analysis or a distance-based shape matching algorithm.

[0044] Finally, the skeleton matching score and the position confidence scores of each joint are weighted and fused according to preset weight coefficients to obtain the visual confidence score:

[0045]

[0046] In the formula, , representing the normalized skeleton matching degree. , representing the average confidence level of the position of each key point. This represents the position confidence score of the j-th joint. n represents the number of joints. and The sum is 1, ensuring that the visual confidence level after fusion is also within the range of 1. interval, and The optimal method was determined based on experimental data. The value is 0.6. The value is 0.4, which focuses more on the matching degree of the overall skeleton shape.

[0047] Using the above method, a visual confidence sequence reflecting the visibility of each leg of the target crab at each time point was obtained. , where i represents the leg number and t represents the time point. The higher the visual confidence level, the more reliable the visual information of that leg at that moment.

[0048] Step S4: Based on the body coordinate system, foot pressure distribution and preset spatial calibration matrix, generate the ground contact intensity that is bound to the corresponding leg of the target crab in the top image of each frame, and obtain the ground contact intensity sequence.

[0049] The purpose of this step is to correlate the pressure distribution information collected by the bottom pressure camera with the leg position from the top viewpoint to achieve cross-modal spatial alignment.

[0050] In this embodiment, the process of generating the ground contact strength associated with the corresponding leg of the target crab in each frame's top image is as follows:

[0051] Based on the body coordinate system and spatial calibration matrix, the foot pressure distribution corresponding to each frame of the top image is mapped to the preset activity area of ​​each leg of the target crab, thus obtaining the ground contact intensity bound to the corresponding leg of the target crab.

[0052] Specifically, spatial calibration matrix The projection transformation from the top visual coordinate system to the bottom pressure coordinate system is described, and can be calculated using corresponding points on the calibration plate in the two viewpoints. This applies to the leg positions detected in the top image. Mapped to the bottom coordinate system via a calibration matrix :

[0053]

[0054] At the mapped bottom coordinates, extract the pressure intensity value of the corresponding area in the pressure distribution map, and use it as the ground contact intensity of that leg. .

[0055] It should be noted that when generating the ground contact intensity associated with the corresponding leg, the symmetrical layout of the crab's eight legs can be further utilized for allocation optimization. Specifically, the crab's eight legs are symmetrically distributed left and right. With the origin of the body coordinate system as the boundary, the four legs on the left and the four legs on the right will exhibit a spatially separated pattern in the bottom pressure map. By dividing the ground contact points in the bottom pressure map into left and right columns along the midline of the body width, the ground contact points in the left column are assigned only to the four legs on the left, and the ground contact points in the right column are assigned only to the four legs on the right. This reduces the candidate range for each ground contact point from eight legs to four legs, reducing ambiguity in ground contact point assignment.

[0056] The above method yielded the ground contact intensity sequence. This provides time-series data of stress modes for subsequent inferences based on gait rhythms.

[0057] Step S5: Based on the visual confidence sequence and the ground contact intensity sequence, determine whether the target crab has missing legs by using a preset gait rule that represents the expected ground contact window of each leg in the crab's gait cycle.

[0058] This step is the core innovation of this invention. Traditional methods rely solely on visual or pressure information to determine leg status, which is easily affected by factors such as occlusion and noise. This invention, by introducing the inherent rhythmic characteristics of crab's sideways gait, constructs a spatiotemporally consistent inference mechanism, enabling more reliable judgments of leg loss supported by evidence from multiple time points.

[0059] In this embodiment, the gait rules representing the expected ground contact window of each leg in the crab's gait cycle are obtained in the following way: using a large number of normal crab top image sequences and corresponding foot pressure distribution data, the ground contact probability, ground contact phase and duration of each leg in the gait cycle are statistically analyzed to generate an average model of the sideways gait. This average model of the sideways gait is the gait rule.

[0060] It should be noted that when using the gait rules obtained in the above manner, at the initial moment of target crab detection, it is necessary to determine the initial ground contact state of each leg based on the observed visual confidence and pressure signals, so as to infer the expected ground contact window of each leg of the target crab within a preset time period based on the gait rules.

[0061] Gait rules can also be obtained in the following ways: First, using the time series of pressure distribution on the target crab's legs, the pressure peak at the moment each leg touches the ground is detected, and the overall gait cycle is estimated through periodic analysis. Then, combining the periodic appearance and disappearance of key points on the legs in the visual confidence sequence, the phase shift of the pressure signal is corrected, and the ground-touching center phase of each leg in the gait cycle is determined. Finally, based on the estimated gait cycle and ground-touching center phase, combined with the proportion of the crab's natural ground-touching phase in the cycle, the expected ground-touching time window is divided for each leg, forming the gait rule. It should be noted that since the target crab may have missing legs, the ground-touching signal of a single leg may be missing or abnormal when constructing gait rules in this way. To ensure the reliability of the gait rules, the following strategy is adopted: If the ground-touching signal of a certain leg is found to be continuously missing or severely abnormal, the expected ground-touching window of the suspected missing leg is supplemented by analyzing the effective ground-touching signals of other intact legs and combining the time synchronization of the left and right symmetrical legs, thereby obtaining a complete gait rule.

[0062] Specifically, in step S5, determining whether a leg is missing includes:

[0063] For any leg, when it is within the expected ground contact time window, if the corresponding visual confidence is lower than the first preset threshold and the ground contact intensity is lower than the second preset threshold, then evidence of the missing leg is generated.

[0064] This step enables dual-modal evidence fusion judgment at a single point in time. Specifically:

[0065] First, for the i-th leg, determine whether it is within the expected ground contact time window at time t according to the gait rules.

[0066] Then, examine the bimodal evidence at that moment:

[0067] Visual modality: if (First preset threshold) indicates that the leg is visually invisible or has extremely low confidence.

[0068] Pressure mode: If (Second preset threshold) indicates that the leg has no contact signal with the ground or the pressure is extremely weak.

[0069] When both conditions are met, it means that within the time window when the leg should have touched the ground, neither the visual shape of the leg can be seen, nor a pressure signal can be detected. According to the biomechanical principles of crab gait, this situation is almost impossible in normal crabs (even with occlusion, pressure would inevitably be generated upon touching the ground). Therefore, evidence of the missing leg is generated. To avoid misjudgment due to error factors, the target crab is only determined to have a missing leg if evidence of its missing leg is present in all expected ground-touching time windows within a preset time period.

[0070] Further steps in determining whether leg loss exists include:

[0071] For any leg, when it is within the expected ground contact time window, if the difference between the ground contact strength of the leg and the ground contact strength of the symmetrical leg is always greater than the third preset threshold, then missing inference evidence is generated for the leg with the smaller ground contact strength.

[0072] This step utilizes the symmetry of crab gait for cross-validation. Specifically, the crab's left and right symmetrical legs exhibit almost mirror-image ground-touching rhythms in their gait. When both symmetrical legs are normal, they should generate pressure signals of similar intensity within their respective ground-touching windows. When the ground-touching intensity of one leg is consistently significantly lower than that of the symmetrical leg, it is highly likely due to the absence of that leg, rather than temporary signal weakening caused by posture or environmental factors. Through symmetry testing, unilateral pressure weakening caused by crab tilting or uneven ground can be effectively ruled out, improving the reliability of the missing leg assessment.

[0073] In the above scheme, evidence of missing legs is required within each expected ground contact time window before the target crab is finally determined to have missing legs. Although this can reduce errors, there is still a risk of misclassifying missing legs as not missing. Therefore, this invention further proposes a second determination method, which is to count the number of evidence of missing legs for each leg of the target crab within a preset time period. When the number of evidence of missing legs for any one leg is greater than a preset threshold, the target crab is determined to have missing legs.

[0074] The second method described above has the following advantages over the previous method: by statistically analyzing missing evidence across multiple time windows, it can reduce the interference of single-point outlier data on the overall judgment, and at the same time, it can improve the stability of the judgment by dynamically adjusting the preset quantity threshold.

[0075] In the above scheme, if a fixed threshold is used for the preset quantity threshold, it cannot adapt to different detection durations. Therefore, this embodiment further proposes:

[0076] Count the number of expected ground contact time windows for each leg within a preset time period;

[0077] The preset quantity threshold is adjusted based on the ratio of the expected number of ground contact time windows to the preset standard number.

[0078] The purpose of the above steps is to make the judgment threshold adaptive and avoid misjudgment due to different detection times.

[0079] Specifically, let the number of expected ground contact time windows for the i-th leg within a preset time period be . The standard quantity is The adjusted preset quantity threshold is:

[0080] ;

[0081] in, The baseline threshold is typically set to 60%-80% of the standard quantity.

[0082] It should be noted that the method provided in the above embodiments of the present invention is not limited to leg loss detection. Based on the collected multimodal data, it can also be further applied to the judgment of other quality indicators. For example, by statistically analyzing the foot pressure distribution, the weight or shell fullness of the crab can be indirectly inferred. Generally speaking, under the same projected area, individuals with higher total pressure values ​​usually have fuller meat and higher unit weight.

[0083] As an extended application of this embodiment, after obtaining the top image sequence and ground contact intensity sequence of the target crab according to the above steps S1 to S4, the following saturation determination steps can also be performed:

[0084] First, identify all complete expected ground contact windows that appear within a preset time period T. For each complete expected ground contact window, calculate the sum of the ground contact strength of all legs within that window. Take the average of the calculation results for all complete expected ground contact windows, and record this average as the representative pressure value for the crab. ;

[0085] Then, select the image with the highest resolution from the top image sequence, and obtain the projected area A of the crab shell through image segmentation;

[0086] Finally, calculate the fullness index. This index eliminates the influence of individual size on the absolute pressure value and reflects the pressure load per unit area. The fullness index can be used to further determine the quality of crabs.

[0087] It should be understood that the above-mentioned method for determining fullness utilizes the pressure data collected synchronously in this embodiment for gait detection, achieving non-destructive and rapid detection of another important commodity quality of crabs without increasing any hardware costs.

[0088] To verify the effectiveness of the fully automated crab quality detection method according to this embodiment, the following experiment was conducted. The test samples collected were swimming crabs (Portunus trituberculatus), sourced from local farms, near-shore fishing areas, and deep-sea aquaculture bases. Collection time was from September to November 2024, with weights ranging from 125-580g and carapace widths from 7.2-14.8cm. Data acquisition employed a dual-modal synchronous method: top image data resolution was 2048×1536 pixels at 120fps, and bottom pressure data sampling frequency was 120Hz. Hardware synchronization ensured a time alignment error of less than 8.3ms between the two data streams. The observation duration for each crab was set to 1.5 seconds (corresponding to a sequence of 180 images) to cover 2-3 complete gait cycles, thus providing sufficient temporal information for gait rhythm analysis.

[0089] A total of 528 valid samples were collected in the experiment, and their specific distribution is shown in Table 1. To fully evaluate the robustness of the method in real-world applications, typical interference samples were introduced: 58 samples with mud and sand attached to their bodies, 42 samples with algae attached, and 67 samples with overlapping legs, to simulate the complex situations that may be encountered in real detection environments.

[0090] Table 1. Composition of experimental samples

[0091]

[0092] The key parameter settings used in the experiment are shown in Table 2. Each threshold parameter was optimized through cross-validation with 50 calibration samples, ensuring both detection accuracy and computational efficiency.

[0093] Table 2 Method Parameter Configuration

[0094]

[0095] To comprehensively evaluate the performance advantages of the method of this invention, three sets of comparative baseline methods were set up: (1) the traditional visual detection method (Baseline-V), which adopts a leg target detection and counting method based on Faster R-CNN; (2) the pure pressure analysis method (Baseline-P), which relies only on pressure sensor data and identifies and counts pressure peak points through clustering algorithms; and (3) the simple fusion method (Baseline-F), which performs a logical OR operation on the visual and pressure detection results, but does not consider gait temporal constraints. All comparative methods were evaluated on the same test set to ensure the fairness of the comparison. The results are shown in Tables 3 and 4.

[0096] Table 3 Comparison of Leg Integrity Testing Performance

[0097]

[0098] Table 4 Detection accuracy for different sample types

[0099]

[0100] A thorough analysis of 28 false positive cases revealed three main error patterns: 11 cases (39.3%) of extreme occlusion misjudgment, where visual and pressure signals were missing throughout all gait cycles when the crab remained still for an extended period and one leg was completely obscured; 9 cases (32.1%) of gait abnormality interference, where some crabs exhibited disrupted gait rhythms due to stress or injury, contradicting the pre-defined ground contact window assumption; and 8 cases (28.6%) of pressure threshold boundary effects, where sensor noise caused instability when foot pressure hovered near the threshold. These findings provide clear directions for future method improvements.

[0101] Table 5 Algorithm Computation Performance Analysis

[0102]

[0103] As shown in Table 5, the visual confidence extraction stage takes the longest time (0.52 seconds, accounting for 28.6%), because this stage requires frame-by-frame skeleton matching and confidence scoring within a preset activity area. The gait analysis and judgment stage is the second longest (0.45 seconds, accounting for 24.7%), mainly involving dynamic programming calculations for time-series data. Through batch processing optimization, the processing time per unit can be reduced to 1.15 seconds, indicating that the algorithm has good parallelization potential. The overall processing speed meets the real-time requirements of industrial applications.

[0104] To evaluate the long-term stability of the method in practical applications, a 30-day continuous operation test (8 hours per day) was conducted in a production environment, with a total of 14,762 crabs tested. Fifty samples were randomly selected each week for manual verification. The results are shown in Table 6.

[0105] Table 6 Long-term stability test results

[0106]

[0107] Long-term testing results show that the method according to this embodiment maintained stable detection performance over 30 consecutive days of operation, with an accuracy fluctuation range of only 2.2 percentage points (standard deviation 0.68%). Linear regression analysis showed no significant performance degradation trend (slope -0.012% / day, p=0.36). Fault location and parameter recalibration were completed within 12 minutes in all three abnormal situations, with a data effectiveness rate of 99.75%, fully verifying the practicality and reliability of the method.

[0108] Furthermore, to verify the potential for expanded application of the method in this embodiment in fullness assessment, fullness index was calculated for 180 samples using foot pressure data collected during the testing process, and compared with the five-level scoring (A-excellent fullness to E-empty shell) by five senior quality inspection experts. The results are shown in Table 7:

[0109] Table 7. Fullness Assessment Results

[0110]

[0111] Table 7 shows that the plumpness index calculated by the fully automated crab quality detection method in this embodiment is highly positively correlated with expert ratings (r=0.887, p<0.001), with an overall classification accuracy of 88.3%, indicating that it is feasible to use pressure data from the gait detection process for plumpness assessment. This extended application requires no additional hardware investment and can be achieved solely through data reuse at the algorithm level, providing an economical and effective technical path for the comprehensive evaluation of crab quality.

[0112] Based on the above experimental results, the fully automated crab quality detection method of this embodiment achieves a 13.4 percentage point improvement in accuracy compared to traditional pure vision methods in leg integrity detection. The processing time per crab is 1.82 seconds, meeting industrial real-time requirements, and it maintains stable performance during a 30-day long-term test. By fusing visual and pressure-based dual-modal information and introducing gait temporal constraints, the method effectively overcomes the limitations of single-modal methods in complex scenarios such as occlusion and weak signals. Furthermore, the pressure data collected during detection can also be used for fullness assessment (correlation coefficient 0.887), demonstrating the method's good scalability.

[0113] In summary, the fully automated crab quality detection method based on multimodal image processing proposed in this invention improves the reliability of crab leg integrity detection by fusing top visual information and bottom pressure information, and combining the inherent rhythmic characteristics of crab's sideways gait. It utilizes the dual absence of visual and pressure signals within the expected ground contact window as a judgment criterion. Compared with traditional visual methods, it improves the accuracy of detection.

[0114] Example 2

[0115] This embodiment provides a fully automated crab quality detection system based on multimodal image processing.

[0116] Figure 2 This is a block diagram of a fully automated crab quality detection system based on multimodal image processing, as described in an embodiment of the present invention.

[0117] like Figure 2 As shown, the fully automated crab quality detection system based on multimodal image processing in this embodiment includes:

[0118] The acquisition module is used to acquire a continuous sequence of top images of the target crab within a preset time period, as well as the foot pressure distribution corresponding to each frame of the top image.

[0119] The coordinate system construction module is used to establish a body coordinate system for locating the spatial position of the legs for each frame of the top image.

[0120] In this embodiment, the body coordinate system is a two-dimensional coordinate system. The establishment of the body coordinate system specifically includes the following steps: identifying the shell width direction and geometric center of the target crab in the top image; establishing the body coordinate system based on the shell width direction, with the origin of the body coordinate system as the geometric center, and the horizontal axis of the body coordinate system parallel to the shell width direction.

[0121] The visual processing module is used to extract visual confidence scores from the preset active areas of each leg of the target crab in each frame of the top image, with the origin of the corresponding body coordinate system as a reference, to obtain a visual confidence score sequence reflecting the visibility of each leg of the target crab at each time point.

[0122] In this embodiment, extracting visual confidence specifically includes: determining the preset activity area of ​​each leg in the body coordinate system based on a preset crab eight-leg topological distribution model; detecting the number of joints of the leg segments within the preset activity area based on a preset posture estimation model, and extracting the coordinate position and position confidence of each joint; calculating the skeletal matching degree between the leg skeleton shape formed by connecting each joint and the preset standard leg skeleton template based on the coordinate position of each joint; and weighting and fusing the skeletal matching degree and the position confidence of each joint according to a preset weighting coefficient to obtain the visual confidence.

[0123] The ground contact strength binding module is used to generate the ground contact strength bound to the corresponding leg of the target crab in each frame of the top image based on the body coordinate system, foot pressure distribution and preset spatial calibration matrix, so as to obtain the ground contact strength sequence.

[0124] In this embodiment, generating the ground contact intensity bound to the corresponding leg of the target crab in each frame of the top image includes: mapping the ground contact intensity distribution corresponding to each frame of the top image to the preset activity area of ​​each leg of the target crab according to the body coordinate system and the spatial calibration matrix, so as to obtain the ground contact intensity bound to the corresponding leg of the target crab.

[0125] The decision module, based on visual confidence sequences and ground contact intensity sequences, uses preset gait rules representing the expected ground contact windows of each leg in the crab's gait cycle to determine whether a target crab has missing legs. The determination of missing legs includes:

[0126] For any leg, when it is within the expected ground contact time window, if the corresponding visual confidence is lower than the first preset threshold and the ground contact intensity is lower than the second preset threshold, then evidence of the missing leg is generated.

[0127] If, within a preset time period, there is evidence of a missing leg in a target crab across all expected ground contact time windows, then the target crab is determined to have a missing leg.

[0128] In this embodiment, determining whether a leg is missing also includes: for any leg, when it is within the expected ground contact time window, if the difference between the ground contact strength of the leg and the ground contact strength of the symmetrical leg is always greater than a third preset threshold, then evidence of missing leg is generated for the leg with the lower ground contact strength.

[0129] In this embodiment, determining that the target crab has missing legs includes: counting the number of inferences about the missing legs of each leg of the target crab within a preset time period; when the number of inferences about the missing legs of any one leg is greater than a preset threshold, it is determined that the target crab has missing legs.

[0130] The preset quantity threshold is dynamically adjusted, specifically including: counting the number of expected ground contact time windows for each leg within a preset time period; and adjusting the preset quantity threshold based on the ratio of the number of expected ground contact time windows to the preset standard number.

[0131] Example 3

[0132] This embodiment provides an electronic device.

[0133] Figure 3 This is a block diagram of an electronic device according to an embodiment of the present invention.

[0134] like Figure 3 As shown, the electronic device includes one or more processors and memory.

[0135] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0136] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0137] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0138] Of course, for the sake of simplicity, Figure 3 Only some of the components of the electronic device relevant to the present invention are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0139] Example 4

[0140] This embodiment provides a computer-readable medium having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps described in Embodiment 1 above according to these various embodiments.

[0141] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0142] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0143] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0144] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0145] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0146] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A full-automatic crab quality detection method based on multi-modal image processing, characterized in that, The method comprises the following steps: Step 1, obtaining a sequence of top images of the target crab in a preset time period and a foot pressure distribution corresponding to each top image; Step 2, establishing a body coordinate system for positioning the spatial position of the leg for each top image; Step 3, extracting a visual confidence in a preset activity area of each leg of the target crab with the target crab as a reference in each top image, to obtain a visual confidence sequence reflecting the visibility of each leg of the target crab at each time point; Step 4, generating a ground contact intensity bound to each leg of the target crab in each top image based on the body coordinate system, the foot pressure distribution and a preset spatial calibration matrix, to obtain a ground contact intensity sequence; Step 5, judging whether the target crab has a leg missing based on the visual confidence sequence and the ground contact intensity sequence through a preset gait rule representing an expected ground contact time window of each leg in a gait cycle, wherein the judgment of whether there is a leg missing comprises: For any leg, if the corresponding visual confidence is lower than a first preset threshold and the ground contact intensity is lower than a second preset threshold when the leg is in the expected ground contact time window, a missing inference evidence of the leg is generated; When there is a leg of the target crab having missing inference evidence in each expected ground contact time window in the preset time period, it is determined that the target crab has a leg missing.

2. The crab quality full-automatic detection method based on multi-modal image processing according to claim 1, wherein: In step 2, the body coordinate system is a two-dimensional coordinate system, and the establishment of the body coordinate system comprises: Identifying the shell width direction and the geometric center of the target crab in the top image; Establishing the body coordinate system according to the shell width direction, wherein the origin of the body coordinate system is the geometric center, and the horizontal axis of the body coordinate system is parallel to the shell width direction.

3. The crab quality full-automatic detection method based on multi-modal image processing according to claim 1, wherein: In step 3, the extraction of the visual confidence specifically comprises: Determining the preset activity area of each leg in the body coordinate system according to a preset crab octopus topology distribution model; Detecting the number of joint points of leg segments in the preset activity area based on a preset posture estimation model, and extracting the coordinate position and position confidence of each joint point; Calculating the skeleton matching degree between the leg skeleton shape formed by the joint points and a preset standard leg skeleton template according to the coordinate position of each joint point; Weighting and fusing the skeleton matching degree and the position confidence of each joint point according to a preset weight coefficient to obtain the visual confidence.

4. The crab quality full-automatic detection method based on multi-modal image processing according to claim 1, wherein: In step 4, the generation of the ground contact intensity bound to each leg of the target crab in each top image comprises: According to the body coordinate system and the space calibration matrix, the foot pressure distribution corresponding to each frame of top image is mapped into a preset activity region of each leg of the target crab to obtain a ground contact intensity corresponding to each leg of the target crab.

5. The crab quality automatic detection method based on multi-modal image processing according to claim 1, characterized in that: In step 5, the judging whether there is leg loss further comprises: For any leg, when the leg is in the expected ground contact time window, if the difference between the ground contact intensity of the leg and the ground contact intensity of the symmetrical leg is always greater than a third preset threshold, a leg loss inference evidence is generated for the leg with smaller ground contact intensity.

6. The crab quality automatic detection method based on multi-modal image processing according to claim 1, characterized in that: The judging whether there is leg loss of the target crab comprises: The number of leg loss inference evidences of each leg of the target crab in the preset time period is counted; When the number of leg loss inference evidences of any leg is greater than a preset number threshold, it is determined that the target crab has leg loss. 7.The full-automatic crab quality detection method based on multi-modal image processing according to claim 6, characterized in that, Further comprising: The number of expected ground contact time windows of each leg in the preset time period is counted; The preset number threshold is adjusted according to the ratio of the number of expected ground contact time windows to a preset standard number.

8. A crab quality full-automatic detection system based on multi-modal image processing, characterized in that, Comprising: An acquisition module is configured to acquire a sequence of continuous top images of a target crab in a preset time period and a foot pressure distribution corresponding to each frame of top image; A coordinate system construction module is configured to establish, for each frame of top image, a body coordinate system for positioning the spatial position of the leg; A visual processing module is configured to extract, for each frame of top image, a visual confidence in a preset activity region of each leg of the target crab with the origin of the corresponding body coordinate system as a reference to obtain a visual confidence sequence reflecting the visibility of each leg of the target crab at each time point; A ground contact intensity binding module is configured to generate, based on the body coordinate system, the foot pressure distribution and a preset space calibration matrix, a ground contact intensity bound to each leg of the target crab in each frame of top image to obtain a ground contact intensity sequence; A decision module is configured to judge, based on the visual confidence sequence and the ground contact intensity sequence, whether the target crab has leg loss through a preset gait rule representing an expected ground contact time window of each leg in a gait cycle of the crab, and the judging whether there is leg loss comprises: For any leg, when the leg is in the expected ground contact time window, if the corresponding visual confidence is lower than a first preset threshold and the ground contact intensity is lower than a second preset threshold, a leg loss inference evidence is generated for the leg; When there is a leg loss inference evidence of any leg of the target crab in each expected ground contact time window in the preset time period, it is determined that the target crab has leg loss. 9.An electronic device comprising a memory and a processor, the electronic device characterized by: The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the method according to any one of claims 1-7.

10. A computer storage medium having stored thereon computer- executable instructions, comprising: The computer executable instructions, when executed by the processor, implement the steps of the method according to any one of claims 1-7.

Images