Training method and device based on multi-modal pig weight estimation

Abstract

The invention provides a training method and device based on multi-modal pig weight estimation, relates to the field of image processing, and solves the technical problems of poor practicability and insufficient weight estimation accuracy in the prior art. The method comprises the following steps: acquiring an RGB image and a depth map of a pig to be subjected to weight estimation; carrying out graying, mask segmentation and scaling standardization processing on the RGB image to obtain first channel data; based on the depth map, pig height features are extracted and standardized, and second channel data are obtained; extracting pig back thickness features based on the depth map and performing standardization processing to obtain third channel data; inputting information of the first channel data, the second channel data and the third channel data into a weight estimation model to obtain an estimated weight; wherein the weight estimation model is obtained based on training after a PyTorch framework is constructed, and the PyTorch framework comprises a backbone network, a regression head module and a loss function module. The method is used in the pig weight estimation process.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a training method and apparatus for multimodal pig weight estimation. Background Technology

[0002] In the field of pig weight estimation, a multimodal instance segmentation model is used to extract pixel information of pig body size. This pixel information, along with a prior calibration template, is then input into a support vector regression model for training. Based on the trained model, the pig's weight can be estimated. However, this approach has limitations. It requires designing specific pigpens, installing cameras at specific locations within the pen, and herding individual pigs into the pen to collect top-down and side-view images. This method places stringent requirements on the angle of data acquisition, resulting in poor practicality and excessive conditions, which can easily lead to insufficient weight estimation accuracy. Summary of the Invention

[0003] This application provides a training method and apparatus for multimodal pig weight estimation, which solves the technical problems of poor practicality caused by the cumbersome process of acquiring images of pigs to be estimated and insufficient accuracy in estimating the weight of pigs in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] Firstly, a training method for multimodal pig weight estimation is provided, comprising the following steps:

[0006] S1. Acquire data images of the pigs to be weighed; wherein, the data images include RGB images and depth maps, and the data images are acquired by multiple 3D cameras installed above the pigs to be weighed, and can simultaneously output pixel-aligned RGB images and corresponding depth maps;

[0007] S2. Using a segmentation module based on YOLOv12, the RGB image of the pig to be weighed is segmented to obtain the corresponding segmentation region and detection box.

[0008] S3. Map the segmented regions and detection boxes of the RGB image onto the depth map by pixel alignment. After counting the non-zero values ​​of the pixels in the non-segmented regions within the detection boxes in the depth map, and considering the unevenness of the actual ground, calculate the average value to obtain the vertical distance h_c between the ground where the pig to be estimated stands and the camera.

[0009] S4. Set the RGB images and depth maps from different 3D cameras to the same scale h_s;

[0010] S5. Perform grayscale conversion, mask segmentation, scaling and standardization on the RGB image to obtain the first channel data;

[0011] S6. Extract the height features of pigs based on the depth map and perform standardization processing to obtain the second channel data;

[0012] S7. Extract the thickness features of the pig's back based on the depth map and perform standardization processing to obtain the third channel data;

[0013] S8. Input the data from the first channel, the second channel, and the third channel into the weight estimation model to obtain the estimated weight; wherein, the weight estimation model is trained based on the construction of the PyTorch framework, which includes a backbone network, a regression head module, and a loss function module.

[0014] It should be noted that the RGB images and depth maps from different 3D cameras are data images acquired when the pig to be weighed moves directly under the corresponding 3D camera.

[0015] Based on the above technical solution, the training method for multimodal pig weight estimation provided in this application only requires data images of the pig's back, and fully combines RGB image and depth map information to estimate the weight of the pig, which simplifies the weight estimation process, enhances practicality and improves the accuracy of weight estimation.

[0016] In conjunction with the first aspect above, in one possible implementation, the vertical distance h_c between the ground where the pig to be weighed is standing and the camera is calculated as follows:

[0017]

[0018] Where M is the total number of pixels that meet the conditions; D(x,y) is the depth value at coordinates (x,y) in the depth map; B is the region of the detection box, B={(x,y)|x min ≤x≤x max ,y min ≤y≤y max}; S is the segmentation mask, S =

[0019] {(x,y)|(x,y)∈s the segmented region}.

[0020] In conjunction with the first aspect mentioned above, in one possible implementation, the RGB image undergoes grayscale conversion, mask segmentation, and scaling / normalization processing. Specific steps include:

[0021] S51. Convert the RGB image to a grayscale image;

[0022] S52. Set all pixel values ​​inside the segmented region of the grayscale image to the preset value p_s, and set all pixel values ​​in the non-segmented region to zero.

[0023] S53. Scale the grayscale image to scale h_s according to the scaling ratio; wherein, the scaling ratio rate is calculated using the following formula:

[0024] S54. Adjust the resolution of the scaled grayscale image to the preset resolution to obtain the first channel data.

[0025] In conjunction with the first aspect above, in one possible implementation, adjusting the resolution of the scaled grayscale image to a preset resolution includes:

[0026] Compare the scaling ratio with the scaling threshold;

[0027] When the scaling ratio is less than 1, the edge regions of the grayscale image are filled with zero pixels to the preset resolution to obtain the first channel data.

[0028] When the scaling ratio is greater than 1, the grayscale image is cropped to a preset resolution according to the principle that no pigs to be estimated are lost, and the first channel data is obtained.

[0029] When the scaling factor is equal to 1, the grayscale image is labeled as the first channel data.

[0030] In conjunction with the first aspect mentioned above, one possible implementation involves extracting pig height features based on depth maps and performing standardization processing, specifically including the following steps:

[0031] S61. Perform a masking operation on the depth map based on the segmented region, retain the original depth values ​​within the segmented region, and set all depth values ​​outside the segmented region to zero to obtain the depth object mask map d_m;

[0032] S62. Generate a blank depth map with the same size as and aligned with the depth object mask map. Set the pixel values ​​within the segmented area of ​​the blank depth map to h_c, and keep the pixel values ​​outside the segmented area to zero to obtain the height reference map d_m_h.

[0033] S63. By performing a difference operation between the height reference map and the depth object mask map, a first image d_h containing information on the distance of each position on the pig's back from the ground is generated;

[0034] S64. Scale the first image d_h according to the scaling ratio rate to obtain the second image d_h_16;

[0035] S65. Set the maximum height of the pig p_m, map the second image d_h_16 from [0, p_m] to [0, 255] according to the pixel value to obtain an 8-bit third image, and mark the third image as the second channel data.

[0036] In conjunction with the first aspect mentioned above, one possible implementation involves extracting pig back thickness features based on depth maps and performing standardization processing, specifically including the following steps:

[0037] S71. Calculate the maximum value p_max and the non-zero minimum value p_min of the pixel values ​​in the first image d_h;

[0038] S72. Subtract the non-zero value from the minimum non-zero value p_min in the first image d_h pixel by pixel to obtain the relative thickness information of the pig's back d_h_ref;

[0039] S73. Perform 8-bit processing on the relative thickness information d_h_ref of the pig's back, traverse the pixels, set the pixel values ​​greater than 255 to 255, and obtain the third channel data.

[0040] In conjunction with the first aspect above, in one possible implementation, the backbone network is MobileNetV4.

[0041] In conjunction with the first aspect mentioned above, in one possible implementation, the regression head module includes: an input layer, N CBS modules, and an output layer; wherein, the input layer takes into account the features output by the backbone network; and the output layer outputs features of size batch_size*1.

[0042] In conjunction with the first aspect above, in one possible implementation, the loss function is a weighted mean squared error loss function (WeightedMSELoss); wherein the formula for calculating WeightedMSELoss is:

[0043]

[0044] In the formula, L w This represents the loss calculated for one sample size; N represents the set value for the sample size; P i t represents the predicted weight of pigs by the model. i This represents the actual weight of the pig.

[0045] Secondly, an electronic device is provided, comprising: a communication unit and a processing unit; the communication unit is used to acquire data images of a pig to be weighed; the processing unit is used to segment the RGB image of the pig to be weighed to obtain corresponding segmented regions and detection boxes; map the segmented regions and detection boxes of the RGB image onto a depth map by pixel alignment, calculate the average value of the non-zero values ​​of pixels in the non-segmented regions within the detection boxes in the depth map, and obtain the vertical distance h_c of the ground where the pig to be weighed is standing from the camera; set the RGB images and depth maps of different 3D cameras to the same scale h_s; perform grayscale conversion, mask segmentation, scaling and standardization processing on the RGB image to obtain the first channel data; extract the height feature of the pig based on the depth map and perform standardization processing to obtain the second channel data; extract the back thickness feature of the pig based on the depth map and perform standardization processing to obtain the third channel data; input the first channel data, second channel data and third channel data information into the weight estimation model to obtain the estimated weight.

[0046] Thirdly, this application provides an electronic device, including: a processor and a storage medium; the storage medium includes instructions, and the processor is configured to execute the instructions to implement the methods described in the first aspect and any possible implementation thereof. This electronic device may be an electronic device or a chip within an electronic device.

[0047] Fourthly, this application provides a training method system for multimodal pig weight estimation, including: a 3D camera and an electronic device; wherein, the 3D camera is used to acquire data images of pigs to be estimated, and the electronic device is used to acquire data images of pigs to be estimated; the RGB image of the pig to be estimated is segmented to obtain corresponding segmented regions and detection boxes; the segmented regions and detection boxes of the RGB image are mapped onto a depth map by pixel alignment, and the non-zero values ​​of pixels in the non-segmented regions within the detection boxes in the depth map are counted, and their average value is calculated to obtain the vertical distance h_c of the ground where the pig to be estimated is standing from the camera; the RGB images and depth maps of different 3D cameras are set to the same scale h_s; the RGB images are grayscaled, masked, scaled, and standardized to obtain the first channel data; the height features of the pig are extracted based on the depth map and standardized to obtain the second channel data; the back thickness features of the pig are extracted based on the depth map and standardized to obtain the third channel data; the first channel data, the second channel data, and the third channel data are input into the weight estimation model to obtain the estimated weight.

[0048] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0049] In a sixth aspect, this application provides a computer program product containing instructions that, when run on an electronic device, cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0050] This application provides a training method and electronic device for multimodal pig weight estimation. It only requires data images of the pig's back, and by combining RGB images and depth maps, it can efficiently estimate the weight of pigs. It is applicable to any height within a certain height range, simplifying the weight estimation process, enhancing practicality, and improving estimation accuracy. Furthermore, a method for calculating pig height is proposed. The height of the pig can be calculated from a top-down view, in pixels, allowing for the calculation of the height at each position on the pig's back.

[0051] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0052] Figure 1 A system architecture diagram of a multimodal pig weight estimation training system provided in this application embodiment;

[0053] Figure 2 A flowchart illustrating a multimodal pig weight estimation training method provided in this application embodiment;

[0054] Figure 3 A schematic diagram illustrating multimodal pig weight estimation provided in an embodiment of this application;

[0055] Figure 4 A flowchart illustrating another multimodal pig weight estimation training method provided in this application embodiment;

[0056] Figure 5 A flowchart illustrating another multimodal pig weight estimation training method provided in this application embodiment;

[0057] Figure 6A flowchart illustrating another multimodal pig weight estimation training method provided in this application embodiment;

[0058] Figure 7 A flowchart illustrating another multimodal pig weight estimation training method provided in this application embodiment;

[0059] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0060] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0061] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0062] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0063] The training method for multimodal pig weight estimation provided in this application can be applied to, for example... Figure 1 In the multimodal pig weight estimation training system 100 shown, such as Figure 1 As shown, the communication system includes: an image acquisition device 10, a cloud computing device 20, and a network device 30.

[0064] Among them, the image acquisition device 10 is used to acquire data images of pigs to be estimated in weight.

[0065] The cloud computing device 20 is used to segment the RGB image of the pig to be weighed, obtaining corresponding segmented regions and detection boxes; the segmented regions and detection boxes of the RGB image are mapped onto the depth map by pixel alignment; after counting the non-zero values ​​of pixels in the non-segmented regions within the detection boxes in the depth map, their average value is calculated to obtain the vertical distance h_c of the ground where the pig to be weighed is standing from the camera; the RGB images and depth maps of different 3D cameras are set to the same scale h_s; the RGB images are grayscaled, masked, and scaled and standardized to obtain the first channel data; the height features of the pig are extracted based on the depth map and standardized to obtain the second channel data; the back thickness features of the pig are extracted based on the depth map and standardized to obtain the third channel data; the first channel data, the second channel data, and the third channel data are input into the weight estimation model to obtain the estimated weight.

[0066] Network device 30 is used to transmit the data acquired by image acquisition device 10 to cloud computing device 20.

[0067] To address the technical problems of poor practicality and insufficient weight estimation accuracy in existing technologies, this application provides a training method for multimodal pig weight estimation, which includes:

[0068] S201. Acquire data images of the pigs to be weighed; wherein, the data images include RGB images and depth maps, and the data images are acquired by multiple 3D cameras installed above the pigs to be weighed, and can simultaneously output pixel-aligned RGB images and corresponding depth maps.

[0069] S202. Using a segmentation module based on YOLOv12, the RGB image of the pig to be weighed is segmented to obtain the corresponding segmentation region and detection box.

[0070] S203. Map the segmented regions and detection boxes of the RGB image onto the depth map by pixel alignment. After counting the non-zero values ​​of the pixels in the non-segmented regions within the detection boxes in the depth map, and considering the unevenness of the actual ground, calculate the average value to obtain the vertical distance h_c between the ground where the pig to be estimated stands and the camera.

[0071] S204. Considering that the installation height of the camera is inconsistent in different scenarios, in order to improve the accuracy of model weight estimation, the RGB images and depth maps of different 3D cameras are set to the same scale h_s.

[0072] S205. Perform grayscale conversion, mask segmentation, scaling and standardization on the RGB image to obtain the first channel data;

[0073] S206. Extract the height features of pigs based on the depth map and perform standardization processing to obtain the second channel data;

[0074] S207. Extract the pig back thickness feature based on the depth map and perform standardization processing to obtain the third channel data;

[0075] S208. Input the data from the first channel, the second channel, and the third channel into the weight estimation model to obtain the estimated weight; wherein, the weight estimation model is trained based on the construction of a PyTorch framework, and the PyTorch framework includes a backbone network, a regression head module, and a loss function module.

[0076] It should be noted that the RGB images and depth maps from different 3D cameras are data images acquired when the pig to be weighed moves directly under the corresponding 3D camera.

[0077] Based on this, the technical problems of poor practicality and insufficient weight estimation accuracy of existing technologies have been solved.

[0078] like Figure 2 As shown in the embodiments of this application, the training method for multimodal pig weight estimation includes:

[0079] S201. Obtain data images of the pigs to be weighed; wherein, the data images include RGB images and depth maps.

[0080] In some implementations, the data images are acquired using multiple 3D cameras mounted above the pigs to be weighed.

[0081] It should be noted that the 3D camera used can simultaneously output pixel-aligned RGB images and corresponding depth maps.

[0082] For example, multiple Intel RealSense D4553D cameras were used as image acquisition devices. The cameras were vertically mounted at a height of 1.5 meters directly above the activity area of ​​the pigs to be tested. The camera tilt angle was adjusted to keep it within ±5° to ensure that it was facing the back of the pigs. RGB images and depth maps with resolutions of 1280*720*3 and 1280*720*1 were acquired.

[0083] S202. Perform image segmentation on the RGB image of the pig to be weighed.

[0084] In some implementations, a segmentation module based on YOLOv12 is used to obtain the segmentation region and detection box of the pig to be weighed.

[0085] For example, the above image segmentation can be implemented using Python, and the specific code is as follows:

[0086] #Initialize the YOLOv12 segmentation model:

[0087] model=YOLOv12Seg(weights="yolov12x_seg.pt",device="cuda")

[0088] # Input image preprocessing;

[0089] image = cv2.imread("pig.jpg")

[0090] input_tensor=model.preprocess(image)

[0091] #Model Inference (Segmentation + Detection):

[0092] results = model(input_tensor)

[0093] mask = results.masks[0][0] # Get the segmentation mask

[0094] box = results.boxes.xyxy[0] # Get the detection boxes

[0095] #Extract the target segmentation region:

[0096] segmented=cv2.bitwise_and(image,image,mask=(mask*255).astype(np.uint8))

[0097] #Visualization results:

[0098] cv2.rectangle(image,(int(box[0]),int(box[1])),(int(box[2]),int(box[3])),(0,255,0),2)

[0099] cv2.imwrite("result.jpg",image)

[0100] The final output is a green pig detection box and a segmentation mask marking the segmented pig region.

[0101] S203. Map the segmented regions and detection boxes of the RGB image onto the depth map by pixel alignment. Count the non-zero values ​​of pixels in the non-segmented regions of the depth map detection box. Considering the unevenness of the actual ground, calculate the average value to obtain the vertical distance h_c between the ground where the pig to be estimated is standing and the camera.

[0102] In some implementations, the vertical distance h_c between the ground where the pig to be weighed is standing and the camera is calculated as follows:

[0103]

[0104] Where M is the total number of pixels that meet the conditions; D(x,y) is the depth value at coordinates (x,y) in the depth map; B is the region of the detection box, B={(x,y)|x min ≤x≤x max ,y min ≤y≤y max}; S is the segmentation mask, S = {(x,y)|(x,y)∈s the segmentation region}.

[0105] It should be noted that since the RGB image and the depth map have been aligned at the pixel level (i.e., the position of each pixel in the two images corresponds exactly), there is no need for complex coordinate transformations. The segmentation results obtained from the RGB image can be used directly to process the depth map.

[0106] For example, the calculation of h_c above can be implemented using Python, and the specific code is as follows:

[0107] #enter

[0108] rgb_img = cv2.imread("pig.jpg") # RGB image

[0109] depth = np.load("depth.npy") # Aligned depth map

[0110] mask = results.masks[0][0] # Separation mask

[0111] box = results.boxes.xyxy[0] # Detection box [x1, y1, x2, y2]

[0112] #Calculate ground height

[0113] roi_depth=depth[int(box[1]):int(box[3]),int(box[0]):int(box[2])]

[0114] ground_pixels=roi_depth[(mask[int(box[1]):int(box[3]),

[0115] int(box[0]):int(box[2])]==0)&(roi_depth>0)]

[0116] h_c = np.mean(ground_pixels) # Vertical distance from the ground to the camera

[0117] #Visualization

[0118] vis=cv2.normalize(depth,None,0,255,cv2.NORM_MINMAX)

[0119] cv2.rectangle(vis,(int(box[0]),int(box[1])),(int(box[2]),int(box[3])),255,2)

[0120] cv2.imwrite("result.jpg",vis)

[0121] The final output is the vertical distance h_c between the ground where the pig to be weighed is standing and the camera.

[0122] S204. Considering that the installation height of the camera is inconsistent in different scenarios, in order to improve the accuracy of model weight estimation, the RGB images and depth maps of different 3D cameras are set to the same scale h_s.

[0123] It should be noted that scaling is done to maintain the geometric consistency of the original data and to eliminate systematic errors introduced by installation height.

[0124] For example, the following code demonstrates how to use Python to unify the scale of images acquired by different 3D cameras:

[0125] #1. Calculate the scaling factor

[0126] h_s = Base height / Current camera height #e.g., 1.5m / 1.8m = 0.83

[0127] #2. Standardize data scales

[0128] defnormalize(rgb, depth, h_s):

[0129] #Adjust depth value

[0130] norm_depth = depth * h_s

[0131] #Adjust image size

[0132] norm_rgb=cv2.resize(rgb,None,fx=h_s,fy=h_s)

[0133] norm_depth=cv2.resize(norm_depth,None,fx=h_s,fy=h_s,

[0134] interpolation=cv2.INTER_NEAREST)

[0135] return norm_rgb,norm_depth

[0136] #3. Usage Example

[0137] norm_rgb,norm_depth=normalize(raw_rgb,raw_depth,h_s)

[0138] Finally, the RGB images and depth maps from different 3D cameras are set to the same scale h_s.

[0139] S205. Perform grayscale conversion, mask segmentation, scaling and standardization on the RGB image to obtain the first channel data.

[0140] In some implementations, the RGB image is subjected to grayscale conversion, masking, scaling, and normalization processing, such as... Figure 4 As shown, it includes:

[0141] S501. Convert the RGB image to a grayscale image. The original RGB image is as follows: Figure 3 As shown;

[0142] S502. In order to enhance the visual saliency of the segmented region of the pig and suppress background interference, all pixel values ​​inside the segmented region of the grayscale image are set to the preset value p_s, and all pixel values ​​in the non-segmented region are set to zero.

[0143] S503. Scale the grayscale image to the h_s scale according to the scaling ratio; wherein, the scaling ratio rate is calculated using the following formula:

[0144] S504. Adjust the resolution of the scaled grayscale image to the preset resolution to obtain the first channel data.

[0145] For example, the acquisition of the first channel data described above can be implemented using Python, and the specific code is as follows:

[0146] import cv2

[0147] import numpy as np

[0148] #Read the original 1280×720 image and mask

[0149] rgb_img = cv2.imread('pig.jpg') # Size: 1280×720

[0150] mask = cv2.imread('mask.png', 0) # Size: 1280×720

[0151] #(1) Grayscale conversion + masking (completed in one step)

[0152] processed = cv2.cvtColor(rgb_img, cv2.COLOR_BGR2GRAY) # Grayscale conversion

[0153] processed = np.where(mask>0,255,0).astype(np.uint8) # Binarization processing

[0154] #(2) Smart scaling (maintain aspect ratio)

[0155] target_h = 360

[0156] scale_factor = target_h / 720 # 720 is the original image height

[0157] resized=cv2.resize(processed,

[0158] (int(1280*scale_factor),target_h), # Calculate the new width

[0159] interpolation=cv2.INTER_AREA)

[0160] #(3) Center fill (automatically handles width and height)

[0161] result=np.zeros((target_h,640),dtype=np.uint8)

[0162] start_x = (640 - resized.shape[1]) / / 2 # Calculate the center position

[0163] result[:,start_x:start_x+resized.shape[1]]=resized

[0164] print(f"Input size: {rgb_img.shape[::-1][1:]}")

[0165] print(f"Output size: {result.shape[::-1]}")

[0166] cv2.imwrite('result.jpg',result)

[0167] The final output is a single-channel data image with a size of 640×360.

[0168] S206. Extract the height features of pigs based on the depth map and perform standardization processing to obtain the second channel data.

[0169] In some implementations, the height features of pigs are extracted based on depth maps and then standardized, such as... Figure 5 As shown, it includes the following steps:

[0170] S601. Perform a masking operation on the depth map based on the segmented regions, retaining the original depth values ​​within the segmented regions and setting all depth values ​​outside the segmented regions to zero, to obtain the depth object mask map d_m; specifically, the expression for the depth object mask map d_m is:

[0171] S602. Generate a blank depth map with the same size as and aligned with the depth object mask map. Set the pixel values ​​within the segmented areas of the blank depth map to h_c, and keep the pixel values ​​outside the segmented areas to zero, thus obtaining the height reference map d_m_h. Specifically, the expression for the height reference map is:

[0172] S603. By performing a difference operation between the height reference map and the depth object mask map, a first image d_h containing information on the distance of each position on the pig's back from the ground is generated; and the expression of the first image d_h is: d_h(x,y)=d_m_h(x,y)-d_m(x,y);

[0173] S604. Scale the first image d_h according to the scaling ratio rate to obtain the second image d_h_16.

[0174] S605. Set the maximum height of the pig p_m, map the second image d_h_16 from [0, p_m] to [0, 255] according to the pixel values ​​to obtain an 8-bit third image, and mark the third image as the second channel data.

[0175] For example, the input is a depth map of the pig with a resolution of 1280×720, and after the following processing steps, the specific Python code is as follows:

[0176] import cv2

[0177] import numpy as np

[0178] # Input data

[0179] depth_map = ... # 16-bit depth map (unit: mm)

[0180] mask = ... #Segmentation mask (pig region = 1, background = 0)

[0181] h_c = 1500 # Reference height (ground depth value)

[0182] p_m = 800 # Maximum height of pigs (obtained through statistics)

[0183] #S61: Depth Object Mask

[0184] d_m=np.where(mask==1,depth_map,0)

[0185] #S62: Height Reference Map

[0186] d_m_h=np.where(mask==1,h_c,0)

[0187] #S63: Height Difference Map

[0188] d_h=np.clip(d_m_h-d_m,0,None)

[0189] #S64: Scale to 640×360

[0190] d_h_16=cv2.resize(d_h,(640,360),interpolation=cv2.INTER_NEAREST)

[0191] #S65: 16-bit to 8-bit conversion

[0192] d_h_8=np.clip(d_h_resized / p_m*255,0,255).astype(np.uint8)

[0193] input_channel2 = d_h_8

[0194] The final output is a single-channel 8-bit image with a size of 640×360, where the pixel value represents the normalized height (0~255) of the pig's back above the ground.

[0195] S207. Extract the back thickness features of pigs based on the depth map and perform standardization processing to obtain the third channel data.

[0196] In some implementations, the thickness features of the pig's back are extracted based on the depth map and then standardized, such as... Figure 6 As shown, it includes the following steps:

[0197] S701. Extract the maximum value p_max and the non-zero minimum value p_min of the pixel values ​​in the first image d_h;

[0198] S702. Subtract the non-zero value from the minimum non-zero value p_min in the first image d_h pixel by pixel to obtain the relative thickness information of the pig's back d_h_ref;

[0199] S703. Perform 8-bit processing on the pig back relative thickness information d_h_ref, traverse each pixel one by one, and set values ​​greater than 255 to 255, thereby obtaining the third channel data in the range of [0,255].

[0200] For example, the acquisition of the third channel data mentioned above can be implemented using Python, and the specific code is as follows:

[0201] import numpy as np

[0202] # Input data (from the second channel processing result)

[0203] d_h = ... # 16-bit height difference image (unit: mm, background is 0)

[0204] #S71: Calculating Extreme Values

[0205] non_zero_pixels = d_h[d_h>0]

[0206] p_max=np.max(non_zero_pixels)

[0207] p_min=np.min(non_zero_pixels)

[0208] #S72: Relative Thickness Chart

[0209] d_h_ref=np.where(d_h>0,d_h-p_min,0)

[0210] #S73: 16-bit to 8-bit conversion

[0211] scale_factor=255 / (p_max-p_min)

[0212] d_h_8=np.clip(d_h_ref*scale_factor,0,255).astype(np.uint8)

[0213] input_channel3=d_h_8

[0214] The final output is a single-channel 8-bit image with a size of 640×360, and the pixel value represents the normalized value (0~255) of the relative thickness of the pig's back.

[0215] S208. Input the data from the first channel, the second channel, and the third channel into the weight estimation model to obtain the estimated weight.

[0216] The weight estimation model is trained based on a PyTorch framework, which includes a backbone network, a regression head module, and a loss function module.

[0217] For example, the input consists of three channels of pre-processed data, which then undergo the following processing steps. The specific Python code implementation is as follows:

[0218] import torch

[0219] #Pre-processed three-channel data (example values)

[0220] #The shapes are all (1, 3, 640, 360) -> [batch_size, channels, height, width]

[0221] input_tensor = torch.randn(1, 3, 640, 360) # Replace with actual data

[0222] #Standardization process (if required)

[0223] input_tensor = input_tensor / 255.0 # Normalize to [0,1]

[0224] model = torch.load('pig_weight_estimation_model.pth') # Load the trained model

[0225] model.eval() # Sets the model to evaluation mode

[0226] with torch.no_grad():

[0227] predicted_weight = model(input_tensor) # Forward propagation

[0228] print(f"Predicted weight:{predicted_weight.item():.2f}kg") #Example output: Predicted weight: 68.34kg

[0229] The final output is the estimated weight of the pig.

[0230] Based on the above technical solution, the training method for multimodal pig weight estimation provided in this application solves the technical problems of poor practicality and insufficient weight estimation accuracy of the prior art.

[0231] In one possible implementation of this application embodiment, the above-mentioned S205 can be specifically implemented by the following S301, which will be described in detail below:

[0232] S301. Adjust the resolution of the scaled grayscale image to the preset resolution.

[0233] In some implementations, adjusting the resolution of the scaled grayscale image to a preset resolution includes:

[0234] Compare the scaling ratio with the scaling threshold;

[0235] When the scaling ratio is less than 1, the edge regions of the grayscale image are filled with zero pixels to the preset resolution to obtain the first channel data.

[0236] When the scaling ratio is greater than 1, the grayscale image is cropped to a preset resolution according to the principle that no pigs to be estimated are lost, and the first channel data is obtained.

[0237] When the scaling factor is equal to 1, the grayscale image is labeled as the first channel data.

[0238] For example, the specific code for adjusting the resolution of a grayscale image to a preset resolution using Python is as follows:

[0239]

[0240] The final output is a scaled grayscale image with a resolution of 640×360.

[0241] In one possible implementation of this application embodiment, the above-mentioned S208 can be specifically implemented by the following S401, S402 and S403, which are described in detail below:

[0242] S401, the backbone network of the weight estimation model is MobileNetV4.

[0243] In some implementations, MobileNetV4 is chosen as the backbone network to ensure training accuracy and efficiency, taking into account edge deployment.

[0244] For example, the MobileNetV4 backbone network is implemented using Python, and the specific code is as follows:

[0245]

[0246]

[0247] By inputting three-channel data (batch_size, 3, 640, 360), a feature map (batch_size, 1280, 20, 11) can be output.

[0248] S402. The regression head module of the weight estimation model includes: input layer, N CBS modules and output layer.

[0249] The input layer takes in the features output by the backbone network, while the output layer outputs features of size batch_size*1.

[0250] In some implementations, such as Figure 7 As shown, the design of the regression head module involves passing the output features of the backbone network through N CBS modules to obtain features with an output size of batch_size*1*1*1, and finally performing a Squeeze operation to obtain an output result of batch_size*1.

[0251] For example, the regression head module can be implemented using Python, and the specific code is as follows:

[0252]

[0253] The final output is the feature of batch_size*1.

[0254] S403. The loss function of the weighting model is the weighted mean squared error loss function WeightedMSELoss.

[0255] The formula for calculating WeightedMSELoss is as follows:

[0256]

[0257] In the formula, L w This represents the loss calculated for one sample size; N represents the set value for the sample size; P i t represents the predicted weight of pigs by the model. i This represents the actual weight of the pig.

[0258] It should be noted that, in designing the loss function, for pigs weighing between 10kg and 160kg, directly using MSE as the loss function would introduce a bias towards larger weights during regression. Therefore, a weighted mean squared error loss function (WeightedMSELoss) is used to reduce the bias introduced by weight differences.

[0259] For example, the WeightedMSELoss loss function is implemented using Python, and the specific code is as follows:

[0260]

[0261] The final output is the loss value.

[0262] Based on the above technical solution, the designed weight estimation model further improves the network's regression ability on pig weight and increases the accuracy of weight estimation.

[0263] The foregoing mainly describes the solutions of the embodiments of this application from the perspective of device implementation. It is understood that each device, such as an electronic device, includes at least one of the hardware structures and software modules corresponding to the execution of each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0264] This application embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0265] When using integrated units, Figure 8 A possible structural schematic diagram of the electronic device (referred to as electronic device 80) involved in the above embodiments is shown. The electronic device 80 includes a processing unit 801 and a communication unit 802, and may also include a storage unit 803. Figure 8 The structural diagram shown can be used to illustrate the structure of the electronic device involved in the above embodiments.

[0266] when Figure 8 The schematic diagram shown is used to illustrate the structure of the electronic device involved in the above embodiments. The processing unit 801 is used to control and manage the operation of the electronic device, the communication unit 802 is used for the electronic device to communicate with other devices, and the storage unit 803 is used to store the program code and data of the electronic device.

[0267] For example, communication unit 802 is used to acquire data images of pigs to be weighed;

[0268] The processing unit 801 is used to segment the RGB image of the pig to be weighed, obtaining the corresponding segmented regions and detection boxes; map the segmented regions and detection boxes of the RGB image onto the depth map by pixel alignment, calculate the average value of the non-zero values ​​of pixels in the non-segmented regions within the detection boxes in the depth map, and obtain the vertical distance h_c of the ground where the pig to be weighed is standing from the camera; set the RGB images and depth maps of different 3D cameras to the same scale h_s; perform grayscale conversion, mask segmentation, scaling and standardization on the RGB image to obtain the first channel data; extract the pig's height feature based on the depth map and perform standardization to obtain the second channel data; extract the pig's back thickness feature based on the depth map and perform standardization to obtain the third channel data; input the first channel data, second channel data and third channel data information into the weight estimation model to obtain the estimated weight.

[0269] The processing unit 801 can be a processor or a controller, and the communication unit 802 can be a communication interface, transceiver, transceiver circuit, transceiver device, etc. The term "communication interface" is a general term and may include one or more interfaces. The storage unit 803 can be a memory. When the electronic device 80 is a chip, the processing unit 801 can be a processor or a controller, and the communication unit 802 can be an input interface and / or an output interface, pins, or circuits, etc. The storage unit 803 can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip (e.g., read-only memory (ROM), random access memory (RAM, etc.).

[0270] The communication unit can also be called a transceiver unit. The antenna and control circuit with transceiver functions in the electronic device 80 can be considered as the communication unit 802 of the electronic device 80, and the processor with processing functions can be considered as the processing unit 801 of the electronic device 80. Optionally, the device in the communication unit 802 used to implement the receiving function can be considered as the communication unit. The communication unit is used to execute the receiving steps in the embodiments of this application, and the communication unit can be a receiver, a receiver circuit, etc. The device in the communication unit 802 used to implement the transmitting function can be considered as the transmitting unit. The transmitting unit is used to execute the transmitting steps in the embodiments of this application, and the transmitting unit can be a transmitter, a transmitter, a transmitting circuit, etc.

[0271] Figure 8If the integrated units in the process are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0272] Figure 8 The units in the process can also be called modules; for example, a processing unit can be called a processing module.

[0273] This application also provides a hardware structure diagram of an electronic device (denoted as electronic device 90), see [link to diagram]. Figure 9 The electronic device 90 includes a processor 901, and optionally, a memory 902 connected to the processor 901.

[0274] In the first possible implementation, see Figure 9 The electronic device 90 also includes a transceiver 903. The processor 901, memory 902, and transceiver 903 are connected via a bus. The transceiver 903 is used to communicate with other devices or communication networks. Optionally, the transceiver 903 may include a transmitter and a receiver. The device in the transceiver 903 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of this application. The device in the transceiver 903 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of this application.

[0275] Based on the first possible implementation method Figure 9 The structural diagram shown can be used to illustrate the structure of the electronic device involved in the above embodiments.

[0276] in, Figure 9 This can also be illustrated by a system chip in an electronic device. In this case, the actions performed by the aforementioned electronic device can be implemented by this system chip; the specific actions performed can be found above and will not be repeated here.

[0277] In implementation, each step of the method provided in this embodiment can be completed by integrated logic circuits in the processor or by instructions in software form. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0278] The processor in this application may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and other computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor may be a standalone semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it may form a System-on-a-Chip (SoC) with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits), or it may be integrated as a built-in processor within an ASIC. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), or logic circuits that implement dedicated logic operations.

[0279] The memory in the embodiments of this application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; or electrically erasable programmable-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0280] This application also provides a computer-readable storage medium including instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0281] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0282] This application also provides a chip including a processor and an interface circuit. The interface circuit is coupled to the processor. The processor is used to run computer programs or instructions to implement the above-described method. The interface circuit is used to communicate with other modules outside the chip.

[0283] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0284] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, the disclosure, and the appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0285] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A training method based on multi-modal pig weighing, characterized in that, The method comprises the following steps: acquiring a data image of a pig to be weighed; wherein the data image comprises an RGB image and a depth map; performing image segmentation on the RGB image of the pig to be weighed to obtain a corresponding segmentation region and a detection box; mapping the segmentation region and the detection box of the RGB image to the depth map through pixel alignment, counting the non-zero values of the pixels in the non-segmentation region within the detection box in the depth map, and calculating the average value to obtain a vertical distance h_c of the ground on which the pig to be weighed stands from the camera; setting the RGB image and the depth map of different 3D cameras to the same scale h_s; performing grayscale, mask segmentation and scaling normalization processing on the RGB image to obtain first channel data; extracting pig height features based on the depth map and performing standardization processing to obtain second channel data; extracting pig back thickness features based on the depth map and performing standardization processing to obtain third channel data; inputting the first channel data, the second channel data and the third channel data into a weight estimation model to obtain an estimated weight; wherein the weight estimation model is obtained based on training a PyTorch framework, and the PyTorch framework comprises a backbone network, a regression head module and a loss function module.

2. The method of claim 1, wherein, The calculation method of the vertical distance h_c of the ground on which the pig to be weighed stands from the camera is as follows: wherein M is the total number of pixels satisfying the condition; D(x, y) is the depth value at coordinate (x, y) in the depth map; B is the area range of the detection box, B = {(x, y) | x min ≤ x ≤ x max ,y min ≤ y ≤ y max}; S is the segmentation mask, S = {(x, y) | (x, y) ∈ segmentation region}.

3. The method of claim 1, wherein, performing grayscale, mask segmentation and scaling normalization processing on the RGB image, comprising: S51, converting the RGB image into a grayscale image; S52, setting all pixel values within the segmentation region of the grayscale image to a preset value p_s, and setting all pixel values of the non-segmentation region to zero; S53, scaling the gray-scale image to the h_s scale according to a scaling ratio; wherein the calculation formula of the scaling ratio rate is: S54, adjusting the resolution of the scaled grayscale image to a preset resolution to obtain the first channel data.

4. The method of claim 3, wherein, The adjustment of the resolution of the scaled grayscale image to the preset resolution comprises: comparing the scaling ratio with a scaling threshold value; when the scaling ratio is less than 1, using zero pixels to fill the edge region of the grayscale image to the preset resolution to obtain the first channel data; when the scaling ratio is greater than 1, cutting the grayscale image to the preset resolution without losing the pig to be weighed to obtain the first channel data; when the scaling ratio is equal to 1, marking the grayscale image as the first channel data.

5. The method of claim 1, wherein, The extraction of pig height features based on the depth map and the standardization processing comprise the following steps: S61, performing a mask operation on the depth map based on the segmentation region, retaining the original depth values within the segmentation region, and setting all depth values outside the segmentation region to zero to obtain a depth object mask graph d_m; S62, setting the pixel values within the blank depth map segmentation region to h_c, and keeping the pixel values outside the segmentation region as zero to obtain a height reference graph d_m_h; S63, generating a first image d_h containing pig back position distance information from the ground by difference operation of the height reference graph and the depth object mask graph; S64, scaling the first image d_h according to a scaling ratio rate to obtain a second image d_h_16; S65, set the maximum height of the pig p_m, map [0, p_m] to [0, 255] in order according to the pixel value of the second image d_h_16, obtain the third image of 8 bits, and mark the third image as the second channel data.

6. The method of claim 1, wherein, The pig back thickness feature is extracted based on the depth map and standardized, including the following steps: S71, extract the maximum value p_max and the non-zero minimum value p_min of the pixel value in the first image d_h. S72, subtract the non-zero value in the first image d_h from the non-zero minimum value p_min pixel by pixel to obtain the pig back relative thickness information d_h_ref. S73, perform 8-bit processing on the pig back relative thickness information d_h_ref, perform pixel traversal, set the pixel value greater than 255 to 255, and obtain the third channel data.

7. The method of claim 1, wherein, The backbone network is MobileNetV4.

8. The method of claim 1, wherein, The regression head module includes an input layer, N CBS modules and an output layer; wherein the input content of the input layer is the feature output by the backbone network; and the output result of the output layer is a feature with a size of batch_size*1.

9. The method of claim 1, wherein, The loss function is a weighted mean squared error loss function WeightedMSELoss; wherein, the calculation formula of WeightedMSELoss is: L w Loss calculated by a sample number; N represents the value of sample number setting; P i The pig weight predicted by the model; t i The true value of the pig weight.

10. An electronic device, comprising: It includes: A communication unit and a processing unit; The communication unit is used to obtain the data image of the pig to be estimated weight. The processing unit is used to perform image segmentation on the RGB image of the pig to be estimated weight to obtain the corresponding segmentation region and detection frame; map the segmentation region and detection frame of the RGB image to the depth map through pixel alignment; after counting the non-zero value of the non-segmentation region pixel in the detection frame in the depth map, calculate the average value to obtain the vertical distance h_c of the ground on which the pig to be estimated weight stands from the camera; set the RGB image and the depth map of different 3D cameras to the same scale h_s; perform grayscale, mask segmentation and scaling normalization processing on the RGB image to obtain the first channel data; extract the pig height feature based on the depth map and perform standardization processing to obtain the second channel data; extract the pig back thickness feature based on the depth map and perform standardization processing to obtain the third channel data; input the first channel data, the second channel data and the third channel data information into the weight estimation model to obtain the estimated weight.

Images