A method and device for estimating the deformation amount of a sampling spoon of a rare earth electrolytic furnace

Abstract

The application provides a deformation amount estimation method and device for a rare earth electrolytic furnace sampling spoon, the method comprising: acquiring an RGB image and a depth image corresponding to a target sampling spoon in a rare earth electrolytic furnace respectively, and obtaining a normal vector map corresponding to the depth image; outputting a plurality of key point corresponding Gaussian heat maps corresponding to the target sampling spoon from a convolutional neural network model; performing three-dimensional projection on each key point to obtain a three-dimensional coordinate of each key point; performing rigid coarse registration on each three-dimensional coordinate based on a preset model of the target sampling spoon to determine a rotation matrix and a translation vector of each key point and performing elastic registration on each key point in the preset model to obtain a target three-dimensional coordinate of each key point and estimate a length deformation amount, a width deformation amount, a height deformation amount and an angle deformation amount of the target sampling spoon. Through the above method, the accuracy, efficiency, real-time performance and stability of deformation detection of the sampling spoon in the rare earth electrolytic furnace are improved.

Description

Technical Field

[0001] This application relates to the field of intelligent detection technology in rare earth smelting processes, and in particular to a method and apparatus for estimating the deformation of a sampling spoon in a rare earth electrolysis furnace. Background Technology

[0002] With the rapid development of the rare earth industry, the requirements for automation and precision in the rare earth metal electrolytic furnace smelting process are becoming increasingly stringent. In rare earth metal electrolytic production, the sampling spoon, as an important tool for scooping during the smelting process, needs to frequently perform scooping operations in harsh environments with high temperatures and strong chemical corrosion. The sampling spoon is prone to deformation when exposed to such environments for a long time. If it is not detected and dealt with in time, it will reduce the stability and safety of the sampling operation and affect the quality of product quality monitoring and process optimization.

[0003] Currently, traditional methods for detecting deformation of high-temperature sampling spoons in rare earth electrolysis furnaces generally rely on periodic manual inspections or empirical judgment based on simple physical parameters. These methods are inefficient and lack real-time performance. They are also limited by the subjective factors of the inspectors and environmental interference, making it difficult to accurately capture the subtle deformation characteristics of the sampling spoons. Furthermore, existing automated detection technologies lack the ability to deeply analyze deformation characteristics under complex working conditions, resulting in significant deficiencies in detection accuracy and reliability, which cannot meet the high-quality and high-efficiency production requirements of rare earth smelting. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method and apparatus for estimating the deformation of a sampling spoon in a rare earth electrolysis furnace. This method involves acquiring RGB and depth images of the sampling spoon in the rare earth electrolysis furnace, obtaining a normal vector map including the normal vector corresponding to each pixel in the depth image, and using a convolutional neural network model to output Gaussian heatmaps corresponding to multiple key points of the sampling spoon based on the RGB images and the normal vector map. Each key point is then subjected to 3D projection to obtain its corresponding 3D coordinates. After rigid and elastic registration of the 3D coordinates, the deformation of the target sampling spoon is estimated. This method adapts to the deformation estimation requirements under various complex working conditions, improving the accuracy and efficiency of deformation detection of the sampling spoon in the rare earth electrolysis furnace, and thus enhancing the real-time performance and stability of deformation detection of the sampling spoon.

[0005] This application provides a method for estimating the deformation of a sampling spoon used in a rare earth electrolysis furnace. The method includes: The RGB image and depth image of the target sampling spoon in the rare earth electrolysis furnace, which are acquired by the image acquisition device, are obtained respectively, and the normal vector corresponding to each pixel in the depth image is determined to obtain the normal vector map corresponding to the depth image. The RGB image and the normal vector map are respectively input into a preset convolutional neural network model to obtain Gaussian heatmaps corresponding to multiple preset key points of the target sampling spoon, which are output by the convolutional neural network model. Based on the Gaussian heatmap corresponding to each key point and the device parameters of the image acquisition device, a three-dimensional projection is performed on each key point to obtain the three-dimensional coordinates corresponding to each key point; Based on the preset model corresponding to the target sampling spoon, rigid coarse registration is performed on each of the three-dimensional coordinates to determine the rotation matrix and translation vector of each key point in the preset model corresponding to each key point. Based on each of the three-dimensional coordinates, the rotation matrix and the translation vector, the calibration three-dimensional coordinates of each key point in the preset model are flexibly registered to obtain the target three-dimensional coordinates of each key point. Based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates, the deformation of the target sampling spoon is estimated; wherein the deformation includes at least length deformation, width deformation, height deformation and angle deformation.

[0006] This application embodiment also provides a deformation estimation device for a sampling spoon of a rare earth electrolysis furnace, the deformation estimation device comprising: The image acquisition module is used to acquire the RGB image and depth image of the target sampling spoon in the rare earth electrolysis furnace acquired by the image acquisition device, and to determine the normal vector corresponding to each pixel in the depth image, so as to obtain the normal vector map corresponding to the depth image. The model processing module is used to input the RGB image and the normal vector map into a preset convolutional neural network model respectively, and obtain the Gaussian heatmaps corresponding to multiple preset key points of the target sampling spoon output by the convolutional neural network model. A 3D projection module is used to perform 3D projection on each of the key points based on the Gaussian heatmap corresponding to each key point and the device parameters of the image acquisition device, so as to obtain the 3D coordinates corresponding to each key point; The rigid registration module is used to perform rigid coarse registration on each of the three-dimensional coordinates based on the preset model corresponding to the target sampling spoon, and to determine the rotation matrix and translation vector of each key point corresponding to each key point in the preset model. The flexible registration module performs flexible registration on the calibrated 3D coordinates of each key point in the preset model based on each of the 3D coordinates, the rotation matrix and the translation vector, to obtain the target 3D coordinates of each key point. The deformation estimation module is used to estimate the deformation of the target sampling spoon based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates; wherein the deformation includes at least length deformation, width deformation, height deformation and angle deformation.

[0007] This application embodiment also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the deformation estimation method for the sampling spoon of the rare earth electrolysis furnace described above are performed.

[0008] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method for estimating the deformation of a sampling spoon in a rare earth electrolysis furnace.

[0009] The present application provides a method and apparatus for estimating the deformation of a sampling spoon in a rare earth electrolysis furnace. The deformation estimation method includes: acquiring an RGB image and a depth image of the target sampling spoon in the rare earth electrolysis furnace, respectively, acquired by an image acquisition device, and determining the normal vector corresponding to each pixel in the depth image to obtain a normal vector map corresponding to the depth image; inputting the RGB image and the normal vector map into a preset convolutional neural network model to obtain Gaussian heatmaps corresponding to multiple preset key points of the target sampling spoon, respectively, output by the convolutional neural network model; and performing three-dimensional analysis on each key point based on the Gaussian heatmap corresponding to each key point and the device parameters of the image acquisition device. The three-dimensional projection is performed to obtain the three-dimensional coordinates corresponding to each key point; based on the preset model corresponding to the target sampling spoon, rigid coarse registration is performed on each of the three-dimensional coordinates to determine the rotation matrix and translation vector corresponding to each key point in the preset model; based on each of the three-dimensional coordinates, the rotation matrix, and the translation vector, elastic registration is performed on the calibration three-dimensional coordinates corresponding to each key point in the preset model to obtain the target three-dimensional coordinates corresponding to each key point; based on the calibration three-dimensional coordinates and the target three-dimensional coordinates, the deformation of the target sampling spoon is estimated; wherein, the deformation includes at least length deformation, width deformation, height deformation, and angle deformation.

[0010] Compared with existing technologies that rely on manual periodic sampling, experience-based judgment based on simple physical parameters, and existing automated detection methods, this new method acquires RGB and depth images of the sampling spoon in a rare earth electrolysis furnace. It obtains a normal vector map including the normal vector of each pixel in the depth image, and uses a convolutional neural network model to output Gaussian heatmaps corresponding to multiple key points of the sampling spoon based on the RGB images and normal vector maps. Each key point is then 3D projected to obtain its corresponding 3D coordinates. After rigid and elastic registration of the 3D coordinates, the deformation of the target sampling spoon is estimated. This method adapts to deformation estimation requirements under various complex working conditions, improving the accuracy and efficiency of deformation detection of the sampling spoon in the rare earth electrolysis furnace, and thus enhancing the real-time performance and stability of deformation detection.

[0011] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a method for estimating the deformation of a sampling spoon in a rare earth electrolysis furnace, provided as an embodiment of this application; Figure 2 A data processing flowchart for a method of estimating the deformation of a sampling spoon in a rare earth electrolysis furnace, provided in an embodiment of this application; Figure 3 A schematic diagram of the deformation estimation device for a rare earth electrolysis furnace sampling spoon provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0015] Research has revealed that traditional methods for detecting deformation of high-temperature sampling spoons in rare earth electrolysis furnaces typically rely on manual periodic sampling or empirical judgment based on simple physical parameters. These methods are inefficient and lack real-time performance, and are limited by the subjective factors of the inspectors and environmental interference, making it difficult to accurately capture the subtle deformation characteristics of the sampling spoons. Furthermore, existing automated detection technologies lack the ability to deeply analyze deformation characteristics under complex working conditions, resulting in significant deficiencies in detection accuracy and reliability, and failing to meet the high-quality and high-efficiency production requirements of rare earth smelting.

[0016] Based on this, this application provides a method for estimating the deformation of a sampling spoon in a rare earth electrolysis furnace. The method involves acquiring RGB and depth images of the sampling spoon in the rare earth electrolysis furnace, obtaining a normal vector map including the normal vector of each pixel in the depth image, and using a convolutional neural network model to output Gaussian heatmaps corresponding to multiple key points of the sampling spoon based on the RGB image and the normal vector map. Each key point is then subjected to 3D projection to obtain its corresponding 3D coordinates. After rigid and elastic registration of the 3D coordinates, the deformation of the target sampling spoon is estimated. This method adapts to the deformation estimation requirements under various complex working conditions, improving the accuracy and efficiency of deformation detection of the sampling spoon in the rare earth electrolysis furnace, and thus enhancing the real-time performance and stability of deformation detection of the sampling spoon.

[0017] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for estimating the deformation of a sampling spoon in a rare earth electrolysis furnace, as provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the method for estimating the deformation of a sampling spoon in a rare earth electrolysis furnace includes: S101. Acquire the RGB image and depth image of the target sampling spoon in the rare earth electrolysis furnace acquired by the image acquisition device, and determine the normal vector corresponding to each pixel in the depth image to obtain the normal vector map corresponding to the depth image.

[0018] It should be noted that a rare earth electrolytic furnace is a key piece of equipment used in rare earth metal smelting. It is mainly used to extract rare earth metals from rare earth oxides through molten salt electrolysis. Its working principle is to dissolve rare earth oxides in molten electrolyte, and then reduce rare earth ions to metals at the cathode under the action of current. A sampling spoon is a commonly used tool in a rare earth electrolytic furnace, used to take samples from molten metal or electrolyte.

[0019] Here, an RGB image is a color image composed of three color channels: red, green, and blue. The color of each pixel is formed by combining different intensities of these three color channels. A depth image, also known as a distance image, is a special type of image that records the distance information between objects in a scene and the camera. Each pixel value in a depth image represents the distance from the actual physical location of that pixel to the camera.

[0020] In this step, an image acquisition device is used to acquire the RGB image and depth image of the target sampling spoon in the rare earth electrolysis furnace at a preset position at the sampling port of the rare earth electrolysis furnace, and the normal vector corresponding to each pixel in the depth image is determined; finally, based on the normal vector corresponding to each pixel, a normal vector map corresponding to the depth image is generated.

[0021] The normal vector map encodes the curvature information of the sampling spoon surface, increasing the deformation sensitivity of the target sampling spoon.

[0022] In this embodiment of the application, the image acquisition device may include an industrial-grade RGB-D camera (e.g., Intel RealSense D435, etc.). When the image acquisition device acquires images, it is necessary to ensure stable ambient lighting to avoid ambient light interference. For example, the image resolution of the RGB image and the depth image can be set to 1920×1080 pixels and the frame rate to 30fps.

[0023] In one embodiment of this application, in specific implementation, the step of determining the normal vector corresponding to each pixel in the depth image in step S101 may include: S1011. Perform Gaussian filtering on the depth image to remove noise from the depth image.

[0024] In this step, a preset Gaussian filter is used to perform Gaussian filtering on the depth image to remove noise from the depth image.

[0025] In this embodiment of the application, the formula for Gaussian filtering is shown below.

[0026] .

[0027] in, Represents the coordinates of each pixel in the depth image; This indicates the standard deviation of the Gaussian filter setting (for example, the standard deviation can be set to 1.5, and the kernel size of the Gaussian filter can be set to 5×5).

[0028] S1012. Using the preset empty scene depth map of the rare earth electrolysis furnace, perform background subtraction on the noise-removed depth image to obtain the region image of the target sampling spoon in the depth image.

[0029] In this step, when performing background subtraction on the noise-removed depth image, firstly, the depth difference between the preset empty scene depth map (without the target sampling spoon) and the noise-removed depth image (including the target sampling spoon) is determined; then, the region in the noise-removed depth image that satisfies the depth difference greater than the preset depth threshold is used as the region image of the target sampling spoon in the depth image.

[0030] In this embodiment of the application, the expression for background difference is as follows.

[0031] .

[0032] in, This represents the region of the target sampling spoon within the depth image; This represents the preset empty scene depth map (without a target sampling spoon) of the rare earth electrolysis furnace. This represents the depth image after noise removal (including the target sampling spoon). This indicates a preset depth threshold (for example, the preset depth threshold can be set to 10mm).

[0033] S1013. Calculate the normal vector corresponding to each pixel in the region image using the surface gradient method.

[0034] In this embodiment of the application, the expression for the normal vector corresponding to each pixel in the region image calculated by the surface gradient method is as follows.

[0035] .

[0036] in, This represents the normal vector corresponding to each pixel in the region image; and These represent the neighborhood depth and neighborhood gradient corresponding to each pixel in the region image, respectively.

[0037] S102. Input the RGB image and the normal vector map into a preset convolutional neural network model respectively to obtain Gaussian heatmaps corresponding to multiple preset key points of the target sampling spoon output by the convolutional neural network model.

[0038] In this embodiment, the preset key points corresponding to the target sampling spoon may include, but are not limited to, key points at the end of the spoon handle, the center of the spoon bottom, the left edge, the right edge, other edge key points, the center of the upper edge of the spoon mouth, the center of the lower edge of the spoon mouth, the edge of the spoon mouth, the target key point of the spoon handle, the key point at the connection of the spoon handle, the key point of the spoon handle, and the key point at the inflection point of the spoon body. A Gaussian heatmap is a method that converts the location information of key points into a two-dimensional probability distribution map. It generates a heatmap that follows a two-dimensional Gaussian distribution by using an NxN region centered on the key point (for example, the region size in this embodiment is 3×3), representing the probability that the key point is the target point.

[0039] In one embodiment of this application, step S102 may include: S1021. Input the RGB image and the normal vector map into a preset convolutional neural network model respectively.

[0040] In this embodiment of the application, the preset convolutional neural network model may include a "UNet encoder-decoder" network model, which performs convolutional feature extraction and upsampling operations on RGB images and normal vector maps to output a Gaussian heatmap including key points of the sampling spoon.

[0041] Here, the convolutional neural network model has at least 4 input channels (three channels for RGB images and one channel for the normal vector map).

[0042] S1022. The encoder set in the convolutional neural network model performs encoding operations on the RGB image and the normal vector map respectively to obtain encoded information.

[0043] In the embodiments of this application, the encoder set in the convolutional neural network model includes multiple encoders; wherein each encoder layer includes a convolutional layer, a ReLU activation layer and a max pooling layer.

[0044] For example, the volume of the convolutional layer can be 3×3, the volume of the max pooling layer can be 2×2, and the number of encoder layers can be set to 5.

[0045] Among them, the ReLU (Rectified Linear Unit) activation layer is a computational layer that applies an activation function. For example, for any input value x, if x is greater than 0, the output is x; if x is less than or equal to 0, the output is 0. Max pooling is a common downsampling operation, mainly used to reduce the spatial dimensions (width and height) of the feature map while retaining the most important information.

[0046] S1023. The decoder set in the convolutional neural network model performs bilinear upsampling and skip connections on the encoded information to obtain the heat value information corresponding to multiple preset key points of the target sampling spoon.

[0047] In this step, the decoder set in the convolutional neural network model performs bilinear upsampling and skip connection decoding operations on the encoded information to obtain the heat value information corresponding to multiple preset key points of the target sampling spoon.

[0048] S1024. The convolutional neural network model performs Sigmoid probability activation on the heat value information corresponding to each key point, and outputs a Gaussian heatmap corresponding to each key point.

[0049] In this step, the convolutional layer (1×1) in the convolutional neural network model performs Sigmoid probability activation on the heat value information corresponding to each key point, and outputs the Gaussian heatmap corresponding to each key point.

[0050] In this embodiment of the application, the expression of the Gaussian heatmap corresponding to each key point is as follows.

[0051] .

[0052] in, Indicates the first Gaussian heatmaps corresponding to key points; and These represent the width and height of the Gaussian heatmap, respectively. Indicates the number of key points.

[0053] Here, the peak position of the Gaussian heatmap corresponding to each key point corresponds to the key point, and the size of the Gaussian heatmap area is set to 3×3 to ensure the robustness of key point localization.

[0054] In this embodiment of the application, the expression for the loss function of the convolutional neural network model is as follows.

[0055] .

[0056] in, Indicates the total number of pixels; Indicates the first Truth heatmaps corresponding to key points; Indicates the first Gaussian heatmaps corresponding to key points; This represents the mean squared error (MSE) loss value.

[0057] S103. Based on the Gaussian heatmap corresponding to each key point and the device parameters of the image acquisition device, perform a three-dimensional projection on each key point to obtain the three-dimensional coordinates corresponding to each key point.

[0058] In this embodiment of the application, the device parameters of the image acquisition device include, but are not limited to, the intrinsic parameter matrix and depth value of the image acquisition device; wherein, the intrinsic parameter matrix includes the focal length and principal point coordinates of the image acquisition device, etc.

[0059] In one embodiment of this application, step S103 may include: S1031. Determine the response pixel coordinates of the key point in the Gaussian heatmap corresponding to each key point.

[0060] In this embodiment, the response pixel coordinates are the maximum response positions of each key point in the corresponding Gaussian heatmap, that is, the peak positions in the Gaussian heatmap corresponding to each key point.

[0061] S1032. Perform sub-pixel optimization on the response pixel coordinates corresponding to each key point to obtain the sub-pixel coordinates corresponding to each key point.

[0062] In this step, taking the response pixel coordinates corresponding to each key point as the center, a neighborhood (e.g., 3×3) within a preset range near the center is taken, and a quadratic surface is fitted by bilinear interpolation to solve for the sub-pixel coordinates corresponding to each key point.

[0063] In this embodiment of the application, the expression for solving the sub-pixel coordinates corresponding to each key point is as follows.

[0064] .

[0065] Among them, setting The response pixel coordinates corresponding to each keypoint. Sub-pixel coordinates corresponding to each keypoint; Represents the x-coordinate of a pixel; Represents the ordinate of a pixel; This represents the probability value in the Gaussian heatmap.

[0066] S1033. Based on the device parameters of the image acquisition device and the coordinates of each sub-pixel, the three-dimensional coordinates corresponding to each key point are determined using a preset projection formula, so as to perform three-dimensional projection on each key point.

[0067] In this embodiment of the application, the expression of the preset projection formula is as follows.

[0068] .

[0069] in, This represents the three-dimensional coordinates corresponding to each key point; and These represent the intrinsic parameter matrix and depth value in the device parameters of the image acquisition device, respectively. This represents the sub-pixel coordinates corresponding to each keypoint, i.e. .

[0070] Here, the expression for the intrinsic parameter matrix in the device parameters of the image acquisition device is shown below.

[0071] .

[0072] in, This represents the intrinsic parameter matrix in the device parameters of the image acquisition device; and Indicates the focal length of the image acquisition device; , () represents the principal point coordinates of the image acquisition device.

[0073] S104. Based on the preset model corresponding to the target sampling spoon, perform rigid coarse registration on each of the three-dimensional coordinates to determine the rotation matrix and translation vector of each key point corresponding to each key point in the preset model.

[0074] In the embodiments of this application, the registration method for each three-dimensional coordinate may include rigid coarse registration and elastic registration; wherein, rigid registration refers to the process of aligning one object with another object through translation and rotation operations. This transformation does not change the shape or size of the object, but only changes its position and orientation; elastic registration allows for more complex transformations, including but not limited to non-rigid deformations such as scaling, twisting, and bending.

[0075] In one embodiment of this application, step S104 may include: S1041. Determine the calibration three-dimensional coordinates corresponding to each key point in the preset model corresponding to the target sampling spoon.

[0076] In this embodiment of the application, the preset model corresponding to the target sampling spoon may include a CAD model, in which the target sampling spoon corresponds to multiple three-dimensional coordinates, that is, each preset key point of the target sampling spoon corresponds to a calibrated three-dimensional coordinate in the CAD model.

[0077] S1042. Based on the calibration three-dimensional coordinates and the three-dimensional coordinates corresponding to each key point, the rotation matrix and translation vector of each three-dimensional coordinate relative to the corresponding calibration three-dimensional coordinate are determined by point distance transformation and iterative nearest point solution method.

[0078] In this step, the three-dimensional coordinates corresponding to each key point are rigidly registered with the calibration three-dimensional coordinates corresponding to each key point in the preset model corresponding to the target sampling spoon to establish an alignment relationship. The rotation matrix and translation vector of each three-dimensional coordinate relative to the corresponding calibration three-dimensional coordinate are determined by the point distance transformation solution method and the iterative nearest point (ICP) algorithm.

[0079] In the embodiments of this application, the expressions for the point distance transformation solution method and the Iterative Closest Point (ICP) algorithm are as follows.

[0080] .

[0081] in, Represents the rotation matrix; Represents the translation vector; This represents the three-dimensional coordinates corresponding to each key point; This represents the calibrated 3D coordinates corresponding to each key point.

[0082] S105. Based on each of the three-dimensional coordinates, the rotation matrix, and the translation vector, perform elastic registration on the calibrated three-dimensional coordinates corresponding to each key point in the preset model to obtain the target three-dimensional coordinates corresponding to each key point.

[0083] In one embodiment of this application, step S105 may include: S1051. Based on the rotation matrix and the translation vector, determine the registration three-dimensional coordinates corresponding to each of the three-dimensional coordinates.

[0084] In this step, the Singular Value Decomposition (SVD) method is used to register the rotation matrix and translation vector to each 3D coordinate, thereby obtaining the registered 3D coordinates corresponding to each key point and completing the rigid registration of each 3D coordinate.

[0085] Here, the convergence threshold of the singular value decomposition method can be set to 1e-5 so that the average error of the point cloud after registration is less than 1mm.

[0086] S1052. Based on the registration three-dimensional coordinates and calibration three-dimensional coordinates corresponding to each key point, the sampling spoon deformation function of each calibration three-dimensional coordinate relative to the corresponding registration three-dimensional coordinate is determined using the TPS elastic registration formula corresponding to the preset sampling spoon deformation function.

[0087] In this embodiment of the application, the expression of the preset deformation function is as follows.

[0088] .

[0089] Among them, deformation function It can be represented by radial basis functions.

[0090] Here, the TPS elastic registration algorithm can achieve smooth image deformation by minimizing "bending energy" while ensuring accurate alignment of matching point pairs. The TPS elastic registration formula can be represented by minimizing the objective function. The expression of the TPS elastic registration formula corresponding to the sampling spoon deformation function is shown below.

[0091] .

[0092] in, This represents the deformation function of the sampling spoon (minimizing the "bending energy"). This represents the registration 3D coordinates corresponding to each key point; This represents the calibrated 3D coordinates corresponding to each key point; Dependent on deformation function exist The Hessian matrix at that location; This represents the regularization parameter (for example, it can be set to 0.1).

[0093] Furthermore, by using the TPS elastic registration algorithm, the deformation function of the sampling spoon is solved, and the calibration three-dimensional coordinates of each key point in the preset model corresponding to the target sampling spoon are deformed, thereby indirectly realizing further correction of the three-dimensional coordinates of each key point, so as to facilitate more accurate measurement of the sampling spoon.

[0094] S1053. Based on the deformation function of the sampling spoon, the calibrated three-dimensional coordinates corresponding to each key point are subjected to bending deformation to obtain the deformation target three-dimensional coordinates corresponding to each key point.

[0095] In this step, the sampling spoon deformation function obtained by solving the sampling spoon deformation function is applied to the calibration three-dimensional coordinates corresponding to each key point to perform elastic registration of the calibration three-dimensional coordinates and obtain the target three-dimensional coordinates corresponding to each key point.

[0096] S106. Based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates, estimate the deformation of the target sampling spoon.

[0097] The deformation variables include at least length deformation variables, width deformation variables, height deformation variables, and angle deformation variables.

[0098] In one embodiment of this application, when estimating the length deformation corresponding to the target sampling spoon, step S106 may include: S1061. Determine the first Manhattan distance between the first calibrated three-dimensional coordinates corresponding to the center key point of the upper edge of the spoon mouth and the second calibrated three-dimensional coordinates corresponding to the center key point of the lower edge of the spoon mouth, and determine the first Manhattan distance as the target length value corresponding to the target sampling spoon.

[0099] In this embodiment of the application, the expression for determining the target length value corresponding to the target sampling spoon in the preset model is as follows.

[0100] .

[0101] in, This indicates the target length value corresponding to the target sampling spoon in the preset model; This represents the first calibrated 3D coordinates corresponding to the key point at the center of the upper edge of the spoon rim. This represents the second calibration three-dimensional coordinates corresponding to the key point at the center of the lower edge of the spoon rim.

[0102] S1062. Calculate the second Manhattan distance between the first target three-dimensional coordinates corresponding to the center key point of the upper edge of the spoon mouth and the second target three-dimensional coordinates corresponding to the center key point of the lower edge of the spoon mouth, and determine the second Manhattan distance as the deformation length value corresponding to the target sampling spoon.

[0103] In this embodiment of the application, the expression for determining the deformation length value corresponding to the target sampling spoon is as follows.

[0104] .

[0105] in, This indicates the deformation length value corresponding to the target sampling spoon; This indicates the three-dimensional coordinates of the first target corresponding to the key point at the center of the upper edge of the spoon mouth; This represents the three-dimensional coordinates of the second target corresponding to the key point at the center of the lower edge of the spoon.

[0106] S1063. The difference between the deformation length value and the target length value is determined as the estimated length deformation of the target sampling spoon.

[0107] In this embodiment of the application, the expression for estimating the length deformation of the target sampling spoon is as follows.

[0108] .

[0109] in, This represents the estimated length deformation of the target sampling spoon; This indicates the deformation length value corresponding to the target sampling spoon; This indicates the target length value corresponding to the target sampling spoon.

[0110] In one embodiment of this application, when estimating the width deformation corresponding to the target sampling spoon, step S106 may include: S1064. Determine the third Manhattan distance between the third calibration three-dimensional coordinates corresponding to the left edge key point and the fourth calibration three-dimensional coordinates corresponding to the right edge key point, and determine the third Manhattan distance as the target width value corresponding to the target sampling spoon.

[0111] In this embodiment of the application, the expression for determining the target width value corresponding to the target sampling spoon in the preset model is as follows.

[0112] .

[0113] in, This indicates the target width value of the target sampling spoon in the preset model; This indicates the third calibration 3D coordinates corresponding to the key points on the left edge. This indicates the fourth calibration 3D coordinates corresponding to the key points on the right edge.

[0114] S1065. Calculate the fourth Manhattan distance between the three-dimensional coordinates of the third target corresponding to the left edge key point and the three-dimensional coordinates of the fourth target corresponding to the right edge key point, and determine the fourth Manhattan distance as the deformation width value corresponding to the target sampling spoon.

[0115] In this embodiment of the application, the expression for determining the deformation width value corresponding to the target sampling spoon is as follows.

[0116] .

[0117] in, This indicates the deformation width value corresponding to the target sampling spoon; This indicates the 3D coordinates of the third target corresponding to the key point on the left edge; This represents the three-dimensional coordinates of the fourth target corresponding to the key point on the right edge.

[0118] S1066. The difference between the deformation width value and the target width value is determined as the estimated width deformation value corresponding to the target sampling spoon.

[0119] In this embodiment of the application, the expression for estimating the width deformation of the target sampling spoon is as follows.

[0120] 。

[0121] in, This represents the estimated width deformation of the target sampling spoon; This indicates the deformation width value corresponding to the target sampling spoon; This indicates the target width value corresponding to the target sampling spoon.

[0122] In one embodiment of this application, when estimating the height deformation corresponding to the target sampling spoon, step S106 may include: S1067. Determine the first vertical distance between the fifth calibrated three-dimensional coordinates corresponding to the center key point of the spoon bottom and the plane fitted by the edge key points of the spoon mouth in the preset model, and determine the first vertical distance as the target height value corresponding to the target sampling spoon.

[0123] Here, there is at least one key point at the center of the spoon bottom; and at least three key points at the edge of the spoon rim.

[0124] In this embodiment of the application, when the number of key points on the edge of the spoon is 4, the expression for determining the target height value corresponding to the target sampling spoon is as follows.

[0125] .

[0126] in, This indicates the fifth calibration three-dimensional coordinate corresponding to the key point at the center of the spoon bottom; This represents the plane equation fitted to the key points of the spoon's edge in the preset model; This indicates the target height value corresponding to the target sampling spoon.

[0127] S1068. Determine the second vertical distance between the third-dimensional coordinates of the fifth target corresponding to the key point at the center of the spoon bottom and the plane fitted by the multiple key points of the spoon mouth, and determine the second vertical distance as the deformation height value corresponding to the target sampling spoon.

[0128] In this embodiment of the application, the fitting expression for determining the deformation height value corresponding to the target sampling spoon is as follows.

[0129] .

[0130] in, This indicates the three-dimensional coordinates of the fifth target corresponding to the key point at the center of the spoon's bottom; This represents the plane equation fitted based on the key points of the deformed spoon edge; This indicates the deformation height value corresponding to the target sampling spoon.

[0131] S1069. The difference between the deformation height value and the target height value is determined as the estimated height deformation value corresponding to the target sampling spoon.

[0132] In this embodiment of the application, the expression for estimating the height deformation of the target sampling spoon is as follows.

[0133] .

[0134] in, This represents the estimated height deformation of the target sampling spoon; This indicates the deformation height value corresponding to the target sampling spoon; This indicates the target height value corresponding to the target sampling spoon.

[0135] In one embodiment of this application, when estimating the angular deformation corresponding to the target sampling spoon, step S106 may include: S10610. Based on the sixth calibration three-dimensional coordinates corresponding to the target key point of the spoon handle, the seventh calibration three-dimensional coordinates corresponding to the key point of the spoon handle connection, and the plane fitted by the key point of the spoon edge in the preset model, the target angle value corresponding to the target sampling spoon is determined using a preset angle formula.

[0136] In this step, during specific implementation, firstly, the difference between the seventh calibration three-dimensional coordinate corresponding to the key point at the connection of the spoon handle and the sixth calibration three-dimensional coordinate corresponding to the target key point of the spoon handle is calculated to determine the handle direction vector in the preset model corresponding to the target sampling spoon; then, based on the handle direction vector and the spoon mouth plane normal vector in the preset model corresponding to the target sampling spoon, the target included angle value corresponding to the target sampling spoon is determined using the preset included angle formula.

[0137] In this embodiment of the application, the expression for determining the target angle value corresponding to the target sampling spoon using the preset angle formula is as follows.

[0138] , .

[0139] in, This indicates the included angle value corresponding to the target sampling spoon; This indicates the sixth calibration 3D coordinate corresponding to the key point of the spoon handle target; The seventh calibration three-dimensional coordinates are shown for the key points at the connection of the spoon handle. This represents the handle direction vector in the preset model corresponding to the target sampling spoon; This represents the normal vector of the spoon rim plane in the preset model corresponding to the target sampling spoon.

[0140] S10611. Determine the difference between the three-dimensional coordinates of the sixth target corresponding to the key point of the spoon handle and the three-dimensional coordinates of the seventh target corresponding to the key point of the spoon handle connection, and normalize the difference. The normalized difference is determined as the deformation vector of the spoon handle direction corresponding to the target sampling spoon.

[0141] In this embodiment of the application, the expression for determining the deformation angle value of the target sampling spoon using the preset angle formula is as follows.

[0142] .

[0143] in, This indicates the included angle value corresponding to the deformation of the target sampling spoon; This indicates the three-dimensional coordinates of the sixth target corresponding to the key point of the spoon handle target; This represents the three-dimensional coordinates of the seventh target corresponding to the key point at the connection of the spoon handle; This represents the fitting of the normal vector of the spoon rim plane in the preset model corresponding to the target sampling spoon.

[0144] S10412. The difference between the deformation angle value and the target angle value is determined as the estimated angle deformation value corresponding to the target sampling spoon.

[0145] In this embodiment of the application, the expression for estimating the angle deformation corresponding to the target sampling spoon is as follows.

[0146] .

[0147] in, This represents the estimated angular deformation corresponding to the target sampling spoon; This indicates the included angle value corresponding to the deformation of the target sampling spoon; This indicates the target included angle value corresponding to the target sampling spoon.

[0148] Please see Figure 2 , Figure 2 This is a data processing flowchart illustrating a method for estimating the deformation of a sampling spoon in a rare earth electrolysis furnace, as provided in an embodiment of this application. Figure 2As shown, the normal vector maps corresponding to the RGB image and the depth image are input into the convolutional neural network model, and the convolutional neural network model outputs Gaussian heatmaps corresponding to multiple preset key points of the target sampling spoon. In each Gaussian heatmap, the response pixel coordinates of the key points in the Gaussian heatmap are determined and sub-pixel optimization is performed. Finally, the length deformation, width deformation, height deformation and angle deformation corresponding to the target sampling spoon are determined through rigid registration, elastic registration and deformation estimation.

[0149] Thus, in the deformation estimation method of the sampling spoon of the rare earth electrolysis furnace provided in this application embodiment, the powerful image feature extraction and analysis capabilities of the convolutional neural network can accurately identify the subtle deformation of the sampling spoon, overcoming the subjectivity and low precision problems of traditional methods; by real-time acquisition and analysis of sampling spoon image data, dynamic monitoring and rapid prediction of deformation state can be achieved, effectively improving the response speed of the production process; it can adapt to the harsh working conditions such as high temperature and complex chemical environment of rare earth electrolysis furnace, ensuring the stability and reliability of the detection process, and providing solid data support and safety guarantee for rare earth smelting production.

[0150] The deformation estimation method for a sampling spoon in a rare earth electrolysis furnace provided in this application involves acquiring RGB and depth images of the sampling spoon in the rare earth electrolysis furnace, obtaining a normal vector map including the normal vector corresponding to each pixel in the depth image, and using a convolutional neural network model to output Gaussian heat maps corresponding to multiple key points of the sampling spoon based on the RGB image and the normal vector map. Each key point is then subjected to 3D projection to obtain its corresponding 3D coordinates. After rigid and elastic registration of the 3D coordinates, the deformation of the target sampling spoon is estimated. This method adapts to the deformation estimation requirements under various complex working conditions, improving the accuracy and efficiency of deformation detection of the sampling spoon in the rare earth electrolysis furnace, and thus enhancing the real-time performance and stability of deformation detection of the sampling spoon.

[0151] Please see Figure 3 , Figure 3 This is a schematic diagram of the deformation estimation device for a sampling spoon in a rare earth electrolysis furnace, provided in an embodiment of this application. Figure 3 As shown, the deformation estimation device 300 includes: The image acquisition module 310 is used to acquire the RGB image and depth image of the target sampling spoon in the rare earth electrolysis furnace acquired by the image acquisition device, and to determine the normal vector corresponding to each pixel in the depth image, so as to obtain the normal vector map corresponding to the depth image. The model processing module 320 is used to input the RGB image and the normal vector map into a preset convolutional neural network model respectively, and obtain the Gaussian heatmaps corresponding to multiple preset key points of the target sampling spoon output by the convolutional neural network model. The three-dimensional projection module 330 is used to perform three-dimensional projection on each key point based on the Gaussian heatmap corresponding to each key point and the device parameters of the image acquisition device, so as to obtain the three-dimensional coordinates corresponding to each key point; The rigid registration module 340 is used to perform rigid coarse registration on each of the three-dimensional coordinates based on the preset model corresponding to the target sampling spoon, and to determine the rotation matrix and translation vector of each key point corresponding to each key point in the preset model. The flexible registration module 350 performs flexible registration on the calibration three-dimensional coordinates of each key point in the preset model based on each of the three-dimensional coordinates, the rotation matrix and the translation vector, to obtain the target three-dimensional coordinates of each key point. The deformation estimation module 360 ​​is used to estimate the deformation of the target sampling spoon based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates; wherein the deformation includes at least length deformation, width deformation, height deformation and angle deformation.

[0152] Furthermore, when determining the normal vector corresponding to each pixel in the depth image, the image acquisition module 310 is used to: Gaussian filtering is applied to the depth image to remove noise from the depth image; Using the preset empty scene depth map of the rare earth electrolysis furnace, the noise-removed depth image is subjected to background subtraction to obtain the region image of the target sampling spoon in the depth image; The surface gradient method is used to calculate the normal vector corresponding to each pixel in the image of the region.

[0153] Furthermore, when the model processing module 320 inputs the RGB image and the normal vector map into a preset convolutional neural network model to obtain Gaussian heatmaps corresponding to multiple preset key points of the target sampling spoon output by the convolutional neural network model, the model processing module 320 is used to: The RGB image and the normal vector map are respectively input into a preset convolutional neural network model; The encoder set in the convolutional neural network model performs encoding operations on the RGB image and the normal vector map respectively to obtain encoded information; The decoder set in the convolutional neural network model performs bilinear upsampling and skip connections on the encoded information to obtain the calorific value information corresponding to multiple preset key points of the target sampling spoon. The convolutional neural network model performs Sigmoid probability activation on the heat value information corresponding to each key point, and outputs a Gaussian heatmap corresponding to each key point.

[0154] Furthermore, when the 3D projection module 330 performs 3D projection on each key point based on the Gaussian heatmap corresponding to each key point and the device parameters of the image acquisition device to obtain the 3D coordinates corresponding to each key point, the 3D projection module 330 is used to: Determine the response pixel coordinates of the key point in the Gaussian heatmap corresponding to each key point; Sub-pixel optimization is performed on the response pixel coordinates corresponding to each key point to obtain the sub-pixel coordinates corresponding to each key point; Based on the device parameters of the image acquisition device and the coordinates of each subpixel, the three-dimensional coordinates corresponding to each key point are determined using a preset projection formula, so as to perform three-dimensional projection on each key point.

[0155] Furthermore, when the rigid registration module 340 performs rigid coarse registration on each of the three-dimensional coordinates based on the preset model corresponding to the target sampling spoon, to obtain the rotation matrix and translation vector of each key point in the preset model corresponding to each key point, the rigid registration module 340 is used to: Determine the calibration three-dimensional coordinates of each key point in the preset model corresponding to the target sampling spoon; Based on the calibration 3D coordinates and the 3D coordinates corresponding to each key point, the rotation matrix and translation vector of each 3D coordinate relative to the corresponding calibration 3D coordinate are determined using point distance transformation and iterative nearest point solution method.

[0156] Furthermore, when the elastic registration module 350 is used to perform elastic registration on the calibrated 3D coordinates of each key point in the preset model based on each of the 3D coordinates, the rotation matrix, and the translation vector, to obtain the target 3D coordinates corresponding to each key point, the elastic registration module 350 is used to: Based on the rotation matrix and the translation vector, determine the registration 3D coordinates corresponding to each 3D coordinate; Based on the registration coordinates and calibration three-dimensional coordinates corresponding to each key point, the TPS elastic registration formula corresponding to the sampling spoon deformation function is used to determine the sampling spoon deformation function of each calibration three-dimensional coordinate relative to the corresponding registration three-dimensional coordinate. Based on the deformation function of the sampling spoon, the calibrated three-dimensional coordinates corresponding to each key point are subjected to bending deformation to obtain the target three-dimensional coordinates corresponding to each key point.

[0157] Furthermore, the key points include at least the center key point of the upper edge of the spoon rim and the center key point of the lower edge of the spoon rim; when the deformation estimation module 360 ​​estimates the deformation of the target sampling spoon based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates, the deformation estimation module 360 ​​is used to: Determine the first Manhattan distance between the first calibrated three-dimensional coordinates corresponding to the center key point of the upper edge of the spoon mouth and the second calibrated three-dimensional coordinates corresponding to the center key point of the lower edge of the spoon mouth, and determine the first Manhattan distance as the target length value corresponding to the target sampling spoon; Calculate the second Manhattan distance between the first target three-dimensional coordinates corresponding to the center key point of the upper edge of the spoon mouth and the second target three-dimensional coordinates corresponding to the center key point of the lower edge of the spoon mouth, and determine the second Manhattan distance as the deformation length value corresponding to the target sampling spoon; The difference between the deformation length value and the target length value is determined as the estimated length deformation of the target sampling spoon.

[0158] Furthermore, the key points include at least left edge key points and right edge key points; when the deformation estimation module 360 ​​estimates the deformation of the target sampling spoon based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates, the deformation estimation module 360 ​​is used to: The third Manhattan distance is determined between the third calibrated three-dimensional coordinates corresponding to the left edge key point and the fourth calibrated three-dimensional coordinates corresponding to the right edge key point, and this third Manhattan distance is determined as the target width value corresponding to the target sampling spoon; Calculate the fourth Manhattan distance between the three-dimensional coordinates of the third target corresponding to the key point on the left edge and the three-dimensional coordinates of the fourth target corresponding to the key point on the right edge, and determine the fourth Manhattan distance as the deformation width value corresponding to the target sampling spoon; The difference between the deformation width value and the target width value is determined as the estimated width deformation value corresponding to the target sampling spoon.

[0159] Furthermore, the key points include at least the center key point of the spoon bottom and at least three key points on the edge of the spoon rim; when the deformation estimation module 360 ​​estimates the deformation of the target sampling spoon based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates, the deformation estimation module 360 ​​is used to: Determine the first vertical distance between the fifth calibrated three-dimensional coordinates corresponding to the center key point of the spoon bottom and the plane fitted by the key points of the spoon edge in the preset model, and determine the first vertical distance as the target height value corresponding to the target sampling spoon; Determine the second vertical distance between the third-dimensional coordinates of the fifth target corresponding to the key point at the bottom of the spoon and the plane fitted by the multiple key points at the mouth of the spoon, and determine the second vertical distance as the deformation height value corresponding to the target sampling spoon; The difference between the deformation height value and the target height value is determined as the estimated height deformation of the target sampling spoon.

[0160] Furthermore, the key points include at least the target key point of the spoon handle, the key point of the spoon handle connection, and at least three key points on the edge of the spoon rim. When the deformation estimation module 360 ​​estimates the deformation of the target sampling spoon based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates, the deformation estimation module 360 ​​is used for: Based on the sixth calibration three-dimensional coordinates corresponding to the target key point of the spoon handle, the seventh calibration three-dimensional coordinates corresponding to the key point of the spoon handle connection, and the plane fitted by the key point of the spoon edge in the preset model, the target angle value corresponding to the target sampling spoon is determined by the preset angle formula. Determine the difference between the three-dimensional coordinates of the sixth target corresponding to the key point of the spoon handle and the three-dimensional coordinates of the seventh target corresponding to the key point of the spoon handle connection, and normalize the difference. The normalized difference is determined as the deformation vector of the spoon handle direction corresponding to the target sampling spoon. Based on the deformation vector of the handle and the plane fitted by the key points of the spoon mouth edge, the deformation angle value corresponding to the target sampling spoon is determined using a preset angle formula. The difference between the deformation angle value and the target angle value is determined as the estimated angular deformation value corresponding to the target sampling spoon.

[0161] The deformation estimation device for a sampling spoon in a rare earth electrolysis furnace provided in this application acquires RGB and depth images of the sampling spoon in the rare earth electrolysis furnace, and obtains a normal vector map including the normal vector corresponding to each pixel in the depth image. Using a convolutional neural network model, it outputs Gaussian heatmaps corresponding to multiple key points of the sampling spoon based on the RGB images and the normal vector map. Each key point is then 3D projected to obtain its corresponding 3D coordinates. After rigid and elastic registration of the 3D coordinates, the deformation of the target sampling spoon is estimated. This device adapts to the deformation estimation requirements under various complex working conditions, improving the accuracy and efficiency of deformation detection of the sampling spoon in the rare earth electrolysis furnace, and thus improving the real-time performance and stability of deformation detection of the sampling spoon.

[0162] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0163] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 The steps of the method for estimating the deformation of the sampling spoon of the rare earth electrolysis furnace in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0164] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the method for estimating the deformation of the sampling spoon of the rare earth electrolysis furnace in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0165] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0167] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0169] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0170] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for estimating the deformation of a sampling spoon in a rare earth electrolysis furnace, characterized in that, The deformation estimation method includes: The RGB image and depth image of the target sampling spoon in the rare earth electrolysis furnace, which are acquired by the image acquisition device, are obtained respectively, and the normal vector corresponding to each pixel in the depth image is determined to obtain the normal vector map corresponding to the depth image. The RGB image and the normal vector map are respectively input into a preset convolutional neural network model to obtain Gaussian heatmaps corresponding to multiple preset key points of the target sampling spoon, which are output by the convolutional neural network model. Based on the Gaussian heatmap corresponding to each key point and the device parameters of the image acquisition device, a three-dimensional projection is performed on each key point to obtain the three-dimensional coordinates corresponding to each key point; Based on the preset model corresponding to the target sampling spoon, rigid coarse registration is performed on each of the three-dimensional coordinates to determine the rotation matrix and translation vector of each key point in the preset model corresponding to each key point. Based on each of the three-dimensional coordinates, the rotation matrix and the translation vector, the calibration three-dimensional coordinates of each key point in the preset model are flexibly registered to obtain the target three-dimensional coordinates of each key point. Based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates, the deformation of the target sampling spoon is estimated; wherein the deformation includes at least length deformation, width deformation, height deformation and angle deformation.

2. The method according to claim 1, characterized in that, Determining the normal vector corresponding to each pixel in the depth image includes: Gaussian filtering is applied to the depth image to remove noise from the depth image; Using the preset empty scene depth map of the rare earth electrolysis furnace, the noise-removed depth image is subjected to background subtraction to obtain the region image of the target sampling spoon in the depth image; The surface gradient method is used to calculate the normal vector corresponding to each pixel in the image of the region.

3. The method according to claim 1, characterized in that, The step of inputting the RGB image and the normal vector map into a preset convolutional neural network model to obtain Gaussian heatmaps corresponding to multiple preset key points of the target sampling spoon, output by the convolutional neural network model, includes: The RGB image and the normal vector map are respectively input into a preset convolutional neural network model; The encoder set in the convolutional neural network model performs encoding operations on the RGB image and the normal vector map respectively to obtain encoded information; The decoder in the convolutional neural network model performs bilinear upsampling and skip connections on the encoded information to obtain the calorific value information corresponding to multiple preset key points of the target sampling spoon. The convolutional neural network model performs Sigmoid probability activation on the heat value information corresponding to each key point, and outputs a Gaussian heatmap corresponding to each key point.

4. The method according to claim 1, characterized in that, The step of performing a three-dimensional projection on each key point based on the Gaussian heatmap corresponding to each key point and the device parameters of the image acquisition device to obtain the three-dimensional coordinates corresponding to each key point includes: Determine the response pixel coordinates of the key point in the Gaussian heatmap corresponding to each key point; Sub-pixel optimization is performed on the response pixel coordinates corresponding to each key point to obtain the sub-pixel coordinates corresponding to each key point; Based on the device parameters of the image acquisition device and the coordinates of each subpixel, the three-dimensional coordinates corresponding to each key point are determined using a preset projection formula, so as to perform three-dimensional projection on each key point.

5. The method according to claim 1, characterized in that, The process of rigidly coarsely registering each of the three-dimensional coordinates based on the preset model corresponding to the target sampling spoon, to obtain the rotation matrix and translation vector of each key point in the preset model corresponding to each key point, includes: Determine the calibration three-dimensional coordinates corresponding to each key point in the preset model corresponding to the target sampling spoon; Based on the calibration 3D coordinates and the 3D coordinates corresponding to each key point, the rotation matrix and translation vector of each 3D coordinate relative to the corresponding calibration 3D coordinate are determined using point distance transformation and iterative nearest point solution method.

6. The method according to claim 1, characterized in that, The step of elastically registering the calibrated 3D coordinates of each key point in the preset model based on each of the 3D coordinates, the rotation matrix, and the translation vector to obtain the target 3D coordinates corresponding to each key point includes: Based on the rotation matrix and the translation vector, determine the registration 3D coordinates corresponding to each 3D coordinate; Based on the registration three-dimensional coordinates and calibration three-dimensional coordinates corresponding to each key point, the TPS elastic registration formula corresponding to the sampling spoon deformation function is used to determine the sampling spoon deformation function of each calibration three-dimensional coordinate relative to the corresponding registration three-dimensional coordinate. Based on the deformation function of the sampling spoon, the calibrated three-dimensional coordinates corresponding to each key point are subjected to bending deformation to obtain the target three-dimensional coordinates corresponding to each key point.

7. The method according to claim 1, characterized in that, The key points include at least the center key point of the upper edge of the spoon rim and the center key point of the lower edge of the spoon rim; estimating the deformation of the target sampling spoon based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates includes: Determine the first Manhattan distance between the first calibrated three-dimensional coordinates corresponding to the center key point of the upper edge of the spoon mouth and the second calibrated three-dimensional coordinates corresponding to the center key point of the lower edge of the spoon mouth, and determine the first Manhattan distance as the target length value corresponding to the target sampling spoon; Calculate the second Manhattan distance between the first target three-dimensional coordinates corresponding to the center key point of the upper edge of the spoon mouth and the second target three-dimensional coordinates corresponding to the center key point of the lower edge of the spoon mouth, and determine the second Manhattan distance as the deformation length value corresponding to the target sampling spoon; The difference between the deformation length value and the target length value is determined as the estimated length deformation of the target sampling spoon.

8. The method according to claim 1, characterized in that, The key points include at least left edge key points and right edge key points; estimating the deformation of the target sampling spoon based on the calibrated 3D coordinates and the target 3D coordinates includes: The third Manhattan distance is determined between the third calibrated three-dimensional coordinates corresponding to the left edge key point and the fourth calibrated three-dimensional coordinates corresponding to the right edge key point, and this third Manhattan distance is determined as the target width value corresponding to the target sampling spoon; Calculate the fourth Manhattan distance between the three-dimensional coordinates of the third target corresponding to the key point on the left edge and the three-dimensional coordinates of the fourth target corresponding to the key point on the right edge, and determine the fourth Manhattan distance as the deformation width value corresponding to the target sampling spoon; The difference between the deformation width value and the target width value is determined as the estimated width deformation value corresponding to the target sampling spoon.

9. The method according to claim 1, characterized in that, The key points include at least the center key point of the spoon bottom and at least three key points on the edge of the spoon rim; estimating the deformation of the target sampling spoon based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates includes: Determine the first vertical distance between the fifth calibrated three-dimensional coordinates corresponding to the center key point of the spoon bottom and the plane fitted by the key points of the spoon edge in the preset model, and determine the first vertical distance as the target height value corresponding to the target sampling spoon; Determine the second vertical distance between the third-dimensional coordinates of the fifth target corresponding to the key point at the center of the spoon bottom and the plane fitted based on multiple key points at the edge of the spoon mouth, and determine the second vertical distance as the deformation height value corresponding to the target sampling spoon; The difference between the deformation height value and the target height value is determined as the estimated height deformation of the target sampling spoon.

10. The method according to claim 1, characterized in that, The key points include at least the target key points of the spoon handle, the key points at the connection of the spoon handle, and at least three key points on the edge of the spoon rim. The estimation of the deformation corresponding to the target sampling spoon based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates includes: Based on the sixth calibration three-dimensional coordinates corresponding to the key points of the target spoon handle, the seventh calibration three-dimensional coordinates corresponding to the key points of the spoon handle connection, and the plane fitted by the key points of the spoon mouth edge in the preset model, the target angle value corresponding to the target sampling spoon is determined using a preset angle formula. Determine the difference between the three-dimensional coordinates of the sixth target corresponding to the key point of the spoon handle and the three-dimensional coordinates of the seventh target corresponding to the key point of the spoon handle connection, and normalize the difference. The normalized difference is determined as the spoon handle direction deformation vector corresponding to the target sampling spoon. Based on the deformation vector of the spoon handle and the plane fitted by the key points of the spoon mouth edge, the deformation angle value corresponding to the target sampling spoon is determined using a preset angle formula; The difference between the deformation angle value and the target angle value is determined as the estimated angular deformation value corresponding to the target sampling spoon.

11. A device for estimating the deformation of a sampling spoon in a rare earth electrolysis furnace, characterized in that, The deformation estimation device includes: The image acquisition module is used to acquire the RGB image and depth image of the target sampling spoon in the rare earth electrolysis furnace acquired by the image acquisition device, and to determine the normal vector corresponding to each pixel in the depth image, so as to obtain the normal vector map corresponding to the depth image. The model processing module is used to input the RGB image and the normal vector map into a preset convolutional neural network model respectively, and obtain the Gaussian heatmaps corresponding to multiple preset key points of the target sampling spoon output by the convolutional neural network model. The three-dimensional projection module is used to perform three-dimensional projection on each of the key points based on the Gaussian heatmap corresponding to each key point and the device parameters of the image acquisition device, so as to obtain the three-dimensional coordinates corresponding to each key point; The rigid registration module is used to perform rigid coarse registration on each of the three-dimensional coordinates based on the preset model corresponding to the target sampling spoon, and to determine the rotation matrix and translation vector of each key point corresponding to each key point in the preset model. The flexible registration module performs flexible registration on the calibrated 3D coordinates of each key point in the preset model based on each of the 3D coordinates, the rotation matrix and the translation vector, to obtain the target 3D coordinates of each key point. The deformation estimation module is used to estimate the deformation of the target sampling spoon based on the calibrated three-dimensional coordinates and the target three-dimensional coordinates; wherein the deformation includes at least length deformation, width deformation, height deformation and angle deformation.

Images