Wall surface positioning method and device based on multi-sensor fusion

Abstract

The invention discloses a wall positioning method and device based on multi-sensor fusion, and relates to the field of positioning. The method comprises the following steps: positioning by using a sensor group to obtain first position information; acquiring a wall image, identifying the wall image, and acquiring a wall material; performing positioning calibration on the first position information according to a wall surface material to obtain second position information; and acquiring the target position based on the second position information. According to the invention, positioning calibration is carried out by introducing the wall surface material, and the signal transmission interference of the wall surface material in the positioning process of the sensor group is optimized, so that more accurate space positioning is obtained.

Description

Technical Field

[0001] This application relates to the field of positioning technology, and particularly to a wall positioning method and device based on multi-sensor fusion. Background Art

[0002] In the current era of rapid technological development, the construction field is undergoing a profound transformation led by automation and intelligence. A significant manifestation is the large-scale use of robots to process walls. This innovative measure not only marks a major step forward in the construction industry towards modernization but also brings many technological innovations and breakthroughs. The existing technology already supports the application of robots in wall grinding and wall spraying scenarios.

[0003] For a robot to accurately reach the wall operation position in a complex construction environment, it depends on advanced positioning and navigation technologies. Multiple sensors such as lidar, vision sensors, and inertial measurement units work together to build an accurate environmental map for the robot. The lidar accurately perceives the distance and position of surrounding objects by emitting laser beams and measuring the time of the reflected light. The vision sensor is like the robot's eyes, capturing real-time image information of the construction site and identifying features such as the boundaries of the wall and the positions of doors and windows. The inertial measurement unit can measure the acceleration and angular velocity of the robot, assisting the positioning system in attitude solution and position tracking. The combination of these technologies enables the robot to autonomously plan a path at the construction site and accurately reach the wall operation point with the error controllable within a very small range.

[0004] However, in the wall painting scenario, the robot is required to have a higher positioning precision, and the traditional positioning technology still cannot meet this requirement. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a wall positioning method and device based on multi-sensor fusion with higher positioning accuracy.

[0006] In a first aspect, this application provides a wall positioning method based on multi-sensor fusion. The method includes:

[0007] Using a sensor group for positioning to obtain first position information;

[0008] Obtaining a wall image and performing recognition on the wall image to obtain the wall material;

[0009] Performing positioning calibration on the first position information according to the wall material to obtain second position information;

[0010] Obtaining a target position based on the second position information.

[0011] In one embodiment, using a sensor group for positioning to obtain first position information includes:

[0012] Obtain speed information, positioning information, acceleration, and angular velocity using a sensor group;

[0013] Fuse the speed information and positioning information using a first-level Kalman filter, and fuse the acceleration and angular velocity using a second-level Kalman filter connected in series with the first-level Kalman filter to obtain the first position information.

[0014] In one embodiment, positioning and calibrating the first position information according to the wall material to obtain the second position information includes:

[0015] Obtain the corresponding correction factor matrix according to the wall material;

[0016] Dynamically update the observation covariance of the first-level Kalman filter using the correction factor matrix;

[0017] At each update, update the first positioning information using the corresponding first-level Kalman filter after updating the observation covariance to obtain the second position information.

[0018] In one embodiment, the correction factor matrix is related to the specular reflection degree, scattering degree, and attenuation degree of the wall material.

[0019] In one embodiment, the updated observation covariance R k Is expressed as:

[0020]

[0021] Where Is the correction factor matrix, and R0 is the observation covariance before update.

[0022] In one embodiment, obtaining the target position based on the second position information includes:

[0023] Obtain the third position information using the time-of-arrival method;

[0024] Fuse the second position information and the third position information to obtain the target position.

[0025] In a second aspect, the present application also provides a wall positioning device based on multi-sensor fusion. The device includes:

[0026] A positioning module for positioning using a sensor group to obtain the first position information;

[0027] An identification module for obtaining a wall image and identifying the wall image to obtain the wall material;

[0028] A calibration module for positioning and calibrating the first position information according to the wall material to obtain the second position information;

[0029] An output module, configured to obtain a target position based on second position information.

[0030] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps in the above-mentioned wall positioning method based on multi-sensor fusion are implemented.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium. On the computer-readable storage medium, a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned wall positioning method based on multi-sensor fusion are implemented.

[0032] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned wall positioning method based on multi-sensor fusion are implemented.

[0033] The above-mentioned wall positioning method and device based on multi-sensor fusion use a sensor group for positioning to obtain first position information; obtain a wall image, and identify the wall image to obtain the wall material; perform positioning calibration on the first position information according to the wall material to obtain second position information; and obtain a target position based on the second position information. The present invention optimizes the signal transmission interference in the positioning process of the sensor group by introducing the wall material for positioning calibration, so as to obtain a more accurate spatial positioning. Description of the Drawings

[0034] Figure 1 It is a schematic flowchart of a wall positioning method based on multi-sensor fusion in an embodiment. Detailed Embodiments

[0035] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] The embodiment of the present application provides a wall positioning method based on multi-sensor fusion. The method is applied to wall working machine equipment, such as angle grinders, grinding machines, wall painting and spraying machines, etc., and acts on a target area by positioning the wall.

[0037] Compared with conventional ground or open - space positioning, there are many complex factors in the wall environment. For example, there is a wide variety of materials with significant differences. The differences in characteristics such as friction and flatness of walls made of different materials will cause inaccurate positioning of the positioning device, seriously affecting the accuracy of traditional positioning methods. At the same time, due to the particularity of the wall space, conventional positioning and calibration means such as RTK (Real - time kinematic) cannot be effectively used. To address this problem, as Figure 1 shown, the method proposed in this application includes the following steps:

[0038] Step 102: Use a sensor group for positioning to obtain the first position information.

[0039] The sensor group is mounted on the wall working machine equipment to perform real - time positioning on the current working area of the wall working machine. The sensor group includes, for example, Hall sensors, photoelectric sensors, acceleration sensors, angular velocity sensors, etc. Using the method of multi - sensor fusion, preliminary positioning is completed.

[0040] The first position information is relative position information. Different from the absolute position information which refers to the exact position in the earth coordinate system or other fixed coordinate systems, the relative position information refers to the relative position with respect to the starting point or the previous positioning point.

[0041] Step 104: Obtain the wall image and perform recognition on the wall image to obtain the wall material.

[0042] In this embodiment, the wall image is obtained by an image acquisition device such as a camera. Then, image cropping and pre - processing are performed, and the pre - trained neural network is input to recognize the wall image to obtain the wall material. Common wall materials include, for example, gypsum board, glass, ceramic tile, cement, etc.

[0043] The above - mentioned neural network uses a classification model, such as CNN, ResNet, etc., and performs model training to update the model parameters until the prediction accuracy reaches the preset requirements.

[0044] The input wall image is I k , and the recognition result is the wall material category c k ∈{c1, c2,..., c N}, and the classification model is defined as c k =argmax c∈C f(I k ; θ). Where f represents the trained classification model, θ represents the model parameters, and C represents the predefined set of wall materials.

[0045] In this embodiment, a neural network is used for wall material recognition, which can comprehensively analyze and learn various features of wall images, such as texture, color, shape, etc., so as to achieve a high recognition accuracy. At the same time, the neural network has strong robustness and can ensure the recognition accuracy in complex environments such as different lighting conditions and perspective changes. In addition, the neural network recognition does not require manual intervention and does not require manual judgment of the wall material, which greatly improves the positioning efficiency of the present invention and reduces the misjudgment rate.

[0046] Preferably, the device for collecting wall images can be a micro CMOS image sensor to achieve sub-millimeter-level image processing, which can more accurately predict the wall material and thus improve the subsequent positioning accuracy.

[0047] Step 106: Perform positioning calibration on the first position information according to the wall material to obtain the second position information.

[0048] Different wall materials will cause specular reflection, scattering, attenuation, etc. of the sensor signal on their surfaces. After identifying the wall material, this embodiment performs targeted optimization on the wall material during the signal transmission process to obtain more accurate second position information.

[0049] Step 108: Obtain the target position based on the second position information.

[0050] The calibrated second position information is still a relative position. If the absolute position in space is to be obtained, it can be determined by combining the absolute position of the starting point or the previous positioning point, or by combining the current absolute position of the machine device.

[0051] For the former, assuming that the absolute position coordinates of the starting point are (x0, y0), and the relative displacements in the x-axis and y-axis directions are Δx and Δy respectively, the current absolute position coordinates (x, y) can be calculated by the following formula:

[0052] x = x0 + Δx;

[0053] y = y0 + Δy.

[0054] For the latter, the absolute position information is used to correct the cumulative error in relative position positioning, and at the same time, the relative position information is used to provide continuous position estimation when the absolute position positioning signal is poor. By fusing these two pieces of information, the reliability and accuracy of positioning are improved.

[0055] In one embodiment, the time-of-arrival method is used to obtain the third position information; the second position information and the third position information are fused to obtain the target position.

[0056] Among them, the third position information is the absolute position information, which is obtained by the Time of Arrival (TOA) method. This is a positioning technology based on measuring the propagation time of signals from the emission source to multiple receiving points to determine the position of the emission source. Its basic principle is to utilize the propagation speed of signals in space (usually known, for example, the propagation speed of radio signals in the air is close to the speed of light), combine the time differences of the signals arriving at different receiving points, and determine the position of the emission source through geometric calculations.

[0057] Fuse the second position information and the third position information through a fusion algorithm such as Kalman filtering to obtain the target position of the machine device.

[0058] In one embodiment, positioning is performed using a sensor group, and obtaining the first position information includes: obtaining speed information, positioning information, acceleration, and angular velocity using the sensor group; fusing the speed information and the positioning information using a first-level Kalman filter, and fusing the acceleration and the angular velocity using a second-level Kalman filter connected in series with the first-level Kalman filter to obtain the first position information.

[0059] The Hall sensor provides speed information, the photoelectric sensor provides position information, the acceleration sensor provides acceleration, and the gyroscope provides angular velocity.

[0060] For the first-level Kalman filter, the state variables are set as:

[0061]

[0062] Among them, p k represents the position at time k, and v k represents the speed at time k.

[0063] The state transition model is:

[0064]

[0065] Among them, Δt is the sampling time interval, Q is the process noise covariance, and w k is the process noise.

[0066] In the observation model, the observation vector is:

[0067]

[0068] Among them, is the speed information, is the unknown information.

[0069] The observation matrix is:

[0070]

[0071] The observation model is:

[0072]

[0073] where, v k represents the observation noise, and R is the covariance of the observation noise.

[0074] The steps of the first-level Kalman filter are:

[0075] Predicted covariance:

[0076] P k|k-1 = AP k-1|k-1 A T + Q

[0077] Kalman gain:

[0078] K k = P k|k-1 H T (HP k|k-1 H T + R) -1

[0079] State update:

[0080] x k|k = x k|k-1 + K k (z k - Hx k|k-1 )

[0081] Covariance update:

[0082] P k|k = (I - K k H)P k|k-1

[0083] where, P k|k-1 is the state estimation error covariance matrix of the previous calculation period, P k|k is the state estimation error covariance matrix of this period, K k is the Kalman gain, and A is the state transition matrix.

[0084] For the second-level Kalman filter, the state vector is set as:

[0085]

[0086] where: Δx k represents the relative displacement (relative to the starting point), θ k represents the attitude angle, v k represents the linear velocity (provided by the first-level Kalman filter), and ω k represents the angular velocity.

[0087] State transition model (based on kinematic model):

[0088]

[0089] The observation model is:

[0090] The observation vector is the linear velocity a k and the angular velocity ω k :

[0091]

[0092] The steps of the second - stage Kalman filter are:

[0093] Predicted covariance:

[0094] P k|k-1 = AP k-1|k-1 A T + Q

[0095] Kalman gain:

[0096] K k = P k|k-1 H T (HP k|k-1 H T + R) -1

[0097] State update:

[0098] x k|k = x k|k-1 + K k (z k - Hx k|k-1 )

[0099] Covariance update:

[0100] P k|k = (I - K k H)P k|k-1

[0101] In this embodiment, two - stage Kalman filters are connected in series. The first - stage Kalman filter fuses velocity and position to provide a stable linear velocity estimate. The second - stage Kalman filter uses this linear velocity and combines attitude angle and gyroscope information to estimate the two - dimensional displacement (Δx, Δy) relative to the starting point, that is, the first position information.

[0102] In this embodiment, the robot's moving position information is collected by Hall sensors, and the collected position information is accurately calibrated by optoelectronic sensors. On this basis, the data of the acceleration sensor and the angular velocity sensor are fused to effectively reduce the sliding and vibration errors caused by uneven walls, and combined with the inertial navigation system to obtain the first position information of the machine device.

[0103] In one embodiment, positioning and calibrating the first position information according to the wall material to obtain the second position information includes: obtaining a corresponding correction factor matrix according to the wall material; dynamically updating the observation covariance of the first-order Kalman filter by using the correction factor matrix; and at each update, updating the first positioning information by using the corresponding first-order Kalman filter after the observation covariance is updated to obtain the second position information.

[0104] This embodiment defines a correction factor matrix for each type of material for correcting the observation covariance in the Kalman filter to dynamically adjust it according to the wall material.

[0105] The updated observation covariance R k is expressed as:

[0106]

[0107] where is the correction factor matrix, and R0 is the observation covariance before update (obtained by system calibration).

[0108] is a diagonal matrix, and the elements of m 1，1 are the velocity covariance correction factors, and the elements of m 2，2 are the covariance correction factors of the position, which are used to correct R0 with only the error values of the velocity sensor and the optoelectronic sensor in the past. Here, is obtained by training a neural network. Through supervised learning, using a high-performance computer, common wall materials are input in advance, and a neural network model is trained with the goal of "minimizing the real positioning error" to output the optimal correction factor matrix.

[0109] Specifically, first, collect data related to common materials and the corresponding positioning information. This data can come from actual experiments, sensor measurements, or simulation environments. For example, conduct positioning experiments using different wall materials and record each wall material type and the corresponding positioning results. At the same time, obtain the real positioning data as a reference for calculating the positioning error. Divide the collected data into a training set, a validation set, and a test set, usually in a ratio of 70%, 15%, and 15%. Use the training set to train a multi-layer perceptron MLP. Define a loss function, and the mean square error (MSE) can be used as the loss function to calculate the error between the predicted positioning result and the real positioning result, and train with the goal of "minimizing the real positioning error". After the training is completed, the optimal correction factor matrix can be output by using the neural network model.

[0110] In one embodiment, in some special occasions, such as when the wall material has high elasticity or low damping characteristics, or when there are large areas of voids or cavity structures on the wall, the wall material may cause resonance interference or noise amplification effects on the accelerometer data, and a secondary correction factor needs to be introduced to correct the secondary Kalman filter, and the method is similar to that of the primary one. At this time, the secondary correction factor matrix is also a diagonal matrix, and the element of m 1， 1 is the covariance correction factor of the linear velocity processed by the gyroscope, and m 2，2 is the covariance correction factor of the angular velocity obtained by the gyroscope.

[0111] For glass materials, it may increase the observation error in the horizontal reflection direction. For cement walls, the attenuation is low and the reflection is weak, so the influence on the observation is small, and the coefficient is close to 1.

[0112] In one embodiment, the correction factor matrix is related to the specular reflection degree, scattering degree and attenuation degree of the wall material.

[0113] Combining the material recognition module with the primary Kalman filter is specifically to update the observation error covariance in the filtering.

[0114] At each update, the Kalman filter can dynamically adjust the "degree of trust" in the observation data according to the wall material, and the corresponding Kalman gain is updated as follows:

[0115]

[0116] In one embodiment, the method uses a micro CMOS sensor to cut out the drawn image and align it with the undrawn image to obtain final sub-millimeter-level precision processing.

[0117] Specifically, align the target image with the drawn image, which requires determining a suitable coordinate system and alignment benchmark to ensure the consistency of the positions and orientations of the two images in space. The method based on feature point matching can be used to extract feature points (such as corner points, edge points, etc.) in the two images, and then minimize the distance between the feature points to achieve the alignment of the template. Through Boolean operations (such as subtraction), subtract the completed part from the target image to obtain the remaining undrawn part. Then, according to the graphics and process requirements of the undrawn part, starting from the target position, generate a path planning algorithm to guide the wall working machine equipment for subsequent drawing operations.

[0118] In one embodiment, sub-millimeter-level image processing technology is used for image tracking, cutting and alignment as the final positioning correction means. By performing sub-millimeter-level precision processing on the acquired image, the position of the robot on the wall can be carefully corrected, greatly improving the accuracy of positioning.

[0119] Through the template matching method in image processing means, the position of the image captured by a high-pixel camera is matched in the target image, so as to locate its position on the wall surface, and then calibrate the position information obtained by the sensor. This method can be used for calibration after position drift occurs after long-term operation, as the posterior estimation of Kalman.

[0120] The present invention introduces wall material detection. By accurately identifying the wall material, the wall surface positioning information is further calibrated. Walls of different materials will have different degrees of influence on positioning. The present invention can keenly detect the characteristics of the wall material, provide key data support for subsequent positioning correction, and achieve precise positioning during the wall painting process.

[0121] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same moment, but can be executed at different moments. The execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or alternately with at least a part of other steps or steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiments of the present application also provide a wall surface positioning device based on multi-sensor fusion for implementing the above-mentioned wall surface positioning method based on multi-sensor fusion. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the following wall surface positioning device based on multi-sensor fusion can refer to the limitations on the wall surface positioning method based on multi-sensor fusion in the above text, and will not be repeated here.

[0123] In one embodiment, a wall surface positioning device based on multi-sensor fusion is provided, including: a positioning module for using a sensor group to perform positioning and obtain first position information;

[0124] An identification module for obtaining a wall surface image and identifying the wall surface image to obtain the wall material;

[0125] A calibration module for calibrating the first position information according to the wall material to obtain second position information;

[0126] An output module for obtaining a target position based on the second position information.

[0127] Each module in the above wall positioning device based on multi-sensor fusion can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0128] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in all the above method embodiments are implemented.

[0129] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in all the above method embodiments are implemented.

[0130] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in all the above method embodiments are implemented.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards in relevant countries and regions.

[0132] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAM), magnetoresistive random access memories (MRAM), ferroelectric random access memories (FRAM), phase change memories (PCM), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0133] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0134] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A wall positioning method based on multi-sensor fusion, characterized in that, The method includes: Performing positioning using a sensor group to obtain first position information; Obtaining a wall image and performing recognition on the wall image to obtain the wall material; Performing positioning calibration on the first position information according to the wall material to obtain second position information; Obtaining a target position based on the second position information.

2. The method according to claim 1, characterized in that, The performing positioning using a sensor group to obtain first position information includes: Obtaining speed information, positioning information, acceleration, and angular velocity using the sensor group; Fusing the speed information and the positioning information using a first-level Kalman filter, and fusing the acceleration and the angular velocity using a second-level Kalman filter connected in series with the first-level Kalman filter to obtain the first position information.

3. The method according to claim 2, wherein The performing positioning calibration on the first position information according to the wall material to obtain second position information includes: Obtaining a corresponding correction factor matrix according to the wall material; Dynamically updating the observation covariance of the first-level Kalman filter using the correction factor matrix; At each update, updating the first positioning information using the first-level Kalman filter corresponding to the updated observation covariance to obtain the second position information.

4. The method according to claim 3, wherein The correction factor matrix is related to the specular reflection degree, scattering degree, and attenuation degree of the wall material.

5. The method according to claim 3, wherein The updated observation covariance R k is expressed as: Among them, is the correction factor matrix, and R0 is the observation covariance before update.

6. The method according to claim 1, wherein The obtaining a target position based on the second position information includes: Obtaining third position information using the time-of-arrival method; Fusing the second position information and the third position information to obtain the target position.

7. A wall positioning device based on multi-sensor fusion, characterized in that, The device includes: A positioning module for performing positioning using a sensor group to obtain first position information; An identification module for obtaining a wall image and performing recognition on the wall image to obtain the wall material; A calibration module for performing positioning calibration on the first position information according to the wall material to obtain second position information; An output module for obtaining a target position based on the second position information.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images