A remote positioning method combining AR camera with Internet of Things system

Abstract

The present application discloses a remote positioning method of an AR camera combined with an Internet of Things system, which relates to the field of Internet of Things device control, including: establishing a UWB network communication connection between an AR camera and a robot; when the robot is in a preset workspace, controlling the AR camera to move and collect image data in the preset workspace, and collecting motion data of the AR camera; using a deep learning algorithm to extract image semantic information of the image data; and fusing the extracted image semantic information with the collected motion data; constructing a feature map of the preset workspace; according to the constructed feature map of the preset workspace, obtaining the coordinate information of the robot in the preset workspace; the AR camera sends a control instruction to the robot through the UWB network based on the obtained coordinate information. In view of the problem of low remote control accuracy of robots in complex environments in the prior art, the present application improves the remote control accuracy through AR technology and deep learning algorithms.

Description

Technical Field

[0001] The present invention relates to the field of Internet of Things device control, and in particular to a remote positioning method for an AR camera combined with an Internet of Things system. Background Art

[0002] With the development of Industry 4.0 and intelligent manufacturing, the demand for the application of industrial robots in complex environments is growing. However, traditional robots mainly rely on autonomous navigation for positioning, which is difficult to achieve high-precision positioning due to large environmental changes and cannot meet the demand for precise control. In order to achieve accurate positioning of robots in complex environments, a new positioning and control solution is urgently needed.

[0003] The existing robot positioning methods mainly include laser radar positioning, visual positioning and inertial navigation combined positioning. However, laser radar is greatly affected by the site, visual positioning requires prior mapping and has drift problems, and inertial navigation combined positioning has large cumulative errors. These methods make it difficult for robots to obtain real-time and accurate positioning in complex and changing environments.

[0004] In the related art, for example, Chinese patent document CN111885707A provides an IoT device control method based on AR devices, including communicating with each other through a dynamic UWB module in an AR device and a static UWB module in an IoT device; moving the AR device along a preset trajectory in space, and calculating the distance parameter between the AR device and the IoT device through the UWB signal transmitted in the UWB network when moving the AR device; establishing a SLAM map through the AR device when moving the AR device; generating the coordinate information of the IoT device in the SLAM map through the distance parameter; and controlling the IoT device through the AR device based on the coordinate information. However, in this scheme, positioning relies on a single UWB distance measurement, which is easily affected by the environment and produces large errors. In complex environments, UWB signals are easily affected by occlusion and multipath effects, and the distance measurement error will increase. Therefore, the remote control accuracy of robots in complex environments needs to be further improved. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In response to the problem of low remote control accuracy of robots in complex environments in the prior art, the present invention provides a remote positioning method combining an AR camera with an Internet of Things system, which improves the remote control accuracy of robots in complex environments through AR technology and deep learning algorithms.

[0007] 2. Technical solution

[0008] The purpose of the present invention is achieved through the following technical solutions.

[0009] The embodiment of the present specification provides a remote positioning method of an AR camera combined with an Internet of Things system, which is used to control a robot, including: establishing a UWB network communication connection between the AR camera and the robot; when the robot is in a preset workspace, controlling the AR camera to move and collect image data in the preset workspace, and collecting motion data of the AR camera; wherein, a UWB network communication connection is established between the AR camera and the robot as a channel for the robot to receive control instructions; applying deep learning algorithms such as convolutional neural networks to extract semantic information of image data and construct image semantic features; matching and fusing the extracted image semantic features with the motion data of the AR camera to construct an initial feature map of the preset workspace; based on the constructed feature map, statistically calculating the coordinate information of the robot in the workspace; the AR camera sends the calculated robot coordinate information to the robot through the UWB network to realize remote positioning of the robot; based on the robot coordinates, the AR camera sends control instructions to the robot via the UWB network to drive the robot to perform specified work.

[0010] The image semantic information of the image data is extracted using a deep learning algorithm, and the extracted image semantic information is fused with the collected motion data to construct a feature map of the preset workspace; wherein, a deep learning model such as a convolutional neural network is applied as a semantic segmentation algorithm to process the input image data. The convolutional neural network includes an encoder and a decoder structure. The encoder extracts multi-level semantic features of the image through a convolution layer. The decoder is controlled by the feature map output by the encoder and calculates the correlation weights between different semantic features. According to the feature map and weight parameters, the decoder outputs the pixel-level semantic information of the image, that is, the image semantic segmentation information. The meta information of the AR camera image data is parsed to extract the timestamp sequence A of the image semantic information. The motion data collected by the AR camera is parsed to extract the timestamp sequence B of the motion data. A Seq2Seq model is established, which takes the timestamp sequences A and B as input and outputs a matching time index sequence. According to the time index sequence, the motion data and the corresponding image semantic information are matched. The matched image semantic information and motion data are used as nodes to construct the initial workspace feature map.

[0011] According to the feature map of the preset workspace, the coordinate information of the robot in the preset workspace is obtained; based on the obtained coordinate information, the AR camera sends a control instruction to the robot through the UWB network. After the feature map of the preset workspace is constructed, the coordinates of each feature point in the map are determined. The current image of the robot is analyzed to detect the feature points around the robot. The feature points around the robot are matched with the feature map, and the coordinates of the matching feature points are statistically calculated. According to the concentrated distribution of the coordinates of the matching feature points, the specific coordinate information of the robot in the map coordinate system is calculated. After the AR camera obtains the coordinate information of the robot, it plans the target path of the robot according to the motion control algorithm. The AR camera sends the coordinate information of the robot's target path point contained in the control instruction to the robot through the UWB network. After receiving the control instruction, the robot parses and obtains the coordinate point information of the target path. The robot moves to the target path point according to the control instruction to achieve precise positioning and navigation.

[0012] Furthermore, constructing a feature map of a preset workspace also includes the following steps: constructing a convolutional neural network as a semantic segmentation network, the semantic segmentation network includes an encoder and a decoder; the encoder performs a convolution operation on the input image data, extracts image features, and outputs an image feature map; the image feature map is input into the decoder, and the decoder calculates the weights between the features by constructing a quadratic kernel matrix; the decoder performs semantic segmentation on the image feature map according to the image feature map output by the encoder and the calculated weights, and generates image semantic information.

[0013] Among them, in this application, semantic information can clearly describe different object categories in the environment, such as aisles, doorways, shelves, etc. These key objects play an important role in robot navigation and positioning. Semantic information can be used to extract these key scene elements as feature points. The feature points constitute the initial map, which can greatly improve the positioning accuracy of the robot in complex environments. Semantic information can also be used for path planning, avoiding obstacles, and realizing intelligent navigation. Compared with traditional visual features, semantic information is more relevant to robot tasks and is more conducive to knowledge expression.

[0014] Furthermore, the ambient lighting information of the preset workspace is collected through an illumination sensor; the image semantic information is used as input, wherein the image semantic information includes the RGB value of each pixel; a Gaussian process regression model between the RGB value of the image semantic information and the ambient lighting information is established; the covariance matrix of the Gaussian process regression model is low-rank approximated through a singular value decomposition algorithm; the Gaussian process regression model after low-rank approximation is used, combined with the ambient lighting information, to predict the RGB value of each pixel in the image semantic information, and generate adjusted image semantic information.

[0015] Among them, in this application, the lighting conditions in the warehouse are complex and changeable, and the direct use of the original image semantics is easily affected by the lighting. The adjusted semantics can filter the impact of ambient light changes and enhance the robustness of the semantic information. It is conducive to extracting reliable and stable semantic information from images acquired at different time periods and locations. Make semantic segmentation and feature extraction more accurate and reduce errors caused by lighting changes. A high-precision feature map can be constructed based on the adjusted semantics. It is conducive to the continuous and accurate positioning of the robot under complex lighting conditions. Lighting adjustment improves the applicability of semantic information in complex environments. Enables the robot to work reliably in different lighting conditions such as indoors and outdoors.

[0016] Furthermore, generating adjusted image semantic information also includes: using a calibration plate to collect the RGB value of each pixel in the image data under different illumination values, and establishing a corresponding relationship between the illumination value and the RGB value; calculating the Gaussian process regression coefficient between the RGB value of each pixel and the corresponding illumination value; establishing a conditional Pixel CNN model based on deep learning to learn the illumination response prior knowledge of each pixel; using the Gaussian process regression coefficient and the illumination response prior knowledge as input, and predicting the RGB value of each pixel through the Gaussian process regression model; adjusting the RGB value of each pixel in the image semantic information according to the predicted RGB value, and generating image semantic information after adjusting the pixel.

[0017] Among them, a calibration plate is used to collect images under different lighting conditions, and the correspondence between the lighting value and the image RGB value is established. The Gaussian process regression coefficient between the RGB response of each pixel and the corresponding lighting value is calculated. A deep learning model based on PixelCNN is constructed to learn the prior knowledge of the pixel's response to lighting. Pixel CNN models the dependency between pixels through masked convolution and outputs the prior probability distribution of the lighting response. The Gaussian process regression coefficient and the prior knowledge learned by Pixel CNN are used as the input of the Gaussian process regression model. The Gaussian process regression model combines the two and outputs the Gaussian probability distribution of the RGB value of each pixel under the new lighting. Each pixel is sampled according to the RGB probability distribution to generate the RGB value after lighting adjustment. The adjusted RGB value is assigned to the corresponding pixel in the image semantic information to complete the adaptive lighting adjustment of the image. Finally, the image semantic information adjusted for the ambient lighting is output.

[0018] Furthermore, the extracted image semantic information is fused with the collected motion data, including: extracting the timestamp sequence A of the image semantic information by parsing the meta-information of the image data; extracting the timestamp sequence B of the motion data by parsing the timestamp of the motion data; establishing a Seq2Seq model based on LSTM, taking the timestamp sequence A and the timestamp sequence B as input, and outputting a matched time index sequence; matching the time series of the image semantic information and the motion data according to the time index sequence.

[0019] Among them, when the image is captured, the camera automatically writes the time metadata of the shooting in the image file. Read the image file, parse the image header information, and extract the shooting time of the image. For each image input into the semantic segmentation network, a unique shooting time can be parsed. The image semantic information generated by language semantic segmentation is bound to the corresponding source image time information. Organize the time information of all image semantic information in chronological order to form a timestamp sequence A. In the timestamp sequence A, each timestamp corresponds to an image semantic information. The timestamp sequence A reflects the order in which the image semantic information changes over time. The timestamp sequence A will be aligned with the motion time series B to match the semantic information and motion data.

[0020] Among them, the motion data comes from the inertial sensors on the robot, such as accelerometers, gyroscopes, etc. These sensors continuously collect the robot's motion data at a certain frequency. In the motion data acquisition module, the timestamp of the data is continuously recorded in conjunction with sensor sampling. The timestamp is generated using the system clock and is accurate to milliseconds. Each motion data sample is bound to its acquisition time timestamp. All motion data timestamps are organized in chronological order to form time series B. Each timestamp in time series B corresponds to a motion data sample. Time series B reflects the changes in the robot's motion data over time. Time series B is aligned with the image time series A to match motion and semantic information.

[0021] Furthermore, it also includes: using the matched time index sequence to obtain the image semantic information of the corresponding time index; inputting the obtained image semantic information of the corresponding time index into the IMM motion model to predict the motion position at the corresponding image acquisition time; pairing the predicted motion position with the image semantic information of the corresponding time index by establishing a matching table. Among them, the time index sequence I matched by the Seq2Seq model is input. According to each time index Ii, the corresponding image semantic information ai is taken out from the aligned image semantic sequence. The motion prediction model of the IMM (interactive multi-model) algorithm is constructed. The IMM model integrates multiple motion dynamics models and data-driven models. Input the time index Ii to the IMM model to predict the motion state and position pi at that moment. Using the matching time index Ii, a table is established to match the semantic information ai and the predicted position pi. Repeat the matching process to obtain the image semantic information and motion prediction corresponding to all time indexes. Finally, a matching table is obtained, in which each line is a time index, the corresponding semantic information and motion prediction. The matching table integrates the spatiotemporal information of image semantic understanding and motion prediction.

[0022] Furthermore, after predicting the motion position at the corresponding image acquisition moment, it also includes: obtaining the corresponding image semantic information paired with the predicted motion position through a matching table; using the obtained image semantic information and the corresponding predicted motion position as nodes, constructing an octree for nearest neighbor search; traversing the search octree, selecting the image features and motion positions with the smallest Euclidean distance between nodes for alignment and fusion; wherein a matching table has been constructed to store image semantic information ai and corresponding predicted motion position pi. Input the new image semantic information an to be matched and the motion position prediction pn. Search the matching table for the nearest neighbor ai of the semantic information an to obtain the paired position pi. Use an and pi to construct a new node and insert it into the octree. In the octree, the node feature is the image semantic information vector. Starting from the newly inserted node, traverse the octree for nearest neighbor search. Calculate the feature Euclidean distance between the new node and the existing node. Select the existing node with the smallest Euclidean distance to the new node. Take out the image semantics and corresponding motion position stored in the nearest neighbor node. Fusion of the nearest neighbor image semantics and motion position with an and pn of the new node. Finally, the image semantics and motion planning information matching the new data are obtained.

[0023] Furthermore, after selecting the image feature and motion position with the smallest Euclidean distance between nodes for alignment and fusion, it also includes: obtaining the pixel coordinates of the selected image feature as the feature point; constructing an extended Kalman filter EKF SLAM model according to the feature point and the corresponding motion position; setting the state variables of the extended Kalman filter EKF SLAM model to include the position of the robot and the position of the feature point; iteratively optimizing the position of the feature point according to the prediction update of the extended Kalman filter EKF SLAM model, and generating an initial feature map of the preset workspace.

[0024] Among them, the node with the minimum Euclidean distance of the image feature has been selected to obtain the image semantic features and the corresponding motion position. The pixel coordinates are extracted from the semantic features as feature points. The EKF SLAM model is constructed, and the state variables contain the robot position and the positions of all feature points. The motion model predicts the robot position, and the measurement model updates the feature point positions based on the feature observations. The observation matrix is ​​set based on the relative observation of the current feature point to other feature points. EKF is predicted and updated iteratively in chronological order to optimize the positions of all feature points. The matched semantic features and motion trajectories are used as the prior input of EKF. After iterating EKFSLAM, the initial feature map is obtained, which represents the distribution of environmental feature points. The map can be incrementally optimized later for robot autonomous navigation.

[0025] Furthermore, after generating the initial feature map, it also includes: constructing a recursive Bayesian filtering model, taking the initial feature map as the observation input; using each pixel coordinate in the initial feature map as the state variable of the recursive Bayesian filtering model; iteratively optimizing each state variable according to the prediction equation and update equation of the recursive Bayesian filtering model; and outputting the final feature map composed of the iteratively optimized state variables.

[0026] The input is the constructed initial feature map, which contains the location distribution of feature points. A recursive Bayesian filtering model is constructed, and the state variable is each feature point in the map. The prediction equation predicts the location distribution of feature points based on the existing map features. The update equation incorporates the actual feature observations and modifies the predicted feature point states. The initial map features are used as the first round of input for recursive filtering. The recursive prediction and update process is iterated to gradually optimize the state of each feature point. The optimization goal is to reduce the variance of the feature point position and improve the positioning accuracy. Repeat the iteration until the feature point state is stable enough. The feature point states obtained in the final iteration constitute the optimized final feature map. As a knowledge representation, the map can support the robot's autonomous tasks and decision-making.

[0027] Furthermore, the recursive Bayesian filtering model adopts a first-order Markov process. Among them, the first-order Markov process (First-order Markov process): a random process in which the future state depends only on the current state and has nothing to do with the past state. It has the property of no aftereffect. Specifically, the recursive Bayesian filtering assumes that the state transition has the Markov property. That is, the state of the feature point at the next moment depends only on the current state. In the prediction step, the transition probability is calculated based on the current feature point state. Generate the predicted distribution of the feature point position at the next moment. In the update step, the predicted distribution is corrected according to the latest observation. The prediction and update processes are recursively performed in sequence. The Markov assumption is used to simplify the recursive operation. Finally, the optimized feature point state is output to construct the map.

[0028] 3. Beneficial effects

[0029] Compared with the prior art, the advantages of the present invention are:

[0030] (1) By constructing a convolutional neural network for image semantic segmentation, the semantic information of the image is extracted, the expressiveness of the image information is improved, and a rich semantic feature description is provided for the subsequent construction of feature maps. This avoids the semantic ambiguity caused by the traditional method of directly using image pixel information;

[0031] (2) The initial feature map is optimized using a recursive Bayesian filtering model, each pixel coordinate is set as a state variable, and iterative optimization is performed based on the prediction and update equations to improve the accuracy of the feature map. This overcomes the shortcoming that ordinary filtering methods cannot effectively eliminate the accumulated errors of feature maps.

[0032] (3) With the help of the octree nearest neighbor search algorithm, the optimal matching of image semantic information and corresponding motion data is achieved, avoiding the large amount of calculation caused by global search, which solves the inefficient matching problem of image and motion data time series;

[0033] To summarize, this application extracts image semantic information through a deep learning algorithm, integrates the image semantic information with the robot motion data, constructs a workspace feature map, and then uses the feature map to locate the robot in real time, and sends control instructions through the Internet of Things network, thereby realizing accurate remote positioning and control of the robot in a complex environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0035] Figure 1 is an exemplary flow chart of a remote positioning method of an AR camera combined with an Internet of Things system according to some embodiments of this specification;

[0036] Figure 2 is a flowchart of exemplary sub-steps for generating image semantic information according to some embodiments of this specification;

[0037] Figure 3 is a flowchart of exemplary sub-steps for adjusting image semantic information according to some embodiments of this specification;

[0038] Figure 4 is a flowchart of exemplary sub-steps for fusing image data and motion data according to some embodiments of the present specification;

[0039] Figure 5 This is a flowchart of exemplary sub-steps for generating a feature map according to some embodiments of this specification. DETAILED DESCRIPTION

[0040] The method and system provided in the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0041] Figure 1 is an exemplary flow chart of a remote positioning method of an AR camera combined with an Internet of Things system according to some embodiments of this specification, such as Figure 1As shown, a remote positioning method of an AR camera combined with an Internet of Things system is used to control a robot, including: S100 establishing a UWB network communication connection between the AR camera and the robot; S200 when the robot is in a preset workspace, controlling the AR camera to move and collect image data in the preset workspace, and collecting motion data of the AR camera.

[0042] Specifically, AR camera refers to augmented reality camera (Augmented Reality Camera). It is a camera device that can comprehensively display virtual images and real environment images. AR camera captures real scenes through video, and then renders virtual content through image processing to achieve virtual and real combination and registration matching to form an augmented reality effect. In this application, the AR camera moves to collect image data in a preset workspace to obtain real information about the environment; the AR camera has its own sensor and can synchronously collect its own motion parameter data; the AR camera can be connected to the Internet of Things to form a communication network with the robot; the AR camera can obtain the coordinate information of the robot in the workspace based on the image recognition algorithm; the AR camera can send control instructions to the robot to achieve remote control.

[0043] More specifically, UWB network refers to Ultra Wide band Network. In this application, UWB network communication connection refers to ultra wide band network communication connection. It uses short-range wireless technology with wide occupied frequency band (greater than 500MHz or relative bandwidth of more than 20%) to achieve high-speed and low-power wireless communication. In this application, a UWB communication network is established between the AR camera and the robot; the UWB network has the advantages of strong anti-interference ability and high point-to-point communication rate; the AR camera sends control information such as robot coordinates to the robot through the UWB network; after the robot receives the control command of the AR camera, it feedbacks the execution status through the UWB network; the UWB network realizes high-speed, low-latency and highly reliable wireless communication between the AR camera and the robot. The use of UWB network for remote communication control solves the problem of reliable transmission of robot remote control commands.

[0044] Specifically, image data refers to the picture or video image information in the preset workspace captured by the AR camera. It records the visual features in the workspace. When the robot is working, the AR camera is used to collect image data in the workspace. The image data is semantically segmented to extract the semantic information of the image. The image semantic information is combined with the motion data to construct a feature map. Specifically, motion data refers to the motion parameter information of the AR camera itself when collecting images, including position, speed, acceleration and other data. It records the motion trajectory of the AR camera. When the AR camera collects image information in the workspace, it simultaneously collects its own motion data. Match the motion data with the corresponding image semantic information. The motion data provides the motion position information of the AR camera itself for constructing the feature map.

[0045] More specifically, in a smart warehouse to which the present application is applied, the image data may include: images of the shelf area of ​​the smart warehouse, including visual information such as shelf numbers and stored goods. Image information of handling robots such as stackers and automatic guided vehicles. Image information of containers, pallets, boxes, etc. for storing goods. Image information such as lighting conditions and identification symbols in the storage environment. Image information of warehouse employees and their operating environment. Motion data may include: the trajectory and path data of the handling robot. Position coordinate change data of the handling robot. Movement speed and acceleration data of the handling robot. Data on the position and grasping status of the handling robot's mechanical arm. Data on the system operating status, alarm signs, etc. of the handling robot. Motion control data such as the start and end time and execution status of the handling task.

[0046] S300 uses a deep learning algorithm to extract image semantic information from image data and fuses the extracted image semantic information; S400 fuses the extracted image semantic information with the collected motion data; S500 constructs a feature map of a preset workspace; wherein, a deep learning model such as a convolutional neural network is applied as a semantic segmentation algorithm to process the input image data. The convolutional neural network includes an encoder and a decoder structure. The encoder extracts multi-level semantic features of the image through a convolution layer. The decoder is controlled by the feature map output by the encoder and calculates the correlation weights between different semantic features. According to the feature map and weight parameters, the decoder outputs pixel-level semantic information of the image, i.e., image semantic segmentation information. The meta information of the AR camera image data is parsed to extract the timestamp sequence A of the image semantic information. The motion data collected by the AR camera is parsed to extract the timestamp sequence B of the motion data. A Seq2Seq model is established, with timestamp sequences A and B as input, and a matching time index sequence is output. According to the time index sequence, the motion data and the corresponding image semantic information are matched. The matched image semantic information and motion data are used as nodes to construct the initial workspace feature map.

[0047] S600 obtains the coordinate information of the robot in the preset workspace based on the constructed feature map of the preset workspace; the AR camera sends control instructions to the robot through the UWB network based on the obtained coordinate information. After the feature map of the preset workspace is constructed, the coordinates of each feature point in the map are determined. The current image of the robot is analyzed to detect the feature points around the robot. The feature points around the robot are matched with the feature map, and the coordinates of the matching feature points are statistically calculated. According to the concentrated distribution of the coordinates of the matching feature points, the specific coordinate information of the robot in the map coordinate system is calculated. After the AR camera obtains the coordinate information of the robot, it plans the target path of the robot according to the motion control algorithm. The AR camera sends the coordinate information of the robot's target path point contained in the control instruction to the robot through the UWB network. After receiving the control instruction, the robot parses and obtains the coordinate point information of the target path. The robot moves to the target path point according to the control instruction to achieve precise positioning and navigation.

[0048] Specifically, feature map refers to a digital map containing environmental spatial feature information, which is based on the extracted image visual features and motion position data to represent the geometric structure and semantic knowledge of the environment. In this application, a deep learning algorithm is used to extract the semantic information of image data. The image semantic information is matched with the motion data collected by the AR camera. The matched image semantic information and motion data are integrated to construct a feature map of the preset workspace. The feature map fully integrates environmental visual information and autonomous mobile position information. According to the constructed feature map, the coordinate information of the robot in the workspace is calculated. The robot can perform precise autonomous positioning and navigation according to the feature map.

[0049] More specifically, in a specific embodiment of the present application, in a smart warehouse, a map containing the coordinates of various feature points, such as entrances, aisles, corners, shipping areas, etc., has been constructed. The robot camera captures the current scene and detects the surrounding feature points through semantic segmentation. The detected feature description is matched and compared with that in the map database. The coordinate distribution of the feature matching points is statistically analyzed to determine the coordinate information of the robot relative to the map. The AR system plans the target coordinates of the robot's next waypoint based on the current positioning results. The AR system sends control instructions through the UWB network, which include the coordinate data of the target point. The robot UWB module receives the instructions and parses the navigation coordinate information. The robot motion control system drives the robot to translate and turn to reach the navigation point. The entire path is navigated in sequence to realize the automatic handling of the robot in the warehouse environment. The positioning is continuously updated through closed-loop feedback to achieve continuous and accurate navigation.

[0050] Figure 2 is a flowchart of exemplary sub-steps for generating image semantic information according to some embodiments of this specification, such as Figure 2As shown, S310 constructs a convolutional neural network as a semantic segmentation network, and the semantic segmentation network includes an encoder and a decoder; S320 encoder performs convolution operation on the input image data, extracts image features, and outputs an image feature map; S330 inputs the image feature map into the decoder, and the decoder calculates the weights between the features by constructing a quadratic kernel matrix; S340 decoder performs semantic segmentation on the image feature map according to the image feature map output by the encoder and the calculated weights, and generates image semantic information.

[0051] Specifically, the quadratic kernel matrix (Quadratic Kernel Matrix): It is a matrix used in the decoder to calculate the correlation weights between different feature maps. Each element in the kernel matrix represents the similarity between two features in the feature map. In the present application, the decoder receives the image feature map from the encoder. Construct a quadratic kernel matrix K, in which the element Kij in the matrix represents the similarity between feature i and feature j in the feature map. Apply the Gaussian kernel function to calculate the feature similarity and obtain the quadratic kernel matrix. The feature map is multiplied into a new feature space, and the new features are mutually orthogonal. According to the quadratic kernel matrix K, the weight of each feature in the feature map can be obtained. The decoder uses the feature weights to perform semantic segmentation on the feature map. Output an image containing semantic information for constructing a feature map. The quadratic kernel matrix calculates the correlation weights between the features in the feature map, providing a basis for subsequent semantic segmentation.

[0052] Specifically, image semantic information refers to the knowledge expression generated after semantic understanding of the image content, which gives semantic descriptions and meanings to elements such as objects and scenes in the image. In this application, the encoder outputs a feature map of the image. The decoder performs semantic segmentation based on the feature map and the calculated feature weights. Semantic segmentation classifies image pixels into different semantic concepts, such as people, cars, roads, etc. Semantic understanding is performed on each image element to obtain image semantic information. Image semantic information characterizes the semantic attributes of environmental targets, such as human posture, car type, etc. Image semantic information will be combined with motion information to construct a semantically rich feature map. Help robots better analyze and understand complex environments.

[0053] Specifically, in a specific embodiment of the present application, a convolutional neural network is constructed as a semantic segmentation model, including an encoder and a decoder. The encoder receives a warehouse scene image as input and uses different convolution kernels to extract multi-scale semantic features such as shelves, robots, and goods. The decoder includes an upsampling layer to restore the resolution of the encoder output feature map. The decoder constructs a quadratic Gaussian kernel matrix to calculate the weights between features such as shelf numbers and cargo categories. The convolution layer filters irrelevant features based on feature weights and outputs a semantic segmentation image of the warehouse scene. In the segmented image, different semantics are classified into different pixel values, such as shelves, ground, cargo, etc. Semantic concepts are generated for each type of target to obtain the semantic information of the scene. The semantic information identifies the semantic attributes of the warehouse target and performs semantic positioning and navigation on the handling robot. Parameters are optimized through model training to improve the semantic segmentation and understanding capabilities in complex warehouse scenes. Finally, a warehouse scene image containing rich semantic information is output.

[0054] S350 obtains ambient lighting information and adjusts the RGB values ​​in the semantic information according to the ambient lighting information. The RGB values ​​of images taken under different lighting conditions will vary greatly. Directly applying the original RGB semantic information will affect the effect of subsequent processing due to lighting changes. Adjusting the RGB values ​​according to the lighting information can eliminate the impact of lighting changes. This makes the semantic information extracted under different lighting more unified and reliable. It is conducive to building a stable and robust semantic map. It improves the applicability of semantic information in complex lighting environments. It makes semantic segmentation and feature extraction more accurate and reduces lighting errors. Ultimately, it can improve the results of tasks based on semantic information, such as positioning and navigation. Overall, it is conducive to enhancing the robustness of the solution and expanding its scope of application.

[0055] Figure 3 is an exemplary sub-step flowchart of adjusting image semantic information according to some embodiments of this specification, such as Figure 3 As shown, S351 collects the ambient lighting information of the preset workspace through the illumination sensor; S352 takes the image semantic information as input, wherein the image semantic information includes the RGB value of each pixel; S353 establishes a Gaussian process regression model between the RGB value of the image semantic information and the ambient lighting information; S354 performs low-rank approximation on the covariance matrix of the Gaussian process regression model through the singular value decomposition algorithm; S355 uses the Gaussian process regression model after low-rank approximation, combined with the ambient lighting information, to predict the RGB value of each pixel in the image semantic information, and generate adjusted image semantic information.

[0056] Specifically, the Gaussian Process Regression Model is a non-parametric regression model based on the Bayesian framework. It uses Gaussian processes to describe the prior distribution of functions and performs probabilistic reasoning based on data. In this application, a joint Gaussian process model between image semantic information RGB values ​​and ambient lighting information is constructed. The model assumes that the target function conforms to the Gaussian process and gives the prior function distribution according to the Gaussian distribution. Bayesian inference is performed on the posterior distribution based on the observed data obtained. The covariance matrix rank is reduced using singular value decomposition to reduce computational complexity. Given a new lighting input, the model outputs a Gaussian probability distribution of the image semantic information RGB value. According to the output probability distribution, the RGB value of each pixel in the image semantic information is predicted. Generate image semantic information adjusted for lighting. In summary, the Gaussian process regression model uses Bayesian ideas to model, perform adaptive lighting adjustment, and improve the robustness of scene understanding.

[0057] Specifically, singular value decomposition (SVD) algorithm: a matrix decomposition method that can decompose a matrix into the product form of several singular values ​​and corresponding left and right singular vectors. In the present application, the covariance matrix of the Gaussian process regression model is subjected to singular value decomposition. The main singular values ​​are selected by the size of the singular values, and the smaller singular values ​​are discarded. Only the main singular values ​​and the corresponding left and right singular vectors are retained, and the original covariance matrix is ​​approximated with a low-rank matrix. Low-rank approximation reduces the dimension and computational complexity of the covariance matrix. Based on the covariance matrix of the low-rank approximation, the probability reasoning of the Gaussian process regression model is performed. The computational efficiency of the model in predicting image semantic information is improved. In summary, the singular value decomposition algorithm reduces the computational complexity of the Gaussian process regression model by reducing the dimensionality through low-rank approximation.

[0058] Specifically, in a specific embodiment of the present application, a low-rank approximation Gaussian process regression model is constructed, the model input is the ambient lighting information, and the output is the RGB value of the image semantic information. Given a set of newly input ambient lighting information variables X*. Calculate the Gaussian process kernel ratio k between X and the model observation data. Multiply k* by the covariance matrix of the low-rank approximation to obtain the predicted covariance matrix. Calculate the Gaussian process kernel ratio k of X* and the output of the model training to obtain the Gaussian distribution of the regression value. For each image pixel, according to the Gaussian distribution of the regression value, sample and generate a Gaussian probability prediction of the RGB value. Sample all image pixels to obtain the RGB value adjusted for the semantic information of the entire image. Generate image semantic information adjusted for the input lighting. Continuously accumulate samples to update the model online to improve prediction accuracy.

[0059] More specifically, Low Rank Approximation: a matrix decomposition technique that approximates a high-dimensional matrix as a product of low-rank matrices. By reducing the rank of the matrix, the amount of storage and computation can be reduced. In a specific embodiment of the present application, a matrix representation is performed on the image generated by semantic segmentation. A low-rank approximation technique is used to reduce the dimension of the image matrix. Using methods such as SVD, it is decomposed into the product of two low-rank matrices. The low-rank representation can reduce the amount of computation for subsequent processing. The ambient lighting information is also expressed in a low-rank manner. In the low-rank space, Gaussian process regression is applied to predict the pixel RGB value. Finally, the adjusted high-resolution image is reconstructed. The process of using lighting information to adjust the image color is accelerated.

[0060] Furthermore, a calibration plate is used to collect the RGB value of each pixel in the image data under different illumination values, and the corresponding relationship between the illumination value and the RGB value is established; the Gaussian process regression coefficient between the RGB value of each pixel and the corresponding illumination value is calculated; a conditional Pixel CNN model based on deep learning is established to learn the illumination response prior knowledge of each pixel; the Gaussian process regression coefficient and the illumination response prior knowledge are used as input, and the RGB value of each pixel is predicted through the Gaussian process regression model; according to the predicted RGB value, the RGB value of each pixel in the image semantic information is adjusted to generate the image semantic information after the adjusted pixels.

[0061] Among them, a calibration plate is used to collect images under different lighting conditions, and the correspondence between the lighting value and the image RGB value is established. The Gaussian process regression coefficient between the RGB response of each pixel and the corresponding lighting value is calculated. A deep learning model based on PixelCNN is constructed to learn the prior knowledge of the pixel's response to lighting. Pixel CNN models the dependency between pixels through masked convolution and outputs the prior probability distribution of the lighting response. The Gaussian process regression coefficient and the prior knowledge learned by Pixel CNN are used as the input of the Gaussian process regression model. The Gaussian process regression model combines the two and outputs the Gaussian probability distribution of the RGB value of each pixel under the new lighting. Each pixel is sampled according to the RGB probability distribution to generate the RGB value after lighting adjustment. The adjusted RGB value is assigned to the corresponding pixel in the image semantic information to complete the adaptive lighting adjustment of the image. Finally, the image semantic information adjusted for the ambient lighting is output.

[0062] Specifically, Calibration Board: a standard test object used to calibrate the parameters of camera equipment, with feature patterns of known size and position printed on it. The calibration board is printed with standard color blocks and grayscale patterns. The camera captures images of the calibration board under different lighting conditions. The mapping relationship between the illumination value and the RGB value of the image pixel is established through the color blocks on the calibration board. The grayscale pattern is used to calibrate the intrinsic parameters and distortion parameters of the camera. The mapping relationship between the illumination value and the RGB value is used to build an illumination adaptive model. The intrinsic parameters and distortion parameters of the camera are helpful for image processing and analysis. Ultimately, it helps to achieve accurate illumination adaptive adjustment of the image semantic information. The calibration board provides accurate illumination-RGB mapping samples for establishing an illumination response model.

[0063] Specifically, in the present application, a Gaussian process regression model is constructed, with the input being the illumination value and the output being the pixel RGB response. Assume that the Gaussian process function f obeys a zero-mean Gaussian distribution, expressing the mapping relationship between pixels and illumination. Calculate the kernel function K of the Gaussian process to represent the similarity between the inputs. The RBF kernel can be used here. Calculate the kernel matrix K based on the existing illumination-RGB corresponding sample data. Use the relationship between the kernel matrix K and the response variable to solve the Gaussian process regression equation. The correlation coefficient β between the response variable and the kernel matrix is ​​calculated during the solution process. This correlation coefficient is the regression coefficient in the Gaussian process regression. Calculate the regression coefficient β between the illumination value and its RGB response for each pixel. Finally, the regression coefficient β describing the illumination response of the entire image is obtained.

[0064] Specifically, in the present application, the Pixel CNN model, an image generation model based on a convolutional neural network, explicitly models the conditional dependencies between pixels through masked convolution. Pixel CNN contains a masked convolution layer, and each pixel depends only on the previously generated pixels on the upper and lower left sides. The model inputs the lighting conditions and outputs the RGB response probability distribution of each pixel. The prior knowledge of the lighting response in the pixel neighborhood is learned through the Localized convolution kernel. The dependency of pixels at each position on lighting changes is captured. The output convolution feature map represents the spatial prior probability of pixel-level lighting response. As the prior knowledge of the Gaussian process regression model, the accuracy of lighting adaptation is improved. It is finally used to predict the probability distribution of each pixel of the image under new lighting. In summary, Pixel CNN learns the lighting response prior by modeling pixel dependencies, providing an effective prior for Gaussian process regression.

[0065] More specifically, in a specific embodiment of the present application, a Gaussian process regression model is constructed, and the input is the regression coefficient of the pixel and the prior knowledge of the lighting response obtained by Pixel CNN learning. The model outputs the predicted probability distribution of the RGB value of each pixel under the new lighting conditions. For each pixel position of the warehouse scene image, Gaussian process regression is performed in turn to obtain the RGB prediction distribution. According to the RGB prediction distribution, each pixel position is sampled to generate a new RGB value. The RGB value generated by the sample is updated to the corresponding pixel position in the semantic information of the original image. Repeat the update process for all pixel positions to obtain the new RGB value after the full image is adjusted. Finally, the image semantic information adjusted for the lighting changes in the warehouse is obtained. This image semantic information can more accurately express the semantic information of the warehouse target under the new lighting. Provide support for subsequent precise positioning and navigation of the robot.

[0066] Figure 4 is an exemplary sub-step flow chart of the fusion of image data and motion data according to some embodiments of this specification, such as Figure 4 As shown, S410 extracts the timestamp sequence A of the image semantic information by parsing the metadata of the image data; S420 extracts the timestamp sequence B of the motion data by parsing the timestamp of the motion data; S430 establishes a Seq2Seq model based on LSTM, takes the timestamp sequence A and the timestamp sequence B as input, and outputs a matched time index sequence; S440 matches the time series of the image semantic information and the motion data according to the time index sequence.

[0067] Among them, when the image is captured, the camera automatically writes the time metadata of the shooting in the image file. Read the image file, parse the image header information, and extract the shooting time of the image. For each image input into the semantic segmentation network, a unique shooting time can be parsed. The image semantic information generated by language semantic segmentation is bound to the corresponding source image time information. Organize the time information of all image semantic information in chronological order to form a timestamp sequence A. In the timestamp sequence A, each timestamp corresponds to an image semantic information. The timestamp sequence A reflects the order in which the image semantic information changes over time. The timestamp sequence A will be aligned with the motion time series B to match the semantic information and motion data.

[0068] Among them, the motion data comes from the inertial sensors on the robot, such as accelerometers, gyroscopes, etc. These sensors continuously collect the robot's motion data at a certain frequency. In the motion data acquisition module, the timestamp of the data is continuously recorded in conjunction with sensor sampling. The timestamp is generated using the system clock and is accurate to milliseconds. Each motion data sample is bound to its acquisition time timestamp. All motion data timestamps are organized in chronological order to form time series B. Each timestamp in time series B corresponds to a motion data sample. Time series B reflects the changes in the robot's motion data over time. Time series B is aligned with the image time series A to match motion and semantic information.

[0069] Specifically, LSTM (Long Short Term Memory): a recurrent neural network structure that can effectively learn time-related features in long sequences and store historical states. Seq2Seq (Sequence to Sequence), an end-to-end encoder and decoder structure, maps fixed-length sequences to fixed-length sequences and can be used for time series matching. In this application, the encoder uses an LSTM network and inputs timestamp sequences A and B. LSTM extracts the temporal features of the time series and learns temporal association information. The decoder uses an LSTM network to output a time-matched index sequence. Seq2Seq maps two time series to the same matched time index end-to-end. Using the storage of LSTM historical states, two asynchronous time series are effectively matched. The output time index sequence matches the temporal correspondence between image semantics and motion data. It is ultimately used to temporally align image semantic information and motion data.

[0070] Specifically, a Seq2Seq model is constructed, with timestamp sequences A and B as input and matching time index sequences as output. A two-layer LSTM is used on the encoder side, and feature sequences of timestamps A and B are input respectively. LSTM learns the temporal dependency of timestamp sequences and outputs temporal features. The temporal features of the two LSTMs are connected into a joint feature at each time step as the input of the decoder. The decoder side also uses the LSTM structure to output the time index step by step according to the joint time features. The output index is consistent with the length of the input time series, indicating the matching time correspondence. The training model adjusts the LSTM weights by matching paired timestamp samples. The final model learns the matching function and can directly output the matching index of the unpaired timestamp. The index points to the matching time position in the input time series.

[0071] Specifically, in the present application, a time-aligned image semantic information sequence A and a motion data sequence B are input. A generated matching time index sequence I is input. According to the time index Ii, the image semantic information ai corresponding to the timestamp is taken out in sequence A. At the same time, the motion data bi corresponding to the index Ii is taken out in sequence B. The matched image semantic information ai and motion data bi are combined as the i-th matching time series sample. The process is repeated to match all data in sequences A and B through all elements in the time index I. Finally, time series data of fully aligned image semantic information and motion data is obtained. The matched time series reflects the visual information corresponding to the robot's motion process. It can be used to construct a data set describing the environment and motion for subsequent modeling.

[0072] S450 uses the matched time index sequence to obtain the image semantic information of the corresponding time index; inputs the obtained image semantic information of the corresponding time index into the IMM motion model to predict the motion position at the corresponding image acquisition moment; and pairs the predicted motion position with the image semantic information of the corresponding time index by establishing a matching table. Among them, the time index sequence I matched by the Seq2Seq model is input. According to each time index Ii, the corresponding image semantic information ai is taken out from the aligned image semantic sequence. The motion prediction model of the IMM (interactive multi-model) algorithm is constructed. The IMM model integrates multiple motion dynamics models and data-driven models. Input the time index Ii into the IMM model to predict the motion state and position pi at that moment. Using the matching time index Ii, a table is established to match the semantic information ai and the predicted position pi. Repeat the matching process to obtain the image semantic information and motion prediction corresponding to all time indexes. Finally, a matching table is obtained, in which each line is a time index, the corresponding semantic information and motion prediction. The matching table integrates the spatiotemporal information of image semantic understanding and motion prediction.

[0073] Specifically, the IMM (Interacting Multiple Model) motion model, a probabilistic data-driven modeling method, performs motion prediction by integrating and interactively adjusting multiple sub-models. In the present application, IMM integrates multiple sub-models such as kinematic models and particle filter models. The kinematic model performs deterministic prediction based on the robot dynamics equation. The particle filter model performs Bayesian prediction under nonlinear non-Gaussian distribution. The model interaction module fuses the model output according to the model probability weight. The IMM model is driven by the time index input to predict the motion state. The output prediction results include the position, speed, etc. at the corresponding moment. Match the image semantic information and motion prediction output corresponding to the time index. The IMM model integrates deterministic and probabilistic methods to improve the robustness of motion prediction.

[0074] Specifically, what is obtained in data collection is the motion data of the robot, including sensor information such as acceleration and angular velocity. The motion position in the IMM model is predicted by integrating acceleration information and other methods, and is the coordinate offset information relative to the starting position. Therefore, the two are different but related concepts. The motion data is the original sensor information, and the motion position is the position estimate obtained by processing and integrating it. The motion position prediction in the IMM model needs to rely on the motion data information obtained in data collection. The motion data provides the input of the kinematic model required to predict the motion position. The predicted motion position and the matching image semantic information are used as nodes to construct an octree for subsequent fusion processing. In general, motion data is the input for motion position prediction, and the two are related through data processing.

[0075] Specifically, in the present application, a matching table can be established by: storing the matching table in CSV format, with each row containing a time index, image semantics, and motion position. The CSV matching table can be read and written using the CSV module in Python. Create a matching table in a database such as SQLite and insert data into the table. The SQLite matching table can be operated using the sqlite3 module in Python. The matching table is stored in a NumPy array, with each row being a matching item. The array can be indexed and sliced, and other operations can be performed. The matching table is represented by a Data Frame in Pandas, with a time index for easy query. The matching table is stored in a dictionary, with the key being the time index and the value being a tuple containing semantics and position. The matching table is stored in JSON format for easy network transmission and cross-language use.

[0076] More specifically, in a specific embodiment of the present application, a time index sequence I, an image semantic sequence A, and a motion prediction position sequence P are input. An empty matching data table is created, including 3 columns: time index, image semantics, and motion prediction position. Traverse the time index I, and take each time index Ii: find the image semantics ai corresponding to Ii in sequence A, and find the corresponding position pi in sequence P. Add Ii, ai, and pi as a row to the matching table. Repeat this process until all Ii are traversed. Finally, each row is a time index, corresponding to the image semantics and motion position at that moment. Save the matching table as a CSV data file or a database table. You can also use a Python dictionary to generate a matching table directly, with the key being the time index and the value being (image semantics, motion position). You can then quickly query the matching table based on the time index to obtain the corresponding data.

[0077] After predicting the motion position at the corresponding image acquisition moment, S460 also includes: obtaining the corresponding image semantic information paired with the predicted motion position through the matching table; using the obtained image semantic information and the corresponding predicted motion position as nodes, constructing an octree for nearest neighbor search; traversing the search octree, selecting the image features and motion positions with the smallest Euclidean distance between nodes for alignment and fusion; wherein a matching table has been constructed to store image semantic information ai and corresponding predicted motion position pi. Input the new image semantic information an to be matched and the motion position prediction pn. Search the matching table for the nearest neighbor ai of the semantic information an to obtain the paired position pi. Use an and pi to construct a new node and insert it into the octree. In the octree, the node feature is the image semantic information vector. Starting from the newly inserted node, traverse the octree for nearest neighbor search. Calculate the feature Euclidean distance between the new node and the existing node. Select the existing node with the smallest Euclidean distance to the new node. Take out the image semantics and corresponding motion position stored in the nearest neighbor node. Fusion the nearest neighbor image semantics and motion position with an and pn of the new node. Finally, the image semantics and motion planning information matching the new data are obtained.

[0078] Specifically, octree: a tree data structure that uses space partitioning to organize point cloud data so that it can efficiently store and access spatial data. It is similar to a binary tree, except that each node of the octree has a maximum of 8 child nodes. In this application, an octree node is defined, which contains an image semantic feature vector and the corresponding motion position coordinates. The center position of the image semantic feature vector is calculated as the origin of the segmented space. Recursively divide the space into 8 octave subspaces to construct an octree. Insert the image semantic feature vector into the corresponding leaf node according to the subspace it is in. For each non-leaf node, store the maximum and minimum coordinates within its range. The construction process continues until the set maximum depth or number of leaf nodes is reached. Finally, an octree with the image semantic feature vector as the point cloud is obtained. The octree node also saves the motion position to facilitate subsequent nearest neighbor search and data matching.

[0079] Specifically, Nearest Neighbor Search: an algorithm for searching for the nearest neighbor sample (with the shortest distance) to a sample in a feature space. It is often used to search for known samples in the space that are most similar to a given query sample. In this application, an octree containing image semantic features has been constructed. Input a new image semantic feature vector as a query sample. In the octree, calculate the distance between the query vector and all node feature vectors, such as the Euclidean distance. Return the node feature vector closest to the query vector. The image semantic information corresponding to the nearest neighbor node is most similar to the input query sample. Take out the motion position of the nearest neighbor node as the matching motion information of the input image semantics. A quick search is achieved to find the matching content that is closest to the input image semantic information.

[0080] Specifically, Euclidean distance: a measure of the straight-line distance between two points, equal to the square root of the sum of the squares of the coordinate differences of the two points. In multidimensional space, the similarity distance between the feature vectors of two points can be represented. In the present application, the Euclidean distance between the semantic feature vector of the new image and each node in the octree is calculated. The node with the smallest Euclidean distance to the new feature vector is selected as the nearest neighbor. The image semantic information and the corresponding motion position stored in the nearest neighbor node are taken out. The semantic feature vectors of the two are concatenated and fused. The motion position coordinates of the two are weighted averaged and fused. The weights are assigned according to the inverse of the Euclidean distance between the two. The closer the distance, the greater the weight. After fusion, a feature representation and motion position that are more similar to the new data are obtained. As the result of the matching adjustment of the new data.

[0081] Figure 5 is a flowchart of exemplary sub-steps for generating a feature map according to some embodiments of this specification, such as Figure 5 As shown, S510 obtains the pixel coordinates of the selected image features as feature points; S520 constructs an extended Kalman filter EKF SLAM model based on the feature points and the corresponding motion positions; S530 sets the state variables of the extended Kalman filter EKF SLAM model to include the position of the robot and the position of the feature points; S540 iteratively optimizes the position of the feature points according to the prediction update of the extended Kalman filter EKF SLAM model, and generates an initial feature map of the preset workspace. Among them, the node with the minimum Euclidean distance of the image feature has been selected to obtain the image semantic features and the corresponding motion positions. Pixel coordinates are extracted from the semantic features as feature points. The EKF SLAM model is constructed, and the state variables include the robot position and the positions of all feature points. The motion model predicts the robot position, and the measurement model updates the feature point positions based on feature observations. The observation matrix is ​​set according to the relative observation of the current feature point to other feature points. EKF predicts and updates iteratively in chronological order to optimize the positions of all feature points. The matched semantic features and motion trajectories are used as the prior input of EKF. After iterating EKF SLAM, the initial feature map is obtained, which represents the distribution of environmental feature points. The map can be optimized incrementally in the future for autonomous robot navigation.

[0082] Specifically, Extended Kalman Filter EKF SLAM (Extended Kalman Filter Simultaneous Localization and Mapping), a recursive probabilistic filtering method, simultaneously performs robot self-positioning and environmental map construction. It linearizes nonlinear systems, performs predictions and updates, and estimates the position distribution of robots and feature points. State variable: a variable that describes the state of the system, representing the state quantity that needs to be estimated. Such as the position of the robot and map feature points that need to be estimated in EKF SLAM. In this application, the state variables include the robot position and the environmental feature point position. The motion model predicts the robot position, and the measurement model updates the feature point position based on feature observations. EKF recursively predicts and updates the state to optimize the position distribution of the robot and feature points. The matched features and motion information are used to initialize the EKF a priori. The feature point positions are optimized through EKF iterations to generate an environmental map. The map represents the knowledge of the position distribution of environmental features and can be used for robot navigation.

[0083] More specifically, feature points (Feature Point), which are representative and distinguishing points in the environment, such as image corner points, edge points, etc., can be used in SLAM to build an environmental map. They have relatively stable features. In this application, pixel coordinates are extracted from the matching image semantics as feature points. The feature point positions are added to the state variables of EKF SLAM. The motion model predicts the robot position, and the measurement model updates the feature point position distribution based on the feature point observations. Iterate EKF SLAM to optimize the feature point positions and generate an environmental feature map. Feature points constitute the nodes of the map, representing the characteristic patterns of the environment. Map knowledge can be used for robot navigation and autonomous exploration.

[0084] More specifically, initialize EKF SLAM, and the state variables contain the robot pose and the extracted feature point positions. Perform predictions for time step k, use the motion model to predict the new pose of the robot, and propagate the prior estimates and covariance matrix of the feature point positions; when there are feature point observations, perform updates: calculate the observation predictions and residuals, use the Kalman gain to update the state variables, including the feature point positions, repeat the prediction and update steps, and iterate the EKF SLAM process. Each update will optimize the posterior estimate of the feature point position distribution. When the iteration terminates, the current position estimates of all feature points constitute the initial feature map. The map contains the location and distribution information of the feature points in the environment, that is, the initial feature representation in the preset workspace is obtained. The map can be optimized later and used for navigation, positioning, etc.

[0085] S550 constructs a recursive Bayesian filter model, taking the initial feature map as the observation input; each pixel coordinate in the initial feature map is used as the state variable of the recursive Bayesian filter model; S560 iteratively optimizes each state variable according to the prediction equation and update equation of the recursive Bayesian filter model; and outputs the final feature map composed of the iteratively optimized state variables. Among them, the constructed initial feature map is input, including the location distribution of feature points. The recursive Bayesian filter model is constructed, and the state variable is each feature point in the map. The prediction equation predicts the location distribution of feature points according to the existing map features. The update equation integrates the actual feature observations and modifies the predicted feature point state. The initial map features are used as the first round of recursive filtering input. Iterate the recursive prediction and update process to gradually optimize the state of each feature point. The optimization goal is to reduce the variance of the feature point position and improve the positioning accuracy. Repeat the iteration until the feature point state is sufficiently stable. The feature point state obtained by the final iteration constitutes the optimized final feature map. As a knowledge representation, the map can support the robot's autonomous tasks and decision-making.

[0086] Specifically, recursive Bayesian filtering (using Bayesian Filtering), a type of probabilistic filtering algorithm that recursively predicts and updates, uses Bayes' rule to achieve online estimation of the state of a dynamic system. In this application, the initial feature map is used as the input observation. The position of each feature point is used as the state variable of the recursive filtering. The prediction equation predicts the position distribution of the feature point based on the current state. The update equation uses the new observation results to modify the predicted distribution of the state. Predict and update recursively to optimize the distribution of feature point positions. The feature point positioning is made more accurate through iteration. Finally, the optimized feature point state is output to form a refined feature map. This map knowledge can support robot autonomous tasks and decision-making.

[0087] Specifically, state variable: a variable that describes the state of the system, which needs to be estimated online using a Bayesian filtering model. Such as the position state of each feature point in the map. In this application, the feature point state variables are initialized, and the initial value and covariance matrix are set. The prediction step at time k is performed: the feature point state prior value and covariance matrix are predicted based on the motion and measurement model. When there is a new observation of the feature point, an update step is performed to calculate the Kalman gain, and update the state posterior estimate and covariance matrix. The prediction and update process is repeated recursively. Each update will reduce the covariance of the state variables and enhance the estimation accuracy. Through recursive iteration, the state variables are continuously optimized with new observations. When the recursion is terminated, the optimized state variables are output.

[0088] Specifically, in the present application, a recursive Bayesian filtering model is constructed, and the state variable is each feature point in the initial feature map. It is optimized according to the prediction and update equation iterations, and multiple rounds of recursive operations are performed. Each round of recursive operation will update the feature point state variables to improve the positioning accuracy. Set the recursive termination condition, such as reaching the specified number of iterations, or the state change is less than a threshold. When the recursion ends, collect the current state variable values ​​of all feature points. The optimized coordinate position of each feature point is used as a vertex. Use the optimized coordinates to reconstruct a two-dimensional grid map or a point cloud map. In the rasterized map, each pixel is described by the final optimized feature. Finally, a refined feature environment map is generated as a knowledge representation of the robot's perception of the world.

[0089] Furthermore, the recursive Bayesian filtering model adopts a first-order Markov process. Among them, the first-order Markov process (First-order Markov process): a random process in which the future state depends only on the current state and has nothing to do with the past state. It has the property of no aftereffect. Specifically, the recursive Bayesian filtering assumes that the state transition has the Markov property. That is, the state of the feature point at the next moment depends only on the current state. In the prediction step, the transition probability is calculated based on the current feature point state. Generate the predicted distribution of the feature point position at the next moment. In the update step, the predicted distribution is corrected according to the latest observation. The prediction and update processes are recursively performed in sequence. The Markov assumption is used to simplify the recursive operation. Finally, the optimized feature point state is output to construct the map.

[0090] More specifically, in a specific embodiment of the present application, cameras are arranged in a smart warehouse to obtain environmental image information. Feature points in the image, such as doorways, aisles, shelves, etc., are extracted through semantic segmentation. A first-order Markov recursive Bayesian filtering model is constructed. The position state of each feature point is used as a filtering state variable. According to the current feature point state, the feature point position distribution at the next moment is predicted. New observations are integrated, the feature point state is updated, and recursive optimization is performed. The Markov property is used to simplify multi-step prediction operations. When the recursion terminates, all feature state variables are collected to generate a map. The optimized feature map is used to accurately locate the robot in the warehouse. The map is used to realize autonomous navigation and path planning of the robot.

Claims

1. A remote positioning method combining an AR camera with an Internet of Things system for controlling a robot, characterized in that: Establish UWB network communication connection between AR camera and robot; When the robot is in the preset workspace, the AR camera is controlled to move and collect image data in the preset workspace, and the motion data of the AR camera is collected; Use deep learning algorithms to extract image semantic information from image data; And the extracted image semantic information is fused with the collected motion data; Construct feature maps of preset workspaces; According to the constructed feature map of the preset workspace, the coordinate information of the robot in the preset workspace is obtained; based on the obtained coordinate information, the AR camera sends control instructions to the robot through the UWB network; Constructing a feature map of a preset workspace also includes the following steps: Construct a convolutional neural network as a semantic segmentation network, which includes an encoder and a decoder; The encoder performs convolution operation on the input image data, extracts image features, and outputs an image feature map; The image feature map is input into the decoder, and the decoder calculates the weights between features by constructing a quadratic kernel matrix; The decoder performs semantic segmentation on the image feature map according to the image feature map output by the encoder and the calculated weights to generate image semantic information; Among them, constructing the secondary kernel matrix includes: Construct a quadratic kernel matrix K, where the element Kij represents the similarity between feature i and feature j in the feature graph; The similarity is calculated using the Gaussian kernel function to obtain the quadratic kernel matrix; According to the constructed secondary kernel matrix K, the weight of each feature in the feature map is calculated; The following steps are also included: Collect ambient light information of a preset workspace through an illumination sensor; Taking image semantic information as input, where the image semantic information includes the RGB value of each pixel; Establish a Gaussian process regression model between the RGB value of image semantic information and ambient lighting information; The covariance matrix of the Gaussian process regression model is low-rank approximated by the singular value decomposition algorithm; The Gaussian process regression model after low-rank approximation is used to combine the ambient lighting information to predict the RGB value of each pixel in the image semantic information and generate the adjusted image semantic information. The generated adjusted image semantic information includes: Perform singular value decomposition on the covariance matrix of the Gaussian process regression model to obtain the product form of multiple singular values ​​and corresponding left and right singular vectors; The main singular value and the corresponding left and right singular vectors are selected according to the size of the singular value, and the low-rank matrix is ​​used to approximate the original covariance matrix to obtain the covariance matrix after low-rank approximation; Based on the covariance matrix after low-rank approximation, the probability inference of the Gaussian process regression model is performed, where: Calculate the Gaussian process kernel function value between the ambient light information variable X and the observed data to obtain the kernel vector k; Multiply the kernel vector k by the inverse matrix of the covariance matrix of the low-rank approximation to obtain the Gaussian distribution mean μ; Multiply the kernel vector k by the inverse matrix of the covariance matrix of the low-rank approximation and then multiply it by the transpose of k to obtain the Gaussian distribution variance ; According to the Gaussian distribution parameters of the regression value , get the probability distribution under the new input X*; For each image pixel, sample and generate a Gaussian probability prediction of the RGB value according to the probability distribution; Sampling all image pixels, obtaining RGB values ​​after image semantic information is adjusted, and generating image semantic information after input illumination adjustment; Generate adjusted image semantic information, including: Use the calibration plate to collect the RGB value of each pixel in the image data under different illumination values, and establish the corresponding relationship between the illumination value and the RGB value; Calculate the Gaussian process regression coefficient between the RGB value of each pixel and the corresponding illumination value; Establish a conditional Pixel CNN model based on deep learning to learn the prior knowledge of the illumination response of each pixel; The Gaussian process regression coefficient and the prior knowledge of illumination response are used as input to predict the RGB value of each pixel through the Gaussian process regression model; According to the predicted RGB value, the RGB value of each pixel in the image semantic information is adjusted to generate the image semantic information after the pixel is adjusted; Among them, obtaining the coordinates of the robot includes: The robot obtains the current environment image through the AR camera; Analyze the acquired environment image to obtain feature points within a preset range of the robot in the environment image; Match the acquired feature points with the constructed feature map; Statistically analyze the coordinate distribution of all matching feature points; According to the coordinate distribution of the matching feature points, the coordinates of the robot in the feature map coordinate system are calculated.

2. The remote positioning method of the AR camera combined with the Internet of Things system according to claim 1, characterized in that: The extracted image semantic information is integrated with the collected motion data, and also includes: By parsing the meta information of the image data, a timestamp sequence A of the image semantic information is extracted; By parsing the timestamp of the motion data, extracting the timestamp sequence B of the motion data; Establish a Seq2Seq model based on LSTM, take timestamp sequence A and timestamp sequence B as input, and output the matched time index sequence; According to the time index sequence, the time sequence of image semantic information and motion data is matched.

3. The remote positioning method of the AR camera combined with the Internet of Things system according to claim 2, characterized in that: Also includes: Using the matched time index sequence, the image semantic information of the corresponding time index is obtained; The acquired image semantic information of the corresponding time index is input into the IMM motion model to predict the motion position of the corresponding image acquisition moment; The predicted motion position is paired with the image semantic information of the corresponding time index by establishing a matching table.

4. The remote positioning method of the AR camera combined with the Internet of Things system according to claim 3 is characterized in that: After predicting the motion position corresponding to the image acquisition time, it also includes: Obtaining corresponding image semantic information paired with the predicted motion position through a matching table; The acquired image semantic information and the corresponding predicted motion position are used as nodes to construct an octree for nearest neighbor search; Traverse the search octree and select the image features and motion positions with the smallest Euclidean distance between nodes for alignment and fusion.

5. The remote positioning method of the AR camera combined with the Internet of Things system according to claim 4, characterized in that: After selecting the image features and motion positions with the smallest Euclidean distance between nodes for alignment and fusion, it also includes: Obtain pixel coordinates of selected image features as feature points; According to the feature points and the corresponding motion positions, the extended Kalman filter EKF SLAM model is constructed; Set the state variables of the extended Kalman filter EKF SLAM model to include the position of the robot and the position of the feature points; According to the prediction update of the extended Kalman filter EKF SLAM model, the position of the feature points is iteratively optimized to generate the initial feature map of the preset workspace.

6. The remote positioning method of the AR camera combined with the Internet of Things system according to claim 5, characterized in that: After generating the initial feature map, it also includes: Construct a recursive Bayesian filtering model, taking the initial feature map as observation input; The coordinates of each pixel in the initial feature map are used as the state variables of the recursive Bayesian filtering model; Iteratively optimize each state variable according to the prediction equation and update equation of the recursive Bayesian filtering model; Output the final feature map consisting of the iteratively optimized state variables.

7. The remote positioning method of the AR camera combined with the Internet of Things system according to claim 1, characterized in that: The recursive Bayesian filtering model adopts a first-order Markov process.

Images