Visual positioning and image processing method, device and storage medium

Abstract

The embodiments of the present application provide a visual positioning and image processing method, device and storage medium. In the embodiments of the present application, a mobile device collects environmental images around its location during movement, and uses a neural network model to analyze the motion properties of at least a portion of the pixels in the environmental image. Then, based on the motion properties of at least a portion of the pixels in the environmental image, the mobile device is positioned using static pixels. Static pixels are feature points whose positions are fixed or substantially unchanged in the environment where the mobile device is currently located. Compared with feature points in motion, using static pixels is more conducive to positioning and helps improve positioning accuracy.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a visual positioning and image processing method, device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, robots have gradually entered people's daily lives, bringing great convenience to people's lives. Regardless of the type of robot, it needs to be positioned during the autonomous movement process.

[0003] In the prior art, robots can achieve autonomous positioning with the help of visual positioning technology. However, the positioning accuracy of the existing visual positioning method is low. Summary of the invention

[0004] Multiple aspects of the present application provide a visual positioning and image processing method, device and storage medium to improve the accuracy of visual positioning.

[0005] The present application embodiment provides a visual positioning method, which is applicable to a mobile device, including:

[0006] During the movement, collect environmental images around the current location;

[0007] Analyzing motion properties of at least a portion of pixels in the environment image using a neural network model;

[0008] Determining static pixels among the at least a portion of pixels according to the motion attributes of the at least a portion of pixels;

[0009] The current location of the mobile device is located according to the static pixel points and a known environment map.

[0010] The present application also provides an image processing method, including:

[0011] Get the image to be processed;

[0012] Using the first network model to extract features from the image to be processed to obtain a plurality of initial feature maps;

[0013] Calculating motion attributes of the multiple initial feature maps using a second network model to obtain motion attributes of at least a portion of pixels in the image to be processed;

[0014] The third network model is used to calculate feature descriptors of the multiple initial feature maps to obtain feature descriptors of at least a portion of pixels in the image to be processed.

[0015] The present application also provides an image processing method, including:

[0016] Inputting the image to be processed into a neural network model, wherein the neural network model includes at least two layers of structure;

[0017] Using the first layer network in the neural network model, extracting features from the image to be processed to obtain a plurality of initial feature maps;

[0018] Using the second layer network in the neural network model, the motion attributes of the multiple initial feature maps are calculated to obtain the motion attributes of at least a portion of the pixels in the image to be processed;

[0019] Determine, according to the motion attribute of at least a portion of the pixels, a target pixel whose position change is less than or equal to a preset threshold;

[0020] The position of the device that collects the image to be processed during the process of collecting the image to be processed is located according to the target pixel point and a known environment map.

[0021] The present application also provides a visual positioning method, which is applicable to a terminal device, and the method includes:

[0022] During the movement of the user, an environmental image around the current location of the user is collected, and the user carries the terminal device;

[0023] Analyzing motion properties of at least a portion of pixels in the environment image using a neural network model;

[0024] Determining static pixels among the at least a portion of pixels according to the motion attributes of the at least a portion of pixels;

[0025] The current location of the user is located according to the static pixel points and a known environment map, and the current location of the user is output to guide the user to move.

[0026] The present application also provides a data processing method applicable to a mobile device, the method comprising:

[0027] During the movement, collect environmental images around the current location;

[0028] Analyzing motion properties of at least a portion of pixels in the environment image using a neural network model;

[0029] Determining static pixels among the at least a portion of pixels according to the motion attributes of the at least a portion of pixels;

[0030] Locating the current location of the mobile device according to the static pixel points and a known environment map;

[0031] The travel state information of the mobile device is calculated according to the current location of the mobile device and at least one previous historical location.

[0032] The embodiment of the present application further provides a mobile device, comprising: a mechanical body; the mechanical body is provided with a visual sensor, one or more processors, and one or more memories; wherein,

[0033] The one or more memories are used to store computer instructions and environment maps;

[0034] The one or more processors, coupled to the one or more memories, are configured to execute the computer instructions for:

[0035] During the movement, collect environmental images around the current location;

[0036] Analyzing the motion properties of each pixel in the environment image using a neural network model;

[0037] Determining static pixels in at least a portion of pixels in the environment image according to motion attributes of at least a portion of pixels in the environment image;

[0038] The current location of the mobile device is located according to the static pixel points and a known environment map.

[0039] The embodiment of the present application also provides a computer device, including: a memory and a processor; wherein:

[0040] The memory is used to store computer instructions;

[0041] The processor, coupled to the one or more memories, is configured to execute the computer instructions for:

[0042] Get the image to be processed;

[0043] Using the first network model to extract features from the image to be processed to obtain a plurality of initial feature maps;

[0044] Calculating motion attributes of the multiple initial feature maps using a second network model to obtain motion attributes of at least a portion of pixels in the image to be processed;

[0045] The third network model is used to calculate feature descriptors of the multiple initial feature maps to obtain feature descriptors of at least a portion of pixels in the image to be processed.

[0046] The embodiment of the present application further provides a computer device, characterized in that it includes: a memory and a processor; wherein:

[0047] The memory is used to store computer instructions;

[0048] The processor, coupled to the one or more memories, is configured to execute the computer instructions for:

[0049] Inputting the image to be processed into a neural network model, wherein the neural network model includes at least two layers of structure;

[0050] Using the first layer network in the neural network model, extracting features from the image to be processed to obtain a plurality of initial feature maps;

[0051] Using the second layer network in the neural network model, the motion attributes of the multiple initial feature maps are calculated to obtain the motion attributes of at least a portion of the pixels in the image to be processed;

[0052] Determine, according to the motion attribute of at least a portion of the pixels, a target pixel whose position change is less than or equal to a preset threshold;

[0053] The position of the device that collects the image to be processed during the process of collecting the image to be processed is located according to the target pixel point and a known environment map.

[0054] The embodiment of the present application further provides a terminal device, comprising: a memory, a processor and a camera; wherein the camera is used to: collect an image of the environment around the current location of the user during the user's movement, and the user carries the terminal device;

[0055] The memory is used to store computer instructions;

[0056] The processor, coupled to the one or more memories, is configured to execute the computer instructions for:

[0057] Analyzing motion properties of at least a portion of pixels in the environment image using a neural network model;

[0058] Determining static pixels among the at least a portion of pixels according to the motion attributes of the at least a portion of pixels;

[0059] The current location of the user is located according to the static pixel points and a known environment map, and the current location of the user is output to guide the user to move.

[0060] The embodiment of the present application also provides a mobile device, comprising: a memory, a processor and a camera; wherein the camera is used to: during the movement of the mobile device, collect an image of the environment around the current location;

[0061] The processor, coupled to the one or more memories, is configured to execute the computer instructions for:

[0062] Analyzing motion properties of at least a portion of pixels in the environment image using a neural network model;

[0063] Determining static pixels among the at least a portion of pixels according to the motion attributes of the at least a portion of pixels;

[0064] Locating the current location of the mobile device according to the static pixel points and a known environment map;

[0065] The travel state information of the mobile device is calculated according to the current location of the mobile device and at least one previous historical location.

[0066] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned visual positioning method.

[0067] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned image processing method.

[0068] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned data processing method.

[0069] In the embodiment of the present application, the mobile device collects the environment image around its location during the movement, and uses the neural network model to analyze the motion properties of at least a part of the pixels in the environment image; then, based on the motion properties of at least a part of the pixels in the environment image, the mobile device is positioned using static pixels. Static pixels are feature points whose positions are fixed or basically unchanged in the environment where the mobile device is currently located. Compared with feature points in motion, using static pixels is more conducive to positioning and helps to improve positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0071] Figure 1a A schematic diagram of a flow chart of a visual positioning method provided in an embodiment of the present application;

[0072] Figure 1b A schematic diagram of the structure of a network model provided in an embodiment of the present application;

[0073] Figure 1c A schematic diagram of the structure of a neural network training model provided in an embodiment of the present application;

[0074] Figure 1d A schematic diagram of a training process of a network model provided in an embodiment of the present application;

[0075] Figure 1e A flowchart of an image processing method provided in an embodiment of the present application;

[0076] Figure 2a A schematic diagram of the structure of an edge cloud network system provided in an embodiment of the present application.

[0077] Figure 2b A flowchart of another image processing method provided in an embodiment of the present application;

[0078] Figure 2c A schematic diagram of an application scenario of an image processing method provided in an embodiment of the present application;

[0079] Figure 3 A hardware structure block diagram of a mobile device provided in an embodiment of the present application;

[0080] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0081] Figure 5 A schematic diagram of the structure of another computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0083] In order to solve the technical problem of low positioning accuracy of existing visual positioning methods, in some embodiments of the present application, during the movement of the mobile device, the motion properties of at least a part of the pixels in the surrounding environment image collected by the mobile device are analyzed using a convolutional neural network model; based on the motion properties of at least a part of the pixels in the surrounding image, the mobile device is positioned using static pixels. Static pixels are feature points whose positions are fixed or basically unchanged in the surrounding environment where the mobile device is currently located. Compared with moving feature points, using static pixels is more conducive to positioning and helps to improve positioning accuracy.

[0084] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0085] Figure 1a The flowchart of a visual positioning method provided in the embodiment of the present application is shown in FIG. The method is applicable to autonomous mobile devices. Figure 1a As shown, the method includes:

[0086] 101. During the movement, collect environmental images around the current location.

[0087] 102. Use a neural network model to analyze the motion properties of at least a portion of the pixels in the environment image.

[0088] 103. Determine static pixels among at least a portion of the pixels in the environment image according to motion attributes of at least a portion of the pixels in the environment image.

[0089] 104. Locate the current location of the autonomous mobile device based on the static pixel points and the known environment map.

[0090] The visual positioning method provided in this embodiment is applicable to autonomous mobile devices. The autonomous mobile device may be a robot that can move autonomously, an unmanned vehicle, or a drone, etc. This embodiment does not limit the shape of the robot, unmanned vehicle, and drone. For example, the robot may be circular, elliptical, triangular, convex polygonal, or humanoid. For another example, the unmanned vehicle may be a small car, a bus, or a freight truck, etc. The autonomous mobile device may implement the logic of the visual positioning method for the autonomous mobile device provided in this embodiment by installing software, an APP, or writing program code in the corresponding device.

[0091] In this embodiment, the autonomous mobile device can complete certain tasks on the basis of movement. The following is an exemplary explanation using a robot as an example. For example, in shopping scenarios such as supermarkets and shopping malls, a shopping cart robot needs to move with customers to accommodate the goods selected by customers. For another example, in some companies' warehousing and sorting scenarios, the sorting robot needs to follow the sorting personnel to move to the shelf picking area, and then start sorting the ordered goods. For another example, in a home cleaning scenario, the sweeping robot needs to clean the living room, bedroom, kitchen and other areas. In these application scenarios, the robot completes the corresponding tasks during autonomous movement.

[0092] Regardless of the implementation form of the autonomous mobile device, it needs to implement autonomous positioning and navigation during the autonomous movement process. In this embodiment, the autonomous positioning of the autonomous mobile device is mainly illustrated by way of example.

[0093] In this embodiment, a visual sensor is provided on the autonomous mobile device. In step 101, the visual sensor can collect an image of the environment around the current location of the autonomous mobile device during the movement of the autonomous mobile device. The visual sensor can be a camera, such as a binocular camera, a monocular camera, or a depth of field camera, but is not limited thereto.

[0094] Next, in step 102, the motion attributes of at least a portion of the pixels in the environment image collected in step 101 are analyzed using a neural network model. In each embodiment of the present application, at least a portion of the pixels in the environment image may be a portion of the pixels in the environment image, or may be all the pixels in the environment image. If at least a portion of the pixels in the environment image are a portion of the pixels in the environment image, the selection of these pixels may be flexibly set, and preferably, at least a portion of the pixels are key points in the environment image. Further, the number of types of motion attributes may be flexibly set according to actual needs. For example, the types of motion attributes may be set to: static, unstable, moving and uncertain, etc., but are not limited thereto. Accordingly, pixels with static, unstable, moving and uncertain attributes are referred to as static pixels, unstable pixels, moving pixels and uncertain pixels, respectively. In particular, static pixels are feature points whose positions are fixed or substantially unchanged in the surrounding environment where the autonomous mobile device is currently located. In particular, substantially unchanged positions refer to points whose position changes are less than or equal to a preset change threshold. The preset change threshold may be flexibly set according to actual needs and is not limited here. The preset change threshold can be 1cm, 5cm, etc., but is not limited to this. Unstable pixel points are feature points whose positions are unstable in the surrounding environment where the autonomous mobile device is currently located. Moving pixel points are feature points whose positions are not fixed and are constantly moving in the surrounding environment where the autonomous mobile device is currently located. Uncertain pixel points refer to pixel points whose motion attributes cannot be determined. Taking outdoor scenes as an example, several types of pixel points are illustrated: among them, static pixel points can be points on the outer walls of buildings, points on zebra crossings on the road, or points on traffic lights, etc.; unstable pixel points can be points on clouds in the sky or points on cars parked on the roadside, etc.; moving pixel points can be people and cars walking on the road, etc.

[0095] Further, in step 103, static pixels in at least a portion of pixels in the environment image may be determined based on the motion attributes of the at least a portion of pixels, and in step 104, the current location of the autonomous mobile device may be located based on the static pixels and the known environment map. Optionally, the current location of the autonomous mobile device may be represented by the position coordinates corresponding to the current location in the environment map, but is not limited thereto.

[0096] In the embodiments of the present application, the known environment map may be a pre-stored environment map or an environment map constructed in real time by an autonomous mobile device during movement. The specific construction process will be described in the embodiments below and will not be elaborated here.

[0097] It is worth noting that, in the present embodiment of the application, the environment image may be an RGB image or a grayscale image. Optionally, if the environment image captured by the visual sensor is an RGB image, it may be converted into a grayscale image before step 102 .

[0098] In this embodiment, the autonomous mobile device collects environmental images around its location during movement, and uses a neural network model to analyze the motion properties of at least a portion of the pixels in the environmental images collected; then, based on the motion properties of at least a portion of the pixels, autonomous positioning is performed using static pixels. Static pixels are feature points whose positions are fixed or substantially unchanged in the environment where the autonomous mobile device is currently located. Compared with feature points in motion, using static pixels is more conducive to positioning and helps improve positioning accuracy.

[0099] In the embodiments of the present application, various neural network models may be used to analyze the motion properties of at least a portion of the pixels in the environment image. For example, the neural network model may be a convolutional neural network model, but is not limited thereto. Further, the convolutional neural network model may be a fully convolutional neural network (FCN) based on ResNet, MobileNet, ShuffleNet, etc., but is not limited thereto. The following is an exemplary description using a traditional convolutional neural network as an example.

[0100] It should be noted that in the embodiments and drawings of the present application, ConvB and Conv both indicate that the layer is a convolutional layer; and each ConvB is followed by only a batch normalization layer (BatchNorm) and a truncated activation layer (not shown in the drawings). Furthermore, a Sigmoid function, a tanh function, or a Relu function may be used in the truncated activation layer. Furthermore, K represents the size of the convolution kernel of the convolution layer or the pooling operation kernel of the pooling layer (kernel size); P represents the size of the feature map edge expansion (paddingsize); S represents the stride size (stride size); and F represents the number of output feature maps.

[0101] In the embodiments of the present application, Figure 1bAs shown, the first network model can be used to extract features from the environmental image to obtain multiple feature maps. Here, multiple refers to 2 or more, and its specific value is determined by the number of output feature maps of the last convolutional layer of the first network model. Here, the number of output feature maps of the last convolutional layer of the first network model can be flexibly set according to application requirements. Further, for the convenience of description and distinction, the feature map output by the first network model is defined as an initial feature map. Further, the second network model can be used to calculate the motion attributes of the multiple initial feature maps output by the first network model to obtain the motion attributes of at least a part of the pixels in the environmental image. In the following embodiments, for the convenience of distinction and description, the number of multiple initial feature maps output by the first network model is represented by M, where M≥2 and is an integer. The convolution layer is in Figure 1b In the example, M=128 is used only for illustration, but is not limited thereto.

[0102] It is worth noting that in the embodiment of the present application, both the first network model and the second network model can be fully convolutional network models. For the convenience of description and distinction, the fully convolutional network models used by the first network model and the second network model are defined as the first fully convolutional network model and the second fully convolutional network model, respectively. Figure 1b The first network model and the second network model are implemented as fully convolutional network models for illustrative purposes only and are not intended to be limiting.

[0103] Further, in an embodiment of the present application, if the second network model is a second fully convolutional network model, the feature map output by the output layer of the second fully convolutional network model is the characteristic embodiment of at least a part of the pixels in the environment image under each motion attribute. For the convenience of description and distinction, the feature map output by the second fully convolutional network model is defined as a first target feature map. There are multiple first target feature maps, which is the same as the number of types of motion attributes defined in this embodiment. Among them, each first target feature map is a characteristic embodiment of the environment image under a motion attribute, that is, the number of first target feature maps output by the last convolutional layer of the second fully convolutional network is equal to the number of types of motion attributes. In an embodiment of the present application, Q represents the number of types of motion attributes, that is, the number of first target feature maps, and Q ≥ 2, and is an integer. For example, Figure 1b As shown in , if the motion attributes include: static, unstable, moving and uncertain, then Q = 4. Based on this, Figure 1b As shown, the above-mentioned M initial feature maps can be used as input images of the second fully convolutional network model and input into the second fully convolutional network model; in the second fully convolutional network model, at least one convolution calculation is performed on the M initial feature maps, and Q first target feature maps can be output; furthermore, the motion attributes of at least a part of the pixels in the environment image can be calculated based on the Q first target feature maps.

[0104] Accordingly, if Figure 1b As shown, if the first network model is the first fully convolutional network model, then when the first network model is used to extract features from the environment image, the environment image can be used as the input image of the first fully convolutional network model and input into the first fully convolutional network model. Further, in the first fully convolutional network model, at least one layer of convolution processing is performed on the environment image to output M initial feature maps; wherein each layer of convolution processing includes: at least one convolution calculation and downsampling processing after at least one convolution calculation. Among them, downsampling processing is performed on the feature map obtained after at least one convolution calculation, which can reduce the dimension of the initial feature map output by the first fully convolutional network model, reduce the subsequent amount of calculation in the second fully convolutional network model, and improve the calculation rate. Furthermore, the process of downsampling the feature map can be completed by the pooling layer in the first fully convolutional network model, but is not limited to this.

[0105] In the first fully convolutional network model, the specific number of layers for convolution processing of the environment image can be flexibly set according to actual needs. Figure 1b In the example, 3 layers of convolution processing are performed on the environment image, but it is not limited to 3 layers, and can also be 2 layers, 4 layers, 5 layers or even more layers. The number of times the environment image is convolved in each layer of convolution processing can be determined by the number of convolution layers used; and the number of convolution calculations in each layer can be the same or different. Figure 1b In the example, each layer includes two convolutional layers, which is not limited thereto. Optionally, Figure 1b As shown in the first fully convolutional network model, after at least one layer of convolution processing, the feature map obtained can continue to be convolved for at least one more time (in Figure 1b Taking 2 convolution calculations as an example), we get M initial feature maps.

[0106] It is worth noting that in the embodiments of the present application, Figure 1b The specific values ​​of the convolution kernel size K, the feature map edge expansion size P, the stride size S, and the number of output feature maps F shown in each convolution layer for convolution calculation are all exemplary and do not constitute a limitation. When downsampling is performed in each pooling layer, the size K and stride size S of the pooling operation kernel used are also exemplary and do not constitute a limitation.

[0107] Accordingly, since the M initial feature maps output by the first fully convolutional network model are feature maps after downsampling processing, their dimensions are smaller than the dimensions of the environment image. In the embodiment of the present application, the dimensions of the environment image and the feature map both refer to the height*width of the corresponding image. Based on the M initial feature maps output by the first fully convolutional network model after downsampling processing, in the second fully convolutional network, when calculating the motion attributes of at least a part of the pixels in the environment image, the Q first target feature maps can be firstly dimensionally upgraded to obtain Q feature maps with the same dimension as the environment image, and then the motion attributes of at least a part of the pixels in the environment image can be calculated based on the Q feature maps after dimension upgrade.

[0108] Optionally, a bilinear interpolation algorithm, a nearest neighbor method, a bicubic interpolation method or an area interpolation method may be used to perform dimensionality upscaling on the Q first target feature maps, but is not limited thereto. If a bilinear interpolation algorithm is used to perform dimensionality upscaling on the Q first target feature maps, bilinear interpolation may be performed on the Q first target feature maps according to the dimensionality difference between the Q first target feature maps and the environment image to obtain Q dimensional upscaled feature maps with the same dimension as the environment image.

[0109] Furthermore, in the second fully convolutional network model, the Q dimensionally upgraded feature maps can be normalized respectively to obtain the probability value of each pixel in at least a portion of the pixels in the environment image under the Q motion attributes; for each pixel in at least a portion of the pixels in the environment image, the maximum probability value can be selected from the probability values ​​of each pixel under the Q motion attributes, and the motion attribute corresponding to the maximum probability value is used as the motion attribute of the pixel. Furthermore, the Q dimensionally upgraded feature maps can be normalized respectively using the Softmax function.

[0110] Furthermore, static pixels may be selected from at least a portion of the pixels in the environment image according to the motion attribute of each pixel in at least a portion of the pixels, and the current location of the autonomous mobile device may be located according to the static pixels and the known environment map. Optionally, feature descriptors of the static pixels may be obtained, and the current location of the autonomous mobile device may be located according to the feature descriptors of the static pixels and the feature descriptors of each location point recorded in the known environment map.

[0111] Furthermore, in order to further improve the positioning accuracy, the static pixel points in the environmental image can be screened, the key pixel points belonging to the key points in the environmental image can be screened out, and the key pixel points can be used to locate the autonomous mobile device. Among them, the key point is a local expression of the characteristics of the environmental image, which can reflect the local particularity of the environmental image. Accordingly, in an embodiment of the present application, the environmental image can also be input into a local feature detector to obtain the key points in the environmental image. Optionally, the local feature detector performs spot detection or corner detection. If the local feature detector performs spot detection, the LOG method, DOH method, SIFI algorithm or SURF algorithm can be used to obtain the key points in the environmental image; if the local detector performs corner detection, the Harris algorithm or FAST algorithm can be used to obtain the key points in the environmental image.

[0112] Further, according to the key points in the environment image, static pixel points belonging to the key points can be selected from static pixel points in at least a portion of the pixel points in the environment image as key pixel points. Further, according to the feature descriptors of the key pixel points and the feature descriptors of each position point in the environment map, the position coordinates of the key pixel points in the environment map are determined; and according to the position coordinates of the key pixel points in the environment map, the position coordinates of the autonomous mobile device in the environment map are calculated.

[0113] Optionally, when determining the position coordinates of the key pixel point in the environment map, the similarity between the feature descriptor of the key pixel point and the feature descriptor of each position point in the environment map can be calculated, and the position point whose similarity is greater than or equal to the set similarity threshold is used as the position point corresponding to the key pixel point in the environment map, that is, the position coordinates of the position point whose similarity is greater than or equal to the set similarity threshold in the environment map are used as the position coordinates of the corresponding key pixel point in the environment map. Afterwards, based on the position coordinates of these key pixel points in the environment map, the position coordinates and orientation of the autonomous mobile device in the environment map can be calculated, that is, the position and posture of the autonomous mobile device can be obtained.

[0114] Among them, the environmental map can be a pre-stored environmental map of the surroundings of the autonomous mobile device at the current location, or it can be an environmental map constructed in real time by the autonomous mobile device. In the case where the autonomous mobile device constructs the environmental map in real time, multiple environmental images of the autonomous mobile device, especially the current location period, can be collected. In order to facilitate description and distinction, the environmental image used for environmental map construction is defined as the initial environmental image. Further, the feature descriptors of at least a part of the pixels in these initial environmental images can be calculated. Further, the environmental map is constructed using the feature descriptors of at least a part of the pixels in these initial environmental images and the internal and external parameters of the visual sensor on the autonomous mobile device that collects the initial environmental image. Among them, the specific process of constructing the environmental map can refer to the relevant content in Simultaneous localization and mapping (SLAM), which will not be repeated here.

[0115] For the feature descriptors of at least a portion of the pixels in the above environment image, such as Figure 1b As shown, the third network model can be used to calculate feature descriptors for the multiple initial feature maps output by the above-mentioned first fully convolutional network model to obtain feature descriptors of at least a part of the pixels in the environment image, and obtain feature descriptors of static pixels from the feature descriptors of at least a part of the pixels in the environment image. Correspondingly, feature descriptors of key pixels can also be obtained from the feature descriptors of static pixels. Since the input image of the third network model is the multiple initial feature maps output by the first fully convolutional network model, which is the same as the input image of the second network model, the motion attributes of at least a part of the pixels in the environment image and the feature descriptors of these pixels can be determined by sharing the same basic network (i.e., the first network model) for feature extraction, thereby realizing computational reuse and reducing the amount of computation, which in turn helps to increase the computational rate and further improve the positioning efficiency.

[0116] Furthermore, the third network model may be a fully convolutional network model. For the convenience of description and distinction, the fully convolutional network model adopted by the third network model is defined as a third fully convolutional network model. Accordingly, the above-mentioned M initial feature maps can be used as input images of the third fully convolutional network model and input into the third fully convolutional network model; in the third fully convolutional network model, the M initial feature maps are subjected to at least one convolution process to obtain N feature maps. Among them, for the convenience of description and distinction, the feature map output by the third fully convolutional network model is defined as a second target feature map. Among them, each second target feature map is a feature description of at least a part of the pixels in the environment image; in this way, the N second target feature maps also form a feature descriptor of at least a part of the pixels in the environment image. In an embodiment of the present application, N is an integer greater than or equal to 2, and its specific value is determined by the number F of feature maps set by the last convolutional layer in the third fully convolutional network model. Figure 1b In the figure, N=F=128 is used for illustration, but it is not limited thereto.

[0117] Furthermore, if the M initial feature maps are feature maps output by the first fully convolutional network model after downsampling, then in the third fully convolutional network, the N second target feature maps can also be subjected to dimensionality upscaling processing to obtain N dimensionality-upscaled feature maps with the same dimension as the environment image, wherein each dimensionality-upscaled second target feature map is a feature description of at least a portion of the pixels in the environment image; thus, the N dimensionality-upscaled second target feature maps also form feature descriptors of at least a portion of the pixels in the environment image. For the implementation method of dimensionality upscaling of the N second target feature maps, please refer to the above-mentioned content of dimensionality upscaling of the Q first target feature maps, which will not be repeated here.

[0118] It is worth noting that in the embodiment of the present application, before using the neural network model to analyze the motion attributes of at least a portion of the pixels in the environmental image, the neural network model may need to be trained. The following takes the joint training of the first fully convolutional network, the second fully convolutional network, and the third fully convolutional network as an example to exemplarily illustrate the training process of the convolutional neural network model provided in the embodiment of the present application.

[0119] In an embodiment of the present application, the network architecture of the initial feature extraction network, the initial motion attribute branch network, and the initial feature descriptor branch network can be preset, wherein the network architecture of the initial feature extraction network includes: convolution layer, pooling layer, the number and setting order of these convolution layers and pooling layers, and the hyperparameters of each convolution layer and pooling layer. Among them, the hyperparameters of the convolution layer include: the size of the convolution kernel K (kernel size), the size of the feature map edge expansion P (paddingsize), the stride size S (stride size), and the number F of output feature maps. The hyperparameters of the pooling layer are the size K of the pooling operation kernel and the stride size S. Further, Figure 1c The number and setting order of convolutional layers and pooling layers in the initial feature extraction network structure shown in , as well as the hyperparameters of each convolutional layer and pooling layer, are only exemplary descriptions and do not constitute limitations thereto.

[0120] Furthermore, the network architecture of the initial motion attribute branch network includes: convolutional layer, normalization layer, number of convolutional layers and hyperparameters of each convolutional layer. Figure 1c The number of convolutional layers in the initial motion attribute branch network and the hyperparameters of each convolutional layer shown in the figure are exemplary and do not constitute a limitation.

[0121] The network architecture of the initial feature descriptor branch network includes: convolutional layers, the number of convolutional layers, and the hyperparameters of each convolutional layer. Similarly, Figure 1c The number of convolutional layers in the initial feature descriptor branch network and the hyperparameters of each convolutional layer shown in are for exemplary purposes only and do not constitute a limitation thereto.

[0122] In an embodiment of the present application, the initial feature extraction network corresponds to the first fully convolutional network model, the initial motion attribute branch network corresponds to the second fully convolutional network model, and the initial feature descriptor branch network corresponds to the third fully convolutional network model. Based on this, according to the sample image, the initial feature extraction network, the initial motion attribute branch network, and the initial feature descriptor branch network are jointly trained to obtain the first fully convolutional network model, the second fully convolutional network model, and the third fully convolutional network model. Among them, the sample image can be one frame or multiple frames, and multiple frames refer to 2 frames or more. The specific value of the number can be flexibly set according to actual needs.

[0123] Furthermore, during the training process of the initial motion attribute branch network, the motion attribute true value map of the sample image can be obtained. Figure 1cIn the motion attribute truth map shown, the "still" attribute shown is a static attribute. Optionally, the motion attribute truth map can be obtained by semantically analyzing the sample image. Further, the goal of the motion attribute branch network training is to make the motion attributes of at least a part of the pixels in the sample image determined by the motion attribute branch network as consistent as possible with the motion attributes in the motion attribute truth map corresponding to the sample image. Since the motion attributes of at least a part of the pixels in the motion attribute truth map are determined, the true value of the motion attribute corresponding to the pixel in the motion sequence truth map is 1. Therefore, the goal of the motion attribute branch network training can also be that the motion attribute corresponding to the maximum probability value of at least a part of the pixels in the sample image calculated by the motion attribute branch network is the same as the motion attribute determined in the motion attribute truth image, and the maximum probability value is as close to 1 as possible.

[0124] Based on the above analysis, for the motion attribute branch network, the cross entropy of the motion attribute branch network can be used as the loss function. The cross entropy loss function can be expressed as:

[0125] L M =-∑ i,j y ij log(p ij ) (1)

[0126] Among them, p ij represents the probability value of at least a portion of the pixels in the sample image under each motion attribute calculated by the motion attribute branch network; i represents the position of at least a portion of the pixels in the sample image; j represents the motion attribute number. Furthermore, the coefficient y of the cross entropy function ij It is determined based on the true value map of the motion attribute corresponding to the sample image. Specifically, according to the true value map of the motion attribute of the sample image, if and only if the motion attribute of pixel i in the sample image is j, y ij =1, otherwise y ij = 0, that is, for each pixel in the sample image, only when the motion attribute of the pixel is the same as that of the corresponding pixel in the motion attribute true value image, y ij =1. For example, suppose the motion attribute truth map of the sample image determines that the motion attribute of a certain pixel in the sample image is a static pixel; the probability values ​​of the pixel under each motion attribute (static, unstable, moving and uncertain) calculated by the motion attribute branch network are 0.92, 0.51, 0.83 and 0.45 respectively; then only the coefficient y corresponding to 0.92 ij=1; the coefficients corresponding to 0.51, 0.83 and 0.45 are all 0. It should be noted that the pixel points in the motion attribute truth map corresponding to each pixel point in the sample image refer to the pixel points whose position coordinates in the motion attribute truth map are the same as the position coordinates of at least a part of the pixel points in the sample image in the sample image.

[0127] When training the feature descriptor branch network, a teacher-student training strategy is adopted. This strategy uses the previous model for calculating local feature descriptors separately as the teacher network. For the sake of convenience, the teacher network that calculates local feature descriptors is referred to as the feature descriptor teacher network. In this embodiment, the feature descriptor branch network is used as the student network. By training the student network, the result of the student network is close to the feature descriptor teacher network, so that the student network obtains a local feature descriptor calculation capability similar to that of the teacher network. Optionally, a HardNet network can be used as the feature descriptor teacher network. Since the input image of the feature descriptor teacher network is the entire sample image, the student network (the third fully convolutional network model) trained with this network as the teacher network can also be directly applied to the feature descriptor calculation of the entire image. Since the entire image has the characteristic of sharing feature calculation results, it helps to improve computational efficiency.

[0128] Furthermore, in order to make the feature descriptor branch network close to the feature descriptor teacher network, the mean square error loss function can be used to train the feature descriptor branch network to obtain the third fully convolutional network model. The mean square error loss function can be expressed as:

[0129]

[0130] Where k represents the feature dimension index, K represents the total number of feature dimensions, and N represents the total number of pixels in the sample image; d ik represents the output of the feature descriptor branch network, Represents the output of the feature descriptor teacher network.

[0131] Further, in the embodiment of the present application, in order to jointly train the initial feature extraction network, the initial motion attribute branch network, and the initial feature descriptor branch network, the cross entropy function of the initial motion attribute branch network and the mean square error function of the feature descriptor output by the initial feature descriptor branch network model and the feature descriptor output by the feature descriptor teacher network can be determined. Specifically, the sum of the above cross entropy loss function and the mean square error loss function can be used as the joint loss function. Among them, the joint loss function can be expressed as:

[0132] L=L M +L D (3)

[0133] Furthermore, the initial feature extraction network, the initial motion attribute branch network, and the initial feature descriptor branch network can be jointly trained by minimizing the joint loss function to obtain the first full convolutional network model, the second full convolutional network model, and the third full convolutional network model. The main training process is: using the initial feature extraction network, the sample image is converted into a sample feature map; and the minimization of the joint loss function is used as the training goal, and the sample feature map is respectively input into the initial motion attribute branch network and the initial feature descriptor branch network for model training, and the first full convolutional network model, the second full convolutional network model, and the third full convolutional network model are obtained.

[0134] In order to explain the above joint training process more clearly, Figure 1d The joint training process provided in this embodiment is exemplified. The main steps of the joint training process are as follows:

[0135] S1: The sample image is used as the input image of the initial feature extraction network and the feature descriptor teacher network, and is input into the initial feature extraction network and the feature descriptor teacher network respectively.

[0136] S2: Use the initial feature extraction network to convert the sample image into a sample feature map.

[0137] S3: Calculate feature descriptors of at least a portion of pixels of the sample image using the feature descriptor teacher network.

[0138] S4: The sample feature map is used as the input image of the initial motion attribute branch network and the initial feature descriptor branch network, and is input into the initial motion attribute branch network and the initial feature descriptor branch network respectively.

[0139] S5: Using the initial motion attribute branch network, calculate the probability value of at least a portion of the pixels in the sample image under each motion attribute.

[0140] S6: Calculate feature descriptors of at least a portion of pixels in the sample image using the initial feature descriptor branch network.

[0141] S7: The probability value of at least a part of the pixels in the sample image under each motion attribute, the feature descriptors of at least a part of the pixels in the sample image calculated by the initial feature descriptor branch network, and the feature descriptors of at least a part of the pixels in the sample image calculated by the feature descriptor teacher network are respectively brought into the above-mentioned joint loss function to calculate the value of the joint loss function.

[0142] S8: Calculate the change range between the joint loss function value calculated this time and the joint function value calculated W times most recently. Wherein, W is an integer greater than or equal to 2, and its specific value can be flexibly configured. For example, W can be equal to 5, 8, 10, etc., but is not limited thereto.

[0143] S9: Determine whether the calculated change amplitude is less than or equal to a preset change amplitude threshold. If the determination result is yes, execute step S11; if the determination result is no, execute step S10.

[0144] S10: Adjust the parameters in at least a part of the network along the negative gradient direction of at least a part of the parameters of the initial feature extraction network, the initial motion attribute branch network and the initial feature descriptor branch network, and use the adjusted three networks as the initial feature extraction network, the initial motion attribute branch network and the initial feature descriptor branch network, respectively, and return to execute step S1.

[0145] S11: Use the current feature extraction network, the current motion attribute branch network, and the current feature descriptor branch network as the first full convolutional network model, the second full convolutional network model, and the third full convolutional network model, respectively.

[0146] It is worth noting that, in this embodiment, steps S2 and S3 can be executed in parallel or in sequence. When steps S2 and S3 are executed in sequence, the order in which they are executed is not limited. Similarly, steps S4 and S5 can be executed in parallel or in sequence. When steps S4 and S5 are executed in sequence, the order in which they are executed is not limited.

[0147] It is also worth mentioning that the visual positioning method provided in this embodiment can be applied to a variety of application scenarios. In some application scenarios, there may be multiple autonomous mobile devices at the same time. Combined with the positions located by the multiple autonomous mobile devices, the multiple autonomous mobile devices can work together. For example, in the warehousing scenarios of some companies, multiple sorting robots are required to perform sorting operations. Each sorting robot can use the method provided in the embodiment of the present application to locate the current position of the locator and send its current position to the server device. The server device can dispatch the sorting robot according to the current position of each sorting robot and the distribution of shelves in the storage space. For another example, in some cleaning scenarios, in order to complete the cleaning operation as soon as possible and improve the cleaning efficiency, multiple cleaning robots can be placed in the cleaning space. During the cleaning process, each cleaning robot can use the visual positioning method provided in this embodiment to perform autonomous positioning and upload its own position to the server device. The server device can then determine the uncleaned areas in the cleaning space based on the position information reported by each cleaning robot and the environmental map of the cleaning space, and dispatch the cleaning robot to the uncleaned areas to perform cleaning tasks, etc., but not limited to this.

[0148] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned visual positioning method.

[0149] In the above embodiments of the present application, autonomous mobile devices are used as an example to describe in detail the process of the vision-based positioning method of the present application, but it is not limited to autonomous mobile devices. Any device that needs to be positioned during movement can be positioned using the method provided in the embodiments of the present application, such as smart phones, smart wearable devices, non-unmanned vehicles, etc.

[0150] Based on this, an embodiment of the present application also provides a positioning method applicable to any mobile device. The method includes:

[0151] Step 1: During the movement, collect the environment image around the current location.

[0152] Step 2: Analyze the motion properties of at least a portion of the pixels in the environment image using a neural network model.

[0153] Step 3: Determine static pixels in at least a portion of the pixels according to the motion attributes of at least a portion of the pixels.

[0154] Step 4: Locate the current location of the mobile device based on the static pixel points and the known environment map.

[0155] In this embodiment, the mobile device may be an autonomous mobile device such as a robot or an unmanned vehicle, or a non-autonomous mobile device such as a smart phone, a wearable device, or a vehicle that requires a person to drive. In this embodiment, the mobile device is provided with a positioning button, or a human-computer interaction interface is provided, and a positioning control is displayed on the interface. The user may trigger the positioning button on the mobile device or the positioning control on the human-computer interaction interface, and the mobile device responds to the user's positioning trigger operation and starts positioning. The user's positioning trigger operation may be performed at any time, for example, during the movement of the mobile device, or when the mobile device stops moving.

[0156] Accordingly, a visual sensor is provided on the mobile device. In step 1, the visual sensor can collect an environmental image around the current location of the mobile device while the mobile device is moving. The visual sensor can be a camera, such as a binocular camera, a monocular camera, or a depth of field camera, but is not limited thereto. Then, in step 2, a neural network model is used to analyze the motion properties of at least a portion of the pixels in the environmental image collected in step 1. For the description of at least a portion of the pixels and the motion properties in the environmental image, please refer to the relevant content of the above-mentioned embodiment, which will not be repeated here.

[0157] Further, in step 3, static pixels in at least a portion of pixels in the environment image may be determined based on the motion attributes of the at least a portion of pixels, and in step 4, the current location of the mobile device may be located based on the static pixels and the known environment map. Optionally, the current location of the mobile device may be represented by the position coordinates corresponding to the current location in the environment map, but is not limited thereto.

[0158] In this embodiment, the mobile device collects the environment image around its location during the movement process, and uses the neural network model to analyze the motion attributes of at least a part of the pixels in the environment image collected; then, based on the motion attributes of at least a part of the pixels, autonomous positioning is performed using static pixels. Static pixels are feature points whose positions are fixed or substantially unchanged in the environment where the mobile device is currently located. Compared with moving feature points, using static pixels is more conducive to positioning and helps to improve positioning accuracy.

[0159] It is worth noting that the specific implementation methods of the above steps 1-4 can all refer to the relevant content of the above-mentioned method based on visual positioning, which will not be repeated here.

[0160] In addition to locating the current location of the mobile device, the visual positioning method provided in the embodiment of the present application can also calculate the mobile device's travel status information. The mobile device's travel status information includes: at least one of the mobile device's travel speed, travel direction, and acceleration, but is not limited thereto.

[0161] Based on this, an embodiment of the present application further provides a data processing method. The method is applicable to a mobile device, and the method includes:

[0162] S11: During the movement, collect environmental images around the current location.

[0163] S12: Analyze motion properties of at least a portion of pixels in the environment image using a neural network model.

[0164] S13: Determine static pixels among at least a portion of the pixels according to the motion attributes of at least a portion of the pixels.

[0165] S14: Locate the current location of the mobile device based on the static pixel points and the known environment map.

[0166] S15: Calculate the travel status information of the mobile device according to the current location of the mobile device and at least one previous historical location.

[0167] In this embodiment, the detailed description and specific implementation methods of steps S11 to S14 can refer to the relevant contents of the above embodiments, which will not be repeated here.

[0168] In this embodiment, the mobile device can collect environmental images around the user's location during its movement, and use a neural network model to analyze the motion attributes of at least a portion of the pixels in the environmental images collected; then, based on the motion attributes of at least a portion of the pixels, autonomous positioning is performed using static pixel pairs. Static pixel points are feature points whose positions are fixed or substantially unchanged in the environment where the autonomous mobile device is currently located. Compared with moving feature points, using static pixel points is more conducive to positioning and helps to improve positioning accuracy.

[0169] Furthermore, since the current location and historical location of the mobile device obtained in this embodiment are of high accuracy, in this embodiment, the travel status information of the mobile device can be calculated based on the current location of the mobile device and at least one previous historical location. This can reduce the number of sensors on the mobile device. For example, there is no need to set up speed sensors, acceleration sensors, etc., which helps to reduce the hardware cost of the mobile device.

[0170] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned data processing method.

[0171] In addition to the above-mentioned method for visual positioning of a mobile terminal device, an embodiment of the present application also provides a visual positioning method suitable for a terminal device. Among them, the terminal device can be a portable terminal device, such as a smart phone, a tablet computer, a smart wearable device, etc., but not limited to this. Among them, the smart wearable device can be an electronic watch, an electronic bracelet, smart glasses, an anti-lost device, etc., but not limited to this. In this embodiment, the terminal device is a terminal device carried or worn by a user, and it moves with the movement of the user. The following is an exemplary description of a visual positioning method suitable for a terminal device provided in an embodiment of the present application. The method mainly includes:

[0172] S21: When the user moves, an image of the environment around the current location of the user is collected, and the user carries a terminal device.

[0173] S22: Analyze the motion properties of at least a portion of the pixels in the environment image using a neural network model.

[0174] S23: Determine static pixels among at least a portion of the pixels according to the motion attributes of at least a portion of the pixels.

[0175] S24: Locate the current location of the user based on the static pixel points and the known environment map.

[0176] S25: Output the user's current location to guide the user to move forward.

[0177] In this embodiment, the detailed description and specific implementation methods of the contents of steps S21 to S24 can refer to the relevant contents of the above embodiments, which will not be repeated here.

[0178] In this embodiment, the user carries a terminal device, which is provided with a camera. The terminal device can collect environmental images around the user's location during the user's movement, and use a neural network model to analyze the motion properties of at least a portion of the pixels in the environmental image. Then, based on the motion properties of at least a portion of the pixels, the user is located using static pixels, and the user's location information is output to guide the user. Static pixels are feature points whose positions are fixed or substantially unchanged in the surrounding environment where the autonomous mobile device is currently located. Compared with feature points in motion, using static pixels is more conducive to positioning, helps to improve positioning accuracy, and then helps to improve user navigation accuracy, thereby improving user experience.

[0179] In one embodiment, the terminal device includes a display. Based on this, the terminal device can display the current position of the user on the display to guide the user to proceed.

[0180] In another embodiment, the terminal device includes an audio component. Based on this, the terminal device can use the audio component to output the user's current location in voice to guide the user. This implementation method can help visually impaired users understand their current location, has high universality, and can further improve user experience.

[0181] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned visual positioning method.

[0182] In addition to the visual positioning method provided in the above embodiment, the embodiment of the present application also provides an image processing method, which is exemplarily described below.

[0183] Figure 1e A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application. Figure 1e As shown, the method includes:

[0184] 201. Obtain an image to be processed.

[0185] 202. Use the first network model to perform feature extraction on the image to be processed to obtain multiple initial feature maps.

[0186] 203. Calculate motion attributes of the multiple initial feature maps using the second network model to obtain motion attributes of at least a portion of pixels in the image to be processed.

[0187] 204. Calculate feature descriptors for the multiple initial feature maps using the third network model to obtain feature descriptors for at least a portion of the pixels in the image to be processed.

[0188] In this embodiment, the image to be processed in step 201 may be an image collected by an image processing device, or may be an image obtained by other means. For example, the image to be processed may be an image sent by other devices, or may be an image read from a storage medium, but is not limited thereto. Further, in step 202, the first network model may be used to extract features from the image to be processed to obtain multiple feature maps. Among them, multiple refers to 2 or more, and its specific value is determined by the number of output feature maps of the last convolutional layer of the first network model. Among them, the number of output feature maps of the last convolutional layer of the first network model can be flexibly set according to application requirements. Further, for the convenience of description and distinction, the feature map output by the first network model is defined as an initial feature map. Then, in step 203, the second network model may be used to calculate the motion attributes of the multiple initial feature maps output by the first network model to obtain the motion attributes of at least a portion of the pixels in the environment image. And in step 204, the third network model is used to calculate the feature descriptors of the multiple initial feature maps to obtain the feature descriptors of at least a portion of the pixels in the image to be processed.

[0189] In this embodiment, at least a portion of the pixels in the image to be processed may be a portion of the pixels in the image to be processed, or may be all the pixels in the image to be processed. If at least a portion of the pixels in the image to be processed are a portion of the pixels in the image to be processed, the selection of the portion of pixels may be flexibly set, and preferably, at least a portion of the pixels are key points in the image to be processed.

[0190] In this embodiment, the motion attribute branch network (second network model) and the feature descriptor branch network (third network model) share a basic network (first network model), realizing the reuse of basic network calculations. When calculating the motion attributes and feature descriptors of at least a part of the pixels in the image to be processed, the basic network reuse can be realized, that is, the calculation reuse can be realized, thereby reducing the amount of calculation and helping to improve the image processing efficiency.

[0191] In this embodiment, the first network model, the second network model, and the third network model may all adopt any form of neural network model, such as a convolutional neural network model, but not limited thereto. Furthermore, the first network model, the second network model, and the third network model may all adopt a full convolutional network model. For ease of description and distinction, the full convolutional network models adopted by the first network model, the second network model, and the third network model are defined as the first full convolutional network model, the second full convolutional network model, and the third full convolutional network model, respectively. Taking the first full convolutional network model, the second full convolutional network model, and the third full convolutional network model as examples, some specific implementation methods of the above steps 202-204 are exemplified below.

[0192] In this embodiment, for the convenience of distinction and description, the number of multiple initial feature maps output by the first fully convolutional network model is represented by M, where M ≥ 2 and is an integer. The specific value of M is determined by the number F of output feature maps of the last convolutional layer of the first fully convolutional network model. Figure 1b In the example, M=F=128 is used only as an example, but it is not limited thereto.

[0193] Furthermore, the feature map output by the output layer of the second fully convolutional network model is the characteristic embodiment of at least a part of the pixels in the image to be processed under each motion attribute. For the convenience of description and distinction, the feature map output by the second fully convolutional network model is defined as a first target feature map. There are multiple first target feature maps, which is the same as the number of types of motion attributes defined in this embodiment. Among them, each first target feature map is a characteristic embodiment of the image to be processed under a motion attribute, that is, the number of first target feature maps output by the last convolutional layer of the second fully convolutional network is equal to the number of types of motion attributes. In the embodiment of the present application, it is represented by Q and is equal to the number of types of motion attributes, that is, the number of first target feature maps, and Q ≥ 2, and is an integer. For example, Figure 1b As shown in , if the motion attributes include: static, unstable, moving and uncertain, then Q = 4. Based on this, Figure 1b As shown, the above M initial feature maps can be used as input images of the second fully convolutional network model and input into the second fully convolutional network model; in the second fully convolutional network model, the M initial feature maps are convolutionally calculated at least once, and Q first target feature maps can be output; further, the motion attributes of at least a part of the pixels in the image to be processed can be calculated based on the Q first target feature maps. Among them, the specific implementation method of outputting M initial feature maps from the first fully convolutional network model and calculating the motion attributes of at least a part of the pixels in the image to be processed based on the Q first target feature maps in the second fully convolutional network model can be referred to the relevant content in the above visual positioning method, which will not be repeated here.

[0194] For the feature descriptors of at least a portion of the pixels in the above-mentioned image to be processed, such as Figure 1b As shown, the third fully convolutional network model can be used to calculate feature descriptors for the multiple initial feature maps output by the above-mentioned first fully convolutional network model to obtain feature descriptors of at least a part of the pixels in the image to be processed, and obtain feature descriptors of static pixels from the feature descriptors of at least a part of the pixels in the image to be processed. Correspondingly, feature descriptors of key pixels can also be obtained from the feature descriptors of static pixels. Since the input image of the third fully convolutional network model is the multiple initial feature maps output by the first fully convolutional network model, which is the same as the input image of the second fully convolutional network model, the motion attributes of at least a part of the pixels in the image to be processed and the feature descriptors of these pixels can be determined by sharing the same basic network (i.e., the first fully convolutional network model) for feature extraction, which realizes computational reuse and reduces the amount of computation, thereby helping to improve the computational rate and further improve the positioning efficiency.

[0195] Furthermore, the above-mentioned M initial feature maps can be used as input images of the third fully convolutional network model and input into the third fully convolutional network model; in the third fully convolutional network model, the M initial feature maps are subjected to at least one convolution process to obtain N feature maps. Among them, for the convenience of description and distinction, the feature map output by the third fully convolutional network model is defined as the second target feature map. Among them, each second target feature map is a feature description of at least a part of the pixels in the environment image; in this way, the N second target feature maps also form a feature descriptor of at least a part of the pixels in the environment image. In this embodiment of the present application, N is an integer greater than or equal to 2, and its specific value is determined by the number F of feature maps set by the last convolutional layer in the third fully convolutional network model. Figure 1b In the example, N=F=128 is used for illustration and is not limited thereto. In the third fully convolutional network model, the specific implementation of obtaining N second target feature maps can be found in the relevant content of the above-mentioned visual positioning embodiment, which will not be described in detail here.

[0196] It is worth noting that in the embodiment of the present application, before using the first fully convolutional network, the second fully convolutional network and the third fully convolutional network for image processing, the first fully convolutional network, the second fully convolutional network and the third fully convolutional network can also be jointly trained, and the joint training process can refer to the relevant content of the above embodiment, which will not be repeated here.

[0197] The above-mentioned image processing method has different uses in different application scenarios. For example, in an application scenario based on visual positioning, after step 204, static pixels in at least a portion of the pixels in the image to be processed can be determined based on the motion attributes of the at least a portion of the pixels in the image to be processed. Then, based on the static pixels and the known environment map, the position of the device that collects the image to be processed during the process of collecting the image to be processed is located.

[0198] Further, the feature descriptors of static pixels can be obtained from the feature descriptors of at least a portion of the pixels calculated in step 204; and based on the feature descriptors of the static pixels and the feature descriptors of each position point recorded in the known environment map, the position of the device that collects the image to be processed during the process of collecting the image to be processed is located. Among them, the specific implementation method of locating the position of the device that collects the image to be processed during the process of collecting the image to be processed can refer to the relevant content of the above-mentioned visual positioning method, which will not be repeated here.

[0199] It is worth noting that the image processing method provided in this embodiment can be applied to any computer device, such as a terminal device (computer, smart phone, wearable device, etc.), a server or server array in the cloud, etc. Of course, it can also be applied to edge computing devices in an edge cloud network system. For example, edge computing devices may include: base stations, home gateways, personal computers, smart phones, street lights, traffic lights, and / or electronic monitoring devices on buildings, etc., which are added to the edge cloud network.

[0200] Combination Figure 2a The edge cloud network system shown in the figure is used as an example to illustrate the image processing method provided in this embodiment. Figure 2a In the example, the device for collecting the image to be processed is taken as a terminal device carried by the user, and Figure 2a Only the terminal device carried by the user is shown as the smart glasses worn by the user. Figure 2a In the edge cloud network system, the edge computing device and the server deployed in the cloud or in the customer's room are included. The server and the edge computing device communicate through the network. The server can respond to the request of the edge computing device and provide relevant cloud services for the edge computing device. In addition, the server can also manage, operate and maintain the edge computing device. The edge computing device includes hardware infrastructure, drivers for the hardware infrastructure, operating systems and related applications. The hardware infrastructure includes but is not limited to: CPU, network card and memory.

[0201] In this embodiment, the terminal device includes a camera, and a neural network model including a two-layer structure is provided inside the edge computing device. Figure 2aAs shown, the terminal device carried by the user collects the environment image around the user's current location and sends the environment image to the edge computing device in the current surrounding environment. The edge computing device receives the environment image and inputs the environment image into a two-layer neural network model. In the two-layer neural network model, the first layer network can extract features from the environment image to obtain multiple initial feature maps; the multiple initial feature maps are sent to the second layer network, and the second layer network calculates the motion attributes of the multiple initial feature maps to obtain the motion attributes of at least a part of the pixels in the environment image. Then, the edge computing device can determine the static pixels in the at least part of the pixels according to the motion attributes of at least a part of the pixels in the environment image, and locate the user's current location according to the feature descriptors of the static pixels and the known environment map. Among them, the specific implementation method of the edge computing device locating the user's current location according to the feature descriptors of the static pixels and the known environment map can refer to the relevant content of the above embodiment, which will not be repeated here. Further, the edge computing device can send the user's current location to the terminal device carried by the user, and the terminal device receives the user's current location and outputs the user's current location to guide the user to move.

[0202] It is worth noting that the "smart glasses worn by the user" in this embodiment can also be replaced by autonomous mobile devices such as robots and drones in other embodiments, or non-autonomous mobile devices such as cars driven by users and bicycles ridden by users. This embodiment only takes "smart glasses worn by users" as an example for explanation.

[0203] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned image processing method.

[0204] Figure 2b A flowchart of another image processing method provided in an embodiment of the present application is shown below. Figure 2b As shown, the method includes:

[0205] S201: Inputting the image to be processed into a neural network model, wherein the neural network model includes at least two layers of structure.

[0206] S202: Using the first layer of the neural network model, extract features from the image to be processed to obtain multiple initial feature maps.

[0207] S203: Utilizing the second layer network in the neural network model, motion attributes are calculated for the multiple initial feature maps to obtain motion attributes of at least a portion of the pixels in the image to be processed.

[0208] S204: Determine target pixel points whose position changes are less than or equal to a preset threshold according to the motion attributes of at least a portion of the pixel points.

[0209] S205: According to the target pixel point and the known environment map, the position of the device for collecting the image to be processed is located during the process of collecting the image to be processed.

[0210] In this embodiment, the image to be processed may be an image collected by an image processing device, or may be an image acquired by other means. For example, the image to be processed may be an image sent by other devices, or may be an image read from a storage medium, but is not limited thereto.

[0211] In this embodiment, the neural network model includes at least two layers, and the number of layers of the network can be flexibly set according to actual functional requirements. In this embodiment, the first layer of the network can extract features from the image to be processed to obtain multiple feature maps. Among them, multiple refers to 2 or more, and its specific value is determined by the number of output feature maps of the last convolutional layer of the first layer of the network. For the convenience of description and distinction, the feature map output by the first network model is defined as the initial feature map. Further, the second layer of the network can calculate the motion attributes of multiple initial feature maps to obtain the motion attributes of at least a part of the pixels in the image to be processed. Among them, the description of at least a part of the pixels can refer to the relevant content of the above embodiment, which will not be repeated here. Then, according to the motion attributes of at least a part of the pixels in the image to be processed, the target pixel whose position change is less than or equal to the preset threshold value in the at least a part of the pixels can be determined. Among them, the preset threshold value can be flexibly set according to actual needs and is not limited here. Optionally, the target pixel point can also be a feature point with a fixed position. Further, according to the target pixel point and the known environment map, the position of the device for collecting the image to be processed during the process of collecting the image to be processed can be located.

[0212] In this embodiment, a neural network model is used to analyze the motion properties of at least a portion of the pixels in the image to be processed; further, based on the motion properties of at least a portion of the pixels, the target pixels whose position changes are less than or equal to a preset change threshold are used to locate the position of the device that collects the image to be processed during the process of collecting the image to be processed. The target pixel is a feature point whose position is fixed or substantially unchanged in the surrounding environment of the device that collects the image to be processed during the process of collecting the image to be processed. Compared with the moving feature points, the use of the target pixel is more conducive to positioning and helps to improve the positioning accuracy.

[0213] In the embodiment of the present application, the neural network model may adopt any form of neural network model, such as a convolutional neural network model, but is not limited thereto. Further, the neural network model may adopt a full convolutional network model, that is, both the first layer network and the second layer network are full convolutional networks. Among them, if both the first layer network and the second layer network are full convolutional networks, the specific implementation of step S202 and step S203 can refer to the relevant content of the above embodiment, and will not be repeated here.

[0214] Further, in this embodiment, the neural network model may also include a third layer network. Further, the third layer network may be used to calculate feature descriptors for multiple initial feature maps to obtain feature descriptors for at least a portion of pixels in the image to be processed. The third layer network may also be a full convolutional network, and the specific implementation method of calculating feature descriptors for multiple initial feature maps using a full convolutional network may also refer to the relevant contents of the above embodiment, which will not be repeated here.

[0215] Furthermore, the feature descriptor of the target pixel point can be obtained from the feature descriptors of at least a portion of the calculated pixel points; and based on the feature descriptors of the target pixel point and the feature descriptors of each position point recorded in the known environment map, the position of the device that collects the image to be processed during the process of collecting the image to be processed can be located. Among them, the specific implementation method of locating the position of the device that collects the image to be processed during the process of collecting the image to be processed can be referred to the relevant content of the above-mentioned visual positioning method, which will not be repeated here.

[0216] It is worth noting that for the specific implementation method of model training for the neural network model, please refer to the relevant content of the above embodiment, which will not be repeated here.

[0217] Combination Figure 2c A scene schematic diagram is shown in FIG. 1 to exemplarily illustrate the image processing method provided in this embodiment. Figure 2c In the example, the mobile device is taken as a robot and the environment in which the robot is located is taken as an outdoor environment. Figure 2c The implementation form of the robot and the environment in which it is located are only exemplary and not limiting. In this embodiment, the robot can move autonomously and can achieve autonomous positioning during the movement. Figure 2cAs shown, the robot includes a camera, and a neural network model including a two-layer structure is arranged inside the robot. During the movement, the robot uses the camera to collect images of the surrounding environment, and inputs the collected environmental images into the two-layer neural network model. In the two-layer neural network model, the first layer network can extract features of the environmental image to obtain multiple initial feature maps; the multiple initial feature maps are sent to the second layer network, and the second layer network calculates the motion attributes of the multiple initial feature maps to obtain the motion attributes of at least a part of the pixels in the environmental image. Then, the robot can determine the target pixel points whose position change is less than or equal to the preset threshold value in at least a part of the pixels in the environmental image according to the motion attributes of at least a part of the pixels in the environmental image, and locate its current position (x0, y0, z0) according to the target pixel points and the known environmental map, that is, perform autonomous positioning.

[0218] It is worth noting that the image processing method provided in this embodiment can be applied to any computer device, such as terminal devices (computers, smart phones, wearable devices, etc.), servers or server arrays in the cloud, etc. Of course, it can also be applied to edge computing devices in edge cloud network systems. For example, street lights, traffic lights, electronic devices on buildings (cameras, etc.). Among them, if the computer device is an edge computing device in an edge cloud network system, its interaction method with the device that collects the image to be processed and the structure of the edge cloud network system can be referred to above. Figure 2a The relevant contents in the description will not be repeated here. Further, in this embodiment, the edge computing device can determine the target pixel whose position change is less than or equal to a preset threshold value among at least a part of the pixel points in the environmental image according to the motion attributes of at least a part of the pixel points in the environmental image, and locate the current location of the user according to the target pixel points and the known environmental map. Further, the edge computing device can send the current location of the user to the terminal device carried by the user, and the terminal device receives the current location of the user and outputs the current location of the user to guide the user to move.

[0219] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned image processing method.

[0220] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 and 102 can be device A; for another example, the execution subject of step 101 can be device A, and the execution subject of step 102 can be device B; and so on. In addition, in some processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 203, 204, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0221] Figure 3 FIG. 1 is a hardware structure diagram of a mobile device provided in this embodiment. Figure 3 As shown, the mobile device 30 comprises a mechanical body 30a on which a visual sensor 30b, one or more processors 30c, and one or more memories 30d are disposed.

[0222] It is worth noting that one or more processors 30c, one or more memories 30d, and the visual sensor 30b can be disposed inside the mechanical body 30a, or can be disposed on the surface of the mechanical body 30a.

[0223] The mechanical body 30a is the actuator of the mobile device 30, which can execute one or more operations specified by the processor 30c in a certain environment. Among them, the mechanical body 30a reflects the appearance of the autonomous mobile device 30 to a certain extent. In this embodiment, the appearance of the autonomous mobile device 30 is not limited. Optionally, the mobile device can be an autonomous mobile device, such as a robot, an unmanned vehicle or a drone. The mobile device can also be a non-autonomous mobile device such as a smart phone, a wearable device, or a person-driven device, but is not limited thereto. The mechanical body 30a mainly refers to the body of the mobile device 30.

[0224] It is worth noting that some basic components of the mobile device 30 are also arranged on the mechanical body 30a, such as a drive component, an odometer, a power component, an audio component, etc. Optionally, the drive component may include a drive wheel, a drive motor, a universal wheel, etc. These basic components and the composition of the basic components included in different mobile devices may be different, and the embodiments of the present application list only some examples.

[0225] One or more memories 30d are mainly used to store computer instructions, which can be executed by one or more processors 30c, so that one or more processors 30c control the mobile device 30 to implement corresponding functions, complete corresponding actions or tasks. In addition to storing computer instructions, one or more memories 30d can also be configured to store various other data to support operations on the mobile device 30. Examples of these data include instructions for any application or method operating on the mobile device 30, and an environment map corresponding to the environment in which the mobile device 30 is located. Among them, the environment map can be one or more maps corresponding to the entire environment stored in advance, or it can also be a partial map that is being constructed before.

[0226] In this embodiment, the visual sensor 30a can be regarded as the "eye" of the mobile device 30, and is mainly used to collect environmental images around the current location of the mobile device 30 during the movement of the mobile device 30. The visual sensor can be implemented by any device with an image acquisition function, such as a camera, a laser sensor, an infrared sensor, etc. Further, the camera can be a binocular camera, a monocular camera, a depth of field camera, etc., but is not limited thereto.

[0227] One or more processors 30c can be regarded as the control system of the mobile device 30, which can be coupled with one or more memories 30d to execute computer instructions stored in one or more memories 30d to control the mobile device 30 to achieve corresponding functions, complete corresponding actions or tasks. It is worth noting that when the mobile device 30 is in different scenarios, the functions, actions or tasks to be achieved will be different; accordingly, the computer instructions stored in one or more memories 30d will also be different, and one or more processors 30c executing different computer instructions can control the mobile device 30 to achieve different functions, complete different actions or tasks.

[0228] In this embodiment, one or more processors 30c are mainly used to: use a neural network model to analyze the motion properties of at least a part of the pixels in the environment image; determine the static pixels in the environment image based on the motion properties of at least a part of the pixels in the environment image; and locate the current position of the mobile device based on the static pixels and a known environment map.

[0229] In some embodiments, when analyzing the motion attributes of at least a portion of the pixels in an environmental image, one or more processors 30c are specifically used to: use a first network model to extract features from the environmental image to obtain multiple initial feature maps; use a second network model to calculate motion attributes on multiple initial feature maps to obtain motion attributes of at least a portion of the pixels in the environmental image.

[0230] Further, the first network model may be a first fully convolutional network model, and the second network model may be a second fully convolutional network model. Accordingly, when one or more processors 30c use the second network model to calculate the motion attributes of multiple initial feature maps, they are specifically used to: use the M initial feature maps as input images of the second fully convolutional network model, and input them into the second fully convolutional network model; in the second fully convolutional network model, perform at least one convolution calculation on the M initial feature maps to output Q first target feature maps, each of which is a feature embodiment of the environment image under a motion attribute; and calculate the motion attributes of at least a portion of the pixels in the environment image based on the Q first target feature maps; wherein M and Q are integers greater than or equal to 2.

[0231] Furthermore, when one or more processors 30c use the first network model to extract features from the environmental image, they are specifically used to: use the environmental image as the input image of the first fully convolutional network model, and input it into the first fully convolutional network model; in the first fully convolutional network model, perform at least one layer of convolution processing on the environmental image to output M initial feature maps; wherein each layer of convolution processing includes: at least one convolution calculation and downsampling processing after at least one convolution calculation.

[0232] Optionally, when calculating the motion attributes of at least a portion of the pixels in the environmental image, one or more processors 30c are specifically used to: perform dimensionality upscaling on the Q first target feature maps respectively to obtain Q upscaled feature maps with the same dimension as the environmental image; and calculate the motion attributes of at least a portion of the pixels in the environmental image based on the Q upscaled feature maps.

[0233] Furthermore, when one or more processors 30c perform dimensionality upscaling processing on the Q first target feature maps, they are specifically used to: perform bilinear interpolation on the Q first target feature maps respectively according to the dimensional differences between the Q first target feature maps and the environmental image, so as to obtain Q up-scaled feature maps with the same dimension as the environmental image.

[0234] Accordingly, when calculating the motion attributes of at least a part of the pixels in the environmental image, one or more processors 30c are specifically used to: normalize the Q upscaled feature maps respectively to obtain the probability value of each pixel in at least a part of the pixels in the environmental image under the Q types of motion attributes; for each pixel in at least a part of the pixels in the environmental image, select the maximum probability value from the probability values ​​of each pixel under the Q types of motion attributes, and use the motion attribute corresponding to the maximum probability value as the motion attribute of the pixel.

[0235] In other embodiments, when locating the current location of the mobile device, one or more processors 30c are specifically used to: obtain feature descriptors of static pixel points; and locate the current location of the mobile device based on the feature descriptors of the static pixel points and the feature descriptors of each location point recorded in a known environment map.

[0236] Furthermore, when locating the current position of the autonomous mobile device, one or more processors 30c are specifically used to: select static pixel points belonging to the key points from the static pixel points as key pixel points based on the key points in the environmental image; determine the position coordinates of the key pixel points in the environmental map based on the feature descriptors of the key pixel points and the feature descriptors of each position point in the environmental map; and calculate the position coordinates of the autonomous mobile device in the environmental map based on the position coordinates of the key pixel points in the environmental map.

[0237] Furthermore, when obtaining feature descriptors of static pixels, one or more processors 30c are specifically used to: calculate feature descriptors of multiple initial feature maps using a third network model to obtain feature descriptors of at least a portion of pixels in the environmental image; and obtain feature descriptors of static pixels from the feature descriptors of at least a portion of pixels in the environmental image.

[0238] Further, the third full network model may be a third full convolutional network model. Accordingly, when one or more processors 30c use the third network model to calculate feature descriptors for multiple initial feature maps, they are specifically used to: use the M initial feature maps as input images of the third full convolutional network model, and input the third full convolutional network model; in the third full convolutional network model, perform at least one convolution process on the M initial feature maps to obtain N second target feature maps, each of which is a feature description of at least a portion of the pixels in the environment image; wherein the N second target feature maps form a feature descriptor of at least a portion of the pixels in the environment image; wherein N is an integer greater than or equal to 2.

[0239] In some further embodiments, before analyzing the motion attributes of at least a portion of the pixels in the environmental image, one or more processors 30c are also used to: jointly train the initial feature extraction network, the initial motion attribute branch network, and the initial feature descriptor branch network based on the sample image to obtain a first fully convolutional network model, a second fully convolutional network model, and a third fully convolutional network model.

[0240] Furthermore, when one or more processors 30c jointly train the initial feature extraction network, the initial motion attribute branch network, and the initial feature descriptor branch network, they are specifically used to: use the initial feature extraction network to convert the sample image into a sample feature map; take the minimization of the joint loss function as the training goal, and input the sample feature map into the initial motion attribute branch network and the initial feature descriptor branch network for model training, so as to obtain a first full convolutional network model, a second full convolutional network model, and a third full convolutional network model; the joint loss function is determined based on the cross entropy function of the initial motion attribute branch network and the mean square error function of the feature descriptor output by the initial feature descriptor branch network model and the feature descriptor output by the feature descriptor teacher network; wherein the coefficient of the cross entropy function is determined based on the motion attribute true value map corresponding to the sample image.

[0241] Optionally, the feature descriptor teacher network is a Hardnet network.

[0242] In an optional embodiment, the one or more processors 30c are further configured to calculate the travel state information of the mobile device according to the current location of the mobile device and at least one previous historical location, wherein the travel state information includes at least one of the travel direction, travel speed, acceleration, etc.

[0243] In another optional embodiment, the mobile device may be some terminal devices that can be worn by the user and can move with the user, such as a smart phone, a tablet computer, a smart wearable device, etc. In this case, the processor in the terminal device may also be used to locate the current location of the user carrying the terminal device based on static pixel points and a known environment map, and output the current location of the user to guide the user to move.

[0244] Optionally, the terminal device further includes: an audio component. Accordingly, when the processor outputs the current location of the user, it can output the current location of the user in a voice manner through the audio component to guide the user to move forward.

[0245] The mobile device provided in this embodiment can collect environmental images around its location during movement, and use a neural network model to analyze the motion properties of at least a portion of the pixels in the collected environmental images; then, based on the motion properties of at least a portion of the pixels, the autonomous mobile device is positioned using static pixels. Static pixels are feature points whose positions are fixed or substantially unchanged in the environment where the mobile device is currently located. Compared with feature points in motion, using static pixels is more conducive to positioning and helps improve positioning accuracy.

[0246] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 4As shown, the computer device includes: a memory 40a and a processor 40b.

[0247] In this embodiment, the memory 40a is used to store computer instructions.

[0248] The processor 40b is coupled to one or more memories 40a, and is used to execute computer instructions to: obtain an image to be processed; use the first network model to perform feature extraction on the image to be processed to obtain multiple initial feature maps; use the second network model to calculate motion attributes on the multiple initial feature maps to obtain motion attributes of at least a portion of pixels in the image to be processed; use the third fully convolutional network model to calculate feature descriptors on the multiple initial feature maps to obtain feature descriptors of at least a portion of pixels in the image to be processed.

[0249] In some embodiments, the processor 40b is also used to: determine static pixels in at least a portion of the pixels in the image to be processed based on motion properties of at least a portion of the pixels in the image to be processed; and locate the position of the device that collects the image to be processed during the process of collecting the image to be processed based on the static pixels and a known environmental map.

[0250] Furthermore, when the processor 40b locates the position of the device that collects the image to be processed during the process of collecting the image to be processed, it is specifically used to: obtain the feature descriptors of static pixels from the feature descriptors of at least a part of the pixels in the image to be processed; and locate the position of the device that collects the image to be processed during the process of collecting the image to be processed based on the feature descriptors of the static pixels and the feature descriptors of each position point recorded in the known environment map.

[0251] In this embodiment, the first network model, the second network model and the third network model may be a first fully convolutional network model, a second fully convolutional network model and a third fully convolutional network model.

[0252] Correspondingly, when the processor 40b uses the second network model to calculate the motion attributes of multiple initial feature maps, it is specifically used to: use the M initial feature maps as input images of the second fully convolutional network model, and input them into the second fully convolutional network model; in the second fully convolutional network model, perform at least one convolution calculation on the M initial feature maps to output Q first target feature maps, each first target feature map is a feature embodiment of the image to be processed under a motion attribute; calculate the motion attributes of at least a part of the pixels in the image to be processed based on the Q first target feature maps; wherein M and Q are integers greater than or equal to 2.

[0253] Furthermore, when the processor 40b uses the third network model to calculate feature descriptors for multiple initial feature maps, it is specifically used to: use the M initial feature maps as input images of the third fully convolutional network model, and input them into the third fully convolutional network model; in the third fully convolutional network model, perform at least one convolution process on the M initial feature maps to obtain N second target feature maps, each second target feature map is a feature description of at least a part of the pixels in the image to be processed; wherein the N second target feature maps form a feature descriptor of at least a part of the pixels in the image to be processed; wherein N is an integer greater than or equal to 2.

[0254] It is worth noting that, in this embodiment, the specific implementation methods of the processor 40b using the second fully convolutional network model to calculate the motion attributes of the processed image, using the third fully convolutional network model to calculate the feature descriptor, and training the first fully convolutional network model, the second fully convolutional network model and the third fully convolutional network model can all be referred to the relevant contents in the above embodiments and will not be repeated here.

[0255] In some optional embodiments, such as Figure 4 As shown, the computer device may further include optional components such as a communication component 40c, a power component 40d, an audio component 40e, a display 40f or a visual sensor 40g. Figure 4 The components are shown only schematically, and do not mean that the computer equipment must include Figure 4 The components shown do not necessarily mean that the computer equipment can only include Figure 4 Components shown.

[0256] In some embodiments, the computer device may be an edge computing device in an edge cloud network system. For an implementation structure of an edge cloud network system, see Figure 2a shown.

[0257] The computer device provided in this embodiment can enable the motion attribute branch network (second network model) and the feature descriptor branch network (third network model) to share a basic network (first network model), thereby realizing the reuse of basic network calculations. When calculating the motion attributes and feature descriptors of each pixel in the image to be processed, the basic network can be reused, that is, the calculation is reused, thereby reducing the amount of calculation and helping to improve image processing efficiency.

[0258] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 5 As shown, the computer device includes: a memory 50a and a processor 50b.

[0259] In this embodiment, the memory 50a is used to store computer instructions.

[0260] The processor 50b is coupled to one or more memories 50a, and is used to execute computer instructions to: input the image to be processed into a neural network model, wherein the neural network model includes at least two layers of structure; utilize the first layer of the neural network model to extract features of the image to be processed to obtain multiple initial feature maps; utilize the second layer of the neural network model to calculate motion attributes of the multiple initial feature maps to obtain motion attributes of at least a portion of the pixels in the image to be processed; determine target pixels whose position changes are less than or equal to a preset threshold value based on the motion attributes of at least a portion of the pixels; and locate the position of the device that collects the image to be processed during the process of collecting the image to be processed based on the target pixels and a known environment map.

[0261] In some optional embodiments, such as Figure 5 As shown, the computer device may also include optional components such as a communication component 50c, a power component 50d, an audio component 50e, a display 50f or a visual sensor 50g. Figure 5 The components are shown only schematically, and do not mean that the computer equipment must include Figure 5 The components shown do not necessarily mean that the computer equipment can only include Figure 5 Components shown.

[0262] In some embodiments, the computer device may be an edge computing device in an edge cloud network system. For an implementation structure of an edge cloud network system, see Figure 2a shown.

[0263] The computer device provided in this embodiment can use a neural network model to analyze the motion attributes of at least a portion of the pixels in the image to be processed; further, based on the motion attributes of at least a portion of the pixels, the target pixel whose position change is less than or equal to a preset change threshold is used to locate the position of the device that collects the image to be processed during the process of collecting the image to be processed. The target pixel is a feature point whose position is fixed or substantially unchanged in the surrounding environment where the device that collects the image to be processed is located during the process of collecting the image to be processed. Compared with the moving feature point, the use of the target pixel is more conducive to positioning and helps to improve the positioning accuracy.

[0264] In an embodiment of the present application, the memory is used to store a computer program and can be configured to store various other data to support operations on the device where it is located. Among them, the processor can execute the computer program stored in the memory to implement the corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0265] In an embodiment of the present application, the communication component is configured to facilitate wired or wireless communication between the device in which it is located and other devices. The device in which the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G or other communication standards that will appear in the future, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can also be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology or other technologies.

[0266] In an embodiment of the present application, the display may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0267] In an embodiment of the present application, a power supply component is configured to provide power to various components of the device in which it is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0268] In an embodiment of the present application, the audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal. For example, for an electronic device with a language interaction function, voice interaction with a user can be achieved through an audio component.

[0269] It should be noted that the descriptions such as “first” and “second” in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit “first” and “second” to different types.

[0270] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0271] The present invention is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0272] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0273] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0274] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0275] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0276] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0277] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0278] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A visual positioning method, applicable to a mobile device, characterized in that: The method comprises: During the movement, collect environmental images around the current location; Analyzing motion properties of at least a portion of pixels in the environment image using a neural network model; Determining static pixels among the at least a portion of pixels according to the motion attributes of the at least a portion of pixels; Acquire the feature descriptor of the static pixel point; locate the current location of the mobile device according to the feature descriptor of the static pixel point and the feature descriptor of each position point recorded in the known environment map.

2. The method according to claim 1, characterized in that The using of a neural network model to analyze the motion attributes of at least a portion of pixels in the environment image includes: Using the first network model to extract features from the environment image to obtain a plurality of initial feature maps; The motion attributes of the multiple initial feature maps are calculated using the second network model to obtain the motion attributes of at least a portion of the pixels in the environment image.

3. The method according to claim 2, characterized in that The first network model is a first fully convolutional network model; the second network model is a second fully convolutional network model.

4. The method according to claim 3, characterized in that: The step of calculating the motion attributes of the plurality of initial feature maps using the second network model to obtain the motion attributes of at least a portion of pixels in the environment image includes: Using the M initial feature maps as input images of the second fully convolutional network model, and inputting them into the second fully convolutional network model; In the second fully convolutional network model, at least one convolution calculation is performed on the M initial feature maps to output Q first target feature maps, each of which is a feature embodiment of the environment image under a motion attribute; Calculating motion attributes of at least a portion of pixels in the environment image according to the Q first target feature maps; Wherein, M and Q are integers greater than or equal to 2.

5. The method according to claim 4, characterized in that The first network model is used to extract features from the environment image to obtain a plurality of initial feature maps, including: Using the environment image as an input image of the first fully convolutional network model, and inputting it into the first fully convolutional network model; In the first fully convolutional network model, performing at least one layer of convolution processing on the environment image to output the M initial feature maps; Each layer of convolution processing includes: at least one convolution calculation and downsampling processing after the at least one convolution calculation.

6. The method according to claim 5, characterized in that The calculating the motion attributes of at least a portion of pixels in the environment image according to the Q first target feature maps includes: Performing dimensionality upscaling processing on the Q first target feature maps respectively to obtain Q dimensionality upscaling feature maps with the same dimension as the environment image; The motion attributes of at least a portion of the pixels in the environment image are calculated based on the Q dimensionally upgraded feature maps.

7. The method according to claim 6, characterized in that The step of performing dimension-upgrading processing on the Q first target feature maps to obtain Q dimension-upgraded feature maps having the same dimension as the environment image includes: Bilinear interpolation is performed on the Q first target feature maps according to the dimensional differences between the Q first target feature maps and the environment image to obtain Q upscaled feature maps with the same dimension as the environment image.

8. The method according to claim 6, characterized in that The calculating the motion attributes of at least a portion of pixels in the environment image according to the Q dimensionally upgraded feature maps includes: Normalizing the Q dimensionally upgraded feature maps respectively to obtain a probability value of each pixel in the at least a portion of the pixels under the Q motion attributes; For each pixel in the at least part of the pixels, a maximum probability value is selected from the probability values ​​of each pixel under Q motion attributes, and the motion attribute corresponding to the maximum probability value is used as the motion attribute of the pixel.

9. The method according to claim 1, characterized in that: The locating the current location of the mobile device according to the feature descriptor of the static pixel point and the feature descriptor of each location point recorded in the known environment map includes: According to the key point in the environment image, a static pixel point belonging to the key point is selected from the static pixel points as a key pixel point; Determining the position coordinates of the key pixel point in the environment map according to the feature descriptor of the key pixel point and the feature descriptors of each position point in the environment map; The position coordinates of the mobile device in the environment map are calculated according to the position coordinates of the key pixel points in the environment map.

10. The method according to claim 4, characterized in that The obtaining of the feature descriptor of the static pixel point comprises: Calculating feature descriptors on the multiple initial feature maps using a third network model to obtain feature descriptors of at least a portion of pixel points in the environment image; The feature descriptor of the static pixel is obtained from the feature descriptors of at least a portion of the pixel points in the environment image.

11. The method according to claim 10, characterized in that The third network model is a third fully convolutional network model.

12. The method according to claim 11, characterized in that The calculating feature descriptors of the plurality of initial feature maps by using the third network model to obtain feature descriptors of at least a portion of pixels in the environment image includes: Using the M initial feature maps as input images of the third fully convolutional network model, and inputting them into the third fully convolutional network model; In the third fully convolutional network model, the M initial feature maps are convolved at least once to obtain N second target feature maps, each of which is a feature description of the at least a portion of the pixels; Wherein, the N second target feature maps form a feature descriptor of the at least a portion of pixels; Wherein, N is an integer greater than or equal to 2.

13. The method according to claim 11, characterized in that Before analyzing the motion attributes of at least a portion of the pixels in the environment image, the method further includes: According to the sample image, the initial feature extraction network, the initial motion attribute branch network and the initial feature descriptor branch network are jointly trained to obtain the first full convolutional network model, the second full convolutional network model and the third full convolutional network model.

14. The method according to claim 13, characterized in that The joint training of the initial feature extraction network, the initial motion attribute branch network and the initial feature descriptor branch network according to the sample image includes: Using the initial feature extraction network, the sample image is converted into a sample feature map; Taking minimization of the joint loss function as a training objective, inputting the sample feature map into the initial motion attribute branch network and the initial feature descriptor branch network for model training, respectively, to obtain the first full convolutional network model, the second full convolutional network model, and the third full convolutional network model; The joint loss function is determined based on the cross entropy function of the initial motion attribute branch network and the mean square error function of the feature descriptor output by the initial feature descriptor branch network model and the feature descriptor output by the feature descriptor teacher network; wherein the coefficient of the cross entropy function is determined based on the motion attribute true value map corresponding to the sample image.

15. The method according to claim 14, characterized in that The feature descriptor teacher network is a Hardnet network.

16. An image processing method, characterized in that: include: Get the image to be processed; Using the first network model to extract features from the image to be processed to obtain a plurality of initial feature maps; Calculating motion attributes of the multiple initial feature maps using a second network model to obtain motion attributes of at least a portion of pixels in the image to be processed; Using a third network model to calculate feature descriptors for the multiple initial feature maps, so as to obtain feature descriptors of at least a portion of pixels in the image to be processed; Determining static pixels among the at least a portion of pixels according to the motion attributes of the at least a portion of pixels; Acquire the feature descriptor of the static pixel from the feature descriptors of the at least a portion of the pixel points; According to the feature descriptors of the static pixel points and the feature descriptors of each position point recorded in the known environment map, the position of the device that collects the image to be processed during the process of collecting the image to be processed is located.

17. The method according to claim 16, characterized in that The first network model is a first fully convolutional network model; the second network model is a second fully convolutional network model; and the third network model is a third fully convolutional network model.

18. The method according to claim 17, characterized in that The step of calculating the motion attributes of the plurality of initial feature maps using the second network model to obtain the motion attributes of at least a portion of pixels in the image to be processed includes: Using the M initial feature maps as input images of the second fully convolutional network model, and inputting them into the second fully convolutional network model; In the second fully convolutional network model, at least one convolution calculation is performed on the M initial feature maps to output Q first target feature maps, each of which is a feature embodiment of the at least part of the pixels under a motion attribute; Calculating the motion attribute of at least a portion of the pixels according to the Q first target feature maps; Wherein, M and Q are integers greater than or equal to 2.

19. The method according to claim 18, characterized in that The calculating feature descriptors of the plurality of initial feature maps by using the third network model to obtain feature descriptors of at least a portion of pixels in the image to be processed includes: Using the M initial feature maps as input images of the third fully convolutional network model, and inputting them into the third fully convolutional network model; In the third fully convolutional network model, the M initial feature maps are convolved at least once to obtain N second target feature maps, each of which is a feature description of the at least a portion of the pixels; Wherein, the N second target feature maps form a feature descriptor of the at least a portion of pixels; Wherein, N is an integer greater than or equal to 2.

20. An image processing method, characterized in that: include: Inputting the image to be processed into a neural network model, wherein the neural network model includes at least two layers of structure; Using the first layer network in the neural network model, extracting features from the image to be processed to obtain a plurality of initial feature maps; Using the second layer network in the neural network model, the motion attributes of the multiple initial feature maps are calculated to obtain the motion attributes of at least a portion of the pixels in the image to be processed; Determine, according to the motion attribute of at least a portion of the pixels, a target pixel whose position change is less than or equal to a preset threshold; Acquire a feature descriptor of the target pixel point; and locate the position of the device that collects the image to be processed during the process of collecting the image to be processed based on the feature descriptor of the target pixel point and the feature descriptors of each position point recorded in a known environment map.

21. A visual positioning method, applicable to a terminal device, characterized in that: The method comprises: During the movement of the user, an environmental image around the current location of the user is collected, and the user carries the terminal device; Analyzing motion properties of at least a portion of pixels in the environment image using a neural network model; Determining static pixels among the at least a portion of pixels according to the motion attributes of the at least a portion of pixels; Acquire the feature descriptor of the static pixel point; locate the current location of the terminal device according to the feature descriptor of the static pixel point and the feature descriptors of each location point recorded in the known environment map, and output the current location of the user to guide the user to move.

22. The method according to claim 21, characterized in that The outputting the current location of the user to guide the user to move includes: The user's current location is output in voice mode to guide the user to move forward.

23. A data processing method, applicable to a mobile device, characterized in that: The method comprises: During the movement, collect environmental images around the current location; Analyzing motion properties of at least a portion of pixels in the environment image using a neural network model; Determining static pixels among the at least a portion of pixels according to the motion attributes of the at least a portion of pixels; Acquire a feature descriptor of the static pixel point; locate the current location of the mobile device according to the feature descriptor of the static pixel point and the feature descriptors of each location point recorded in a known environment map; The travel state information of the mobile device is calculated according to the current location of the mobile device and at least one previous historical location.

24. A mobile device, characterized in that: include: A mechanical body; the mechanical body is provided with a visual sensor, one or more processors, and one or more memories; wherein, The one or more memories are used to store computer instructions and environment maps; The one or more processors, coupled to the one or more memories, are configured to execute the computer instructions for: During the movement, collect environmental images around the current location; Analyzing motion properties of at least a portion of pixels in the environment image using a neural network model; Determining static pixels in at least a portion of pixels in the environment image according to motion attributes of at least a portion of pixels in the environment image; Acquire the feature descriptor of the static pixel point; locate the current location of the mobile device according to the feature descriptor of the static pixel point and the feature descriptor of each position point recorded in the known environment map.

25. The device according to claim 24, characterized in that The mobile device is an autonomous mobile device.

26. The device according to claim 25, characterized in that The autonomous mobile device is a robot, an unmanned vehicle or a drone.

27. A computer device, characterized in that: include: A memory and a processor; wherein, The memory is used to store computer instructions; The processor, coupled to the one or more memories, is configured to execute the computer instructions for: Get the image to be processed; Using the first network model to extract features from the image to be processed to obtain a plurality of initial feature maps; Calculating motion attributes of the multiple initial feature maps using a second network model to obtain motion attributes of at least a portion of pixels in the image to be processed; Using a third network model to calculate feature descriptors for the multiple initial feature maps, so as to obtain feature descriptors of at least a portion of pixels in the image to be processed; Determining static pixels among the at least a portion of pixels according to the motion attributes of the at least a portion of pixels; Acquire the feature descriptor of the static pixel from the feature descriptors of the at least a portion of the pixel points; According to the feature descriptors of the static pixel points and the feature descriptors of each position point recorded in the known environment map, the position of the device that collects the image to be processed during the process of collecting the image to be processed is located.

28. The device according to claim 27, characterized in that The computer device is an edge computing device in an edge cloud network system.

29. A computer device, characterized in that: include: A memory and a processor; wherein, The memory is used to store computer instructions; The processor, coupled to the one or more memories, is configured to execute the computer instructions for: Inputting the image to be processed into a neural network model, wherein the neural network model includes at least two layers of structure; Using the first layer network in the neural network model, extracting features from the image to be processed to obtain a plurality of initial feature maps; Using the second layer network in the neural network model, the motion attributes of the multiple initial feature maps are calculated to obtain the motion attributes of at least a portion of the pixels in the image to be processed; Determine, according to the motion attribute of at least a portion of the pixels, a target pixel whose position change is less than or equal to a preset threshold; Acquire a feature descriptor of the target pixel point; and locate the position of the device that collects the image to be processed during the process of collecting the image to be processed based on the feature descriptor of the target pixel point and the feature descriptors of each position point recorded in a known environment map.

30. The device according to claim 29, characterized in that The computer device is an edge computing device in an edge cloud network system.

31. A terminal device, characterized in that: include: A memory, a processor and a camera; wherein the camera is used to: collect an image of the environment around the current location of the user during the user's movement, and the user carries the terminal device; The memory is used to store computer instructions; The processor, coupled to the one or more memories, is configured to execute the computer instructions for: Analyzing motion properties of at least a portion of pixels in the environment image using a neural network model; Determining static pixels among the at least a portion of pixels according to the motion attributes of the at least a portion of pixels; Acquire the feature descriptor of the static pixel point; locate the current location of the terminal device according to the feature descriptor of the static pixel point and the feature descriptors of each location point recorded in the known environment map to guide the user to move.

32. A mobile device, characterized in that: include: A memory, a processor and a camera; wherein the camera is used to: during the movement of the mobile device, collect an image of the environment around the current location; The processor, coupled to the one or more memories, is configured to execute computer instructions for: Analyzing motion properties of at least a portion of pixels in the environment image using a neural network model; Determining static pixels among the at least a portion of pixels according to the motion attributes of the at least a portion of pixels; Acquire a feature descriptor of the static pixel point; locate the current location of the mobile device according to the feature descriptor of the static pixel point and the feature descriptors of each location point recorded in a known environment map; The travel state information of the mobile device is calculated according to the current location of the mobile device and at least one previous historical location.

33. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the steps in the method according to any one of claims 1 to 23.

Images