Positioning method and device of mobile device, storage medium and electronic device

Abstract

This application provides a positioning method and apparatus, storage medium, and electronic device for a mobile device. The method includes: acquiring three-dimensional spatial data of a space to be measured using a first sensor; performing feature recognition on the three-dimensional spatial data to obtain object features of an object to be matched; determining the relative position between the object to be matched and the target mobile device if the object features of the object to be matched match the object features of a preset target object; and positioning the target mobile device based on the relative position between the object to be matched and the target mobile device and the preset object position of the target object, wherein the preset object position is the position of the target object within a target area map. This technical solution solves the problem of low positioning accuracy in related mobile device positioning methods due to poor stability of sensor data.

Description

[Technical Field]

[0001] This application relates to the field of smart homes, and more specifically, to a method and apparatus for locating mobile devices, a storage medium, and an electronic device. [Background Technology]

[0002] Currently, mobile devices (e.g., cleaning robots) can be equipped with sensing sensors (e.g., laser sensors). By using the sensor data detected by these sensors, they can perceive information about surrounding obstacles and match the perceived two-dimensional information of the obstacles with a preset area map. Based on the matching results, the mobile device can be located.

[0003] When locating a mobile device using two-dimensional information collected by sensing sensors, the stability of the sensor data is poor if the mobile device is bumped or tilted, making it impossible to accurately perceive information about surrounding obstacles and thus impossible to locate the mobile device.

[0004] Therefore, it is evident that the positioning methods for mobile devices in related technologies suffer from low positioning accuracy due to the poor stability of sensor data. [Summary of the Invention]

[0005] The purpose of this application is to provide a positioning method and apparatus, storage medium and electronic device for mobile devices, so as to at least solve the problem of low positioning accuracy of mobile devices in related technologies due to poor stability of sensor data.

[0006] The purpose of this application is to achieve the following technical solution:

[0007] According to one aspect of the embodiments of this application, a method for locating a mobile device is provided, comprising: acquiring three-dimensional spatial data of a space to be measured through a first sensor; performing feature recognition on the three-dimensional spatial data to obtain object features of an object to be matched; if the object features of the object to be matched match the object features of a preset target object, determining the relative position between the object to be matched and the target mobile device based on two-dimensional spatial data; and locating the target mobile device based on the relative position between the object to be matched and the target mobile device and a preset object position of the target object, wherein the preset object position is the position of the target object within a target area map.

[0008] In one exemplary embodiment, before acquiring the three-dimensional spatial data of the space to be measured through the first sensor, the method further includes: if it is determined that the target mobile device is in an abnormal posture state, controlling the first sensor to be switched from a closed state to an open state.

[0009] In one exemplary embodiment, the method further includes: detecting pose reference parameters of the target mobile device by a pose detection component, wherein the pose reference parameters include at least one of the following: a position change parameter representing the position change of the target mobile device along a target axis, and a tilt angle parameter representing the tilt angle of the target mobile device; and determining that the target mobile device is in the abnormal posture state if the target mobile device is determined to be in a bumpy state based on the position change parameter or in a tilted state based on the tilt angle parameter.

[0010] In an exemplary embodiment, after performing feature recognition on the three-dimensional spatial data to obtain the object features of the object to be matched, the method further includes: determining, among a plurality of preset reference objects, a reference object whose object features match the object features of the object to be matched, wherein the object features of the object to be matched include at least one of the following: the object type of the object to be matched, and the object size of the object to be matched.

[0011] In an exemplary embodiment, after performing feature recognition on the three-dimensional spatial data to obtain the object features of the object to be matched, the method further includes: when the object to be matched contains multiple candidate objects, matching the multiple candidate objects with the multiple reference objects based on the object features of each candidate object, the positional relationship between the multiple candidate objects, the object features of a preset multiple reference objects, and the positional relationship between the multiple reference objects; and when the object features of the multiple candidate objects match the object features of multiple target objects among the multiple reference objects, and the positional relationship between the multiple candidate objects matches the positional relationship between the multiple target objects, determining the multiple target objects as matching objects that match the multiple candidate objects.

[0012] In one exemplary embodiment, determining the relative position between the object to be matched and the target mobile device includes: determining the relative position between the object to be matched and the target mobile device based on two-dimensional spatial data acquired by the second sensor.

[0013] In an exemplary embodiment, acquiring three-dimensional spatial data of the space to be measured via a first sensor includes: acquiring three-dimensional point cloud data of the space to be measured via a time-of-flight sensor, wherein the first sensor is the time-of-flight sensor and the three-dimensional spatial data is the three-dimensional point cloud data; the method further includes: acquiring laser point cloud data of the space to be measured via a laser sensor, wherein the second sensor is the laser sensor and the two-dimensional spatial data is the laser point cloud data.

[0014] In one exemplary embodiment, the method further includes: locating the target mobile device using the second sensor to obtain the target location of the target mobile device in the target area map; acquiring object features of the target object within the area containing the target location using the first sensor; determining the preset object location based on the target location and the relative position between the target mobile device and the target object; and saving the preset object location and the object features of the target object that have a corresponding relationship.

[0015] According to another aspect of the embodiments of this application, a positioning device for a mobile device is also provided, comprising: a first acquisition unit, configured to acquire three-dimensional spatial data of a space to be measured through a first sensor; an identification unit, configured to perform feature identification on the three-dimensional spatial data to obtain object features of an object to be matched; a first determination unit, configured to determine the relative position between the object to be matched and a target mobile device based on two-dimensional spatial data when the object features of the object to be matched match the object features of a preset target object; and a first positioning unit, configured to position the target mobile device based on the relative position between the object to be matched and the target mobile device and a preset object position of the target object, wherein the preset object position is the position of the target object within a target area map.

[0016] In one exemplary embodiment, the device further includes a control unit, configured to control the first sensor to switch from a closed state to an open state if it is determined that the target mobile device is in an abnormal posture state before the first sensor acquires three-dimensional spatial data of the space to be measured.

[0017] In one exemplary embodiment, the apparatus further includes: a detection unit, configured to detect pose reference parameters of the target mobile device via a pose detection component, wherein the pose reference parameters include at least one of the following: a position change parameter representing a change in position of the target mobile device along a target axis, and a tilt angle parameter representing a tilt angle of the target mobile device; and a second determination unit, configured to determine that the target mobile device is in the abnormal posture state if the target mobile device is determined to be in a bumpy state based on the position change parameter or in a tilted state based on the tilt angle parameter.

[0018] In an exemplary embodiment, the apparatus further includes: a third determining unit, configured to, after performing feature recognition on the three-dimensional spatial data to obtain the object features of the object to be matched, determine, among a preset plurality of reference objects, a reference object whose object features match the object features of the object to be matched, wherein the object features of the object to be matched include at least one of the following: the object type of the object to be matched, and the object size of the object to be matched.

[0019] In one exemplary embodiment, the apparatus further includes: a matching unit, configured to, after performing feature recognition on the three-dimensional spatial data to obtain the object features of the object to be matched, and when the object to be matched includes multiple candidate objects, match the multiple candidate objects with the multiple reference objects based on the object features of each of the multiple candidate objects, the positional relationship between the multiple candidate objects, the object features of a preset multiple reference objects, and the positional relationship between the multiple reference objects; and a fourth determining unit, configured to, when the object features of the multiple candidate objects match the object features of the multiple target objects among the multiple reference objects, and the positional relationship between the multiple candidate objects matches the positional relationship between the multiple target objects, determine the multiple target objects as matching objects that match the multiple candidate objects.

[0020] In an exemplary embodiment, the first determining unit includes a determining module, configured to determine the relative position between the object to be matched and the target mobile device based on two-dimensional spatial data acquired by the second sensor.

[0021] In one exemplary embodiment, the first acquisition unit includes an acquisition module, and the device further includes a second acquisition unit, wherein the acquisition module is configured to acquire three-dimensional point cloud data of the space to be measured via a time-of-flight sensor, wherein the first sensor is the time-of-flight sensor, and the three-dimensional spatial data is the three-dimensional point cloud data; the second acquisition unit is configured to acquire laser point cloud data of the space to be measured via a laser sensor, wherein the second sensor is the laser sensor, and the two-dimensional spatial data is the laser point cloud data.

[0022] In one exemplary embodiment, the apparatus further includes: a second positioning unit, configured to locate the target mobile device using the second sensor to obtain the target location of the target mobile device in the target area map; a third acquisition unit, configured to acquire object features of the target object within the area containing the target location using the first sensor; a fifth determination unit, configured to determine the preset object location based on the target location and the relative position between the target mobile device and the target object; and a storage unit, configured to store the preset object location and the object features of the target object that have a corresponding relationship.

[0023] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the test method of the above-described interface at runtime.

[0024] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the test method of the interface described above through the computer program.

[0025] In this embodiment, a method is adopted to first perform loop closure detection on the device based on three-dimensional spatial data, and then locate the device based on the loop closure result. Three-dimensional spatial data of the space to be measured is acquired through a first sensor; feature recognition is performed on the three-dimensional spatial data to obtain the object features of the object to be matched; if the object features of the object to be matched match the object features of a preset target object, the relative position between the object to be matched and the target mobile device is determined based on two-dimensional spatial data; the target mobile device is located based on the relative position between the object to be matched and the target mobile device and the preset object position of the target object, wherein the preset object position is the position of the target object within the target area map. Since the space to be measured is identified based on three-dimensional spatial data... The system identifies the object features of the target object within the device, determines a preset object that matches the target object based on these features, and locates the device based on the position of the preset object and the relative position between the detected target object and the mobile device. Compared to positioning based solely on two-dimensional information collected by sensors, this method combines the preset object position of the target object determined based on three-dimensional information. This reduces the impact of the instability of sensor data related to two-dimensional information on the positioning results, thereby improving the accuracy of device positioning. This solves the problem of low device positioning accuracy caused by poor sensor data stability in related technologies. [Attached Image Description]

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the hardware environment of an optional mobile device positioning method according to an embodiment of this application;

[0029] Figure 2 This is a flowchart illustrating an optional mobile device positioning method according to an embodiment of this application;

[0030] Figure 3 This is a flowchart illustrating another optional mobile device positioning method according to an embodiment of this application;

[0031] Figure 4 This is a structural block diagram of an optional positioning device for a mobile device according to an embodiment of this application;

[0032] Figure 5 This is a structural block diagram of an optional electronic device according to an embodiment of this application.

Detailed Implementation Methods

[0033] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0035] According to one aspect of the embodiments of this application, a method for locating a mobile device is provided. Optionally, in this embodiment, the above-described method for locating a mobile device can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102, mobile device 104, and server 106. For example... Figure 1 As shown, the terminal device 102 can connect to the mobile device 104 and / or the server 106 (e.g., an IoT platform or a cloud server) via a network to control the mobile device 104, such as binding to the mobile device 104 and configuring the device functions of the mobile device 104.

[0036] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth, infrared. The network used by terminal device 102 to communicate with mobile device 104 and / or server 106 may be the same as or different from the network used by mobile device 104 to communicate with server 106. Terminal device 102 may not be limited to PC, mobile phone, tablet computer, etc. Mobile device 104 may be a mobile robot, such as a cleaning device (e.g., a robot vacuum cleaner), a delivery device (e.g., a food delivery robot), or other types of mobile devices.

[0037] The mobile device positioning method of this application embodiment can be executed by a single terminal device 102, mobile device 104, or server 106, or it can be executed jointly by at least two of the terminal device 102, mobile device 104, and server 106. Alternatively, the mobile device positioning method of this application embodiment can be executed by a client installed on the terminal device 102 or mobile device 104.

[0038] Taking the positioning method of the mobile device in this embodiment as an example, which is executed by the mobile device 104, Figure 2 This is a flowchart illustrating an optional mobile device positioning method according to an embodiment of this application, as shown below. Figure 2 As shown, the process of this method may include the following steps:

[0039] Step S202: Acquire three-dimensional spatial data of the space to be measured through the first sensor.

[0040] The mobile device positioning method in this embodiment can be applied to scenarios where the mobile device is located during operation, such as when a cleaning device is performing area cleaning or other tasks. The mobile device can be a cleaning device, a delivery device, or other types of mobile devices. When the mobile device is picked up, the positioning failure occurs for a preset duration, or during normal movement, the location of the mobile device can be determined by the sensor data from its configured sensing sensors.

[0041] The current location of the mobile device can be its position within a preset area map. The target mobile device can be equipped with one or more sensors, such as a first sensor for sensing three-dimensional spatial information. The first sensor can detect the target mobile device's location within the target area map, based on the detected sensor data.

[0042] The first sensor mentioned above can be any type of sensor used to sense three-dimensional spatial information, such as a TOF (Time of Flight) sensor, an infrared sensor, or other types of sensors. In this embodiment, the sensor type of the first sensor is not limited.

[0043] In addition, the mobile device may also be equipped with a second sensor for sensing two-dimensional spatial information. That is, the second sensor may be a sensor for sensing two-dimensional spatial information, such as a laser sensor. The laser sensor may be an LDS (Laser Distance Sensor) sensor or other sensors. In this embodiment, the sensor type of the second sensor is not limited.

[0044] In related technologies, the sensors used for locating mobile devices are usually two-dimensional sensing sensors, such as LDS sensors. LDS sensors can only sense the distance between obstacles and the sensor, losing information in three-dimensional space. When locating mobile devices, the mobile devices cannot handle abnormal states such as bumps or tilts, otherwise the accuracy and stability of the positioning will be reduced.

[0045] In this embodiment, to at least partially solve the aforementioned technical problems, three-dimensional spatial data (e.g., three-dimensional point cloud data) and two-dimensional spatial data (e.g., laser point cloud data) detected by a first sensor can be combined. The mobile device can be located using the information from the three-dimensional space of the target space combined with the information from the two-dimensional space, thereby improving the accuracy of mobile device positioning. For the target mobile device, the first sensor configured on the target mobile device can acquire the three-dimensional spatial data of the target space.

[0046] Step S204: Perform feature recognition on the three-dimensional spatial data to obtain the object features of the object to be matched.

[0047] For three-dimensional spatial data, the target mobile device can perform feature recognition on the three-dimensional spatial data to identify the features of preset types of objects in the space to be tested, such as furniture, doors, etc., thereby obtaining the object features of the object to be matched. The object features of the object to be matched can be used to describe the characteristics of the object to be matched, such as object type, object size, object color, etc. In this embodiment, the object features are not limited.

[0048] The object to be matched can contain one or more objects. For objects of the preset object type, they can be identified and the object features of all objects that meet the requirements (i.e., belong to the preset object type) can be retained.

[0049] Step S206: If the object features of the object to be matched match the object features of the preset target object, determine the relative position between the object to be matched and the target mobile device based on the two-dimensional spatial data.

[0050] After obtaining the object features of the object to be matched, the target mobile device can match these features with the object features of multiple preset reference objects. When a reference object with the same object features is matched, that is, when the object features of the object to be matched match the object features of the preset target object (i.e., the matching object), it is determined that this feature has been observed before. Here, "identical object features" means that the object features are the same (e.g., object type) or the deviation is within a certain range.

[0051] If a matching object exists, the relative position between the object to be matched and the target mobile device can be determined based on the two-dimensional spatial data mentioned above. The relative position here can be the relative position of the two objects in a two-dimensional plane.

[0052] Optionally, when the two-dimensional spatial data is acquired by the second sensor, the relative position between the object to be matched and the target mobile device can be determined based on the relative position between the centroid of the second sensor and the target mobile device. Alternatively, the relative position between the object to be matched and the target mobile device can also be represented by the relative position between the object to be matched and the second sensor.

[0053] Of course, two-dimensional spatial data can also be obtained by any other means, and is not limited to those obtained through a second sensor. For example, it can be obtained by further analyzing three-dimensional spatial information obtained by a first sensor, etc., and this is not limited here.

[0054] Step S208: The target mobile device is located based on the relative position between the object to be matched and the target mobile device and the preset object position of the target object, wherein the preset object position is the position of the target object within the target area map.

[0055] The target object is a preset reference object, which may have a corresponding object position, i.e., a preset object position. The preset object position can be pre-specified by the user of the target mobile device, or it can be automatically determined based on the construction of the area map or other scenarios. Based on the relative position between the object to be matched and the target mobile device and the aforementioned preset object position, the target mobile device can be located, thereby determining the current position of the target mobile device within the target area map.

[0056] Optionally, the preset object position can be the position coordinates of the preset object point of the target object within the target area map, while the relative position between the object to be matched and the target mobile device is the relative position between the object point at the same position on the object to be matched and the target mobile device. It can be in the form of a vector. By superimposing the relative position vector on the position coordinates of the preset object point of the target object, the current position coordinates of the target mobile device can be determined.

[0057] Through steps S202 to S208, three-dimensional spatial data of the space to be measured is acquired by the first sensor; feature recognition is performed on the three-dimensional spatial data to obtain the object features of the object to be matched; if the object features of the object to be matched match the object features of the preset target object, the relative position between the object to be matched and the target mobile device is determined based on the two-dimensional spatial data; the target mobile device is located based on the relative position between the object to be matched and the target mobile device and the preset object position of the target object, wherein the preset object position is the position of the target object within the target area map. This solves the problem of low accuracy in mobile device positioning due to poor stability of sensor data in related technologies, and improves the accuracy of device positioning.

[0058] In one exemplary embodiment, determining the relative position between the object to be matched and the target mobile device includes:

[0059] S11, Based on the two-dimensional spatial data obtained by the second sensor, determine the relative position between the object to be matched and the target mobile device.

[0060] In this embodiment, the target mobile device may also be equipped with a second sensor. The type of the second sensor is similar to that in the previous embodiments and will not be described in detail here. The sensor data acquired by the second sensor is two-dimensional spatial data. Here, the two-dimensional spatial data and the three-dimensional spatial data can be spatial data of the space to be measured detected at the same time, or spatial data of the space to be measured detected at two times with a detection interval of less than a preset time interval.

[0061] By combining three-dimensional spatial data with two-dimensional spatial data, joint localization of the target mobile device can be achieved. Optionally, the two-dimensional spatial data can be used to determine the relative position between the object to be matched and the target mobile device; that is, the relative position can be determined based on the two-dimensional spatial data acquired by the second sensor. The two-dimensional spatial data may include distance data between the second sensor and the object to be matched. In this case, the second sensor can be a ranging sensor, and the two-dimensional spatial data can be two-dimensional ranging data. Based on the ranging result, the relative position between the object to be matched and the target mobile device can be determined.

[0062] This embodiment uses two-dimensional spatial data to determine the relative position between an object and a mobile device, which can improve the accuracy and convenience of position detection, while also being compatible with the existing positioning functions of mobile devices.

[0063] In one exemplary embodiment, acquiring three-dimensional spatial data of the space to be measured via a first sensor includes:

[0064] S21, acquire three-dimensional point cloud data of the space to be measured through a time-of-flight sensor, wherein the first sensor is a time-of-flight sensor and the three-dimensional spatial data is three-dimensional point cloud data.

[0065] For scenarios involving joint localization based on two-dimensional and three-dimensional spatial data, the above methods also include:

[0066] S22, acquire laser point cloud data of the space to be measured through a laser sensor, wherein the second sensor is a laser sensor and the two-dimensional spatial data is laser point cloud data.

[0067] In this embodiment, the first sensor can be a Time-of-Flight (TOF) sensor, and the second sensor can be a laser sensor, such as an LDS sensor. Using the time-of-flight sensor, the target mobile device can acquire three-dimensional point cloud data of the space under test (in this case, the three-dimensional spatial data can include three-dimensional point cloud data). Using the laser sensor, the target mobile device can acquire laser point cloud data of the space under test (in this case, the two-dimensional spatial data can include laser point cloud data).

[0068] Here, the Time-of-Flight (TOF) sensor can observe point cloud data in three-dimensional space, easily identifying features such as furniture, flower pots, doors, and windows within a home. This compensates for the limitations of laser sensors, which can only observe two-dimensional information and cannot distinguish indoor features. However, TOF sensors have limited viewing angles and sensing distances, and are prone to failure when encountering obstructions during localization. Laser sensors, on the other hand, can perceive data within a 350° range and have a greater measurement distance than TOF sensors, resulting in more stable localization results. Combining TOF and laser sensors can improve the success rate and accuracy of loop closure.

[0069] This embodiment uses a combination of a TOF sensor and a laser sensor to perform loop closure detection on mobile devices, which can improve the success rate and accuracy of loop closure.

[0070] In one exemplary embodiment, the above method further includes:

[0071] S31, The target mobile device is located by the second sensor to obtain the target location of the target mobile device in the target area map;

[0072] S32, acquire the object features of the target object within the area containing the target location using the first sensor;

[0073] S33, determine the preset object position based on the target position and the relative position between the target mobile device and the target object;

[0074] S34, save the preset object positions and target object features with corresponding relationships.

[0075] When constructing a target area map (e.g., when the target mobile device moves within the target area for the first time), the target mobile device can be located using a second sensor, and object features within a certain range of the current location can be detected using a first sensor. Based on these features, an area map can be constructed to obtain the target area map.

[0076] In this embodiment, the target mobile device can be located using a second sensor to obtain its target location on a target area map. This location can be SLAM (Simultaneous Localization and Mapping) location. Here, SLAM can be: the mobile device starts moving from an unknown location in an unknown environment, performs its own localization based on its location and map during the movement, and simultaneously builds an incremental map based on its own localization, thereby achieving autonomous localization and navigation of the mobile device.

[0077] Meanwhile, the target mobile device can acquire the object features of the target object within the area containing the target location through the first sensor, such as the features of furniture, flower pots, doors and windows in a home, and determine the preset object position based on the target location and the relative position between the target mobile device and the target object. Here, the method of determining the preset object position is similar to the method of determining the device position of the target mobile device mentioned above, and will not be described in detail here.

[0078] After obtaining the preset object position, the target mobile device can save the preset object position and the object features of the target object with corresponding relationships. By continuously executing the above process of determining the object features and object position, the object features and object positions of multiple reference objects can be obtained, thereby facilitating loop closure detection of the target mobile device based on the object features and object positions of multiple reference objects, so as to locate the target mobile device.

[0079] It should be noted that saving the corresponding preset object positions and target object features can be done by saving the corresponding preset object positions and target object features to a target feature map (which can be the same map as the target area map or a different map). The aforementioned matching of the object features of the object to be matched with the object features of multiple preset reference objects can include: matching the object features of the object to be matched with the object features within the target feature map.

[0080] For example, using a TOF sensor can perceive three-dimensional environmental information and identify features such as furniture and doors. These features are then matched one-to-one with the SLAM localization information results from an LDS sensor at the same time to construct a map based on these features. During its movement, the mobile robot (a type of mobile device) searches the feature map to see if it has observed the feature before. If the feature has been observed repeatedly, a closed-loop (closed-loop) constraint can be established.

[0081] This embodiment saves the correspondence between object features and object positions when constructing a regional map, which facilitates loop closure detection of mobile devices and improves the accuracy of mobile device positioning.

[0082] In one exemplary embodiment, before acquiring the three-dimensional spatial data of the space to be measured via the first sensor, the method further includes:

[0083] S41, if it is determined that the target mobile device is in an abnormal posture state, control the first sensor to be changed from the off state to the on state.

[0084] The first sensor can remain on at all times, meaning that the mobile device is located using a method similar to that in the previous embodiments during its operation. However, this method requires joint device positioning, which increases the complexity of the positioning process and consumes a significant amount of computing resources. Consequently, it suffers from long device positioning latency and high resource consumption.

[0085] In this embodiment, when the target mobile device is in a normal posture state, such as with minimal bumps or a small tilt angle, it can be located using only a two-dimensional sensing sensor (e.g., a second sensor), while the first sensor is in a turned-off state. Only when the target mobile device is in an abnormal posture state is the first sensor switched from a turned-off state to an turned-on state, and the target mobile device is then located using either the first sensor or a combination of the first sensor and the two-dimensional sensing sensor.

[0086] In this embodiment, when a mobile device is in an abnormal posture, activating a three-dimensional sensing sensor to locate the mobile device can shorten the device location latency and reduce the resource consumption for device location.

[0087] In one exemplary embodiment, the above method further includes:

[0088] S51, the pose reference parameters of the target mobile device are detected by the pose detection component, wherein the pose reference parameters include at least one of the following: a position change parameter for representing the position change of the target mobile device along the target axis, and a tilt angle parameter for representing the tilt angle of the target mobile device.

[0089] S52, if the target mobile device is determined to be in an abnormal attitude state based on the position change parameter or the target mobile device is determined to be in an tilted state based on the tilt angle parameter.

[0090] In this embodiment, there can be multiple abnormal posture states, including but not limited to one of the following: a bumpy state, a tilted state. Correspondingly, there can be multiple ways to determine that the target mobile device is in an abnormal posture state, including but not limited to one of the following:

[0091] Method 1: Detect the position change parameters of the target mobile device through a pose detection component. The position change parameters are used to represent the position change of the target mobile device along the target axis. If the target mobile device is determined to be in a bumpy state based on the position change parameters, the target mobile device is determined to be in an abnormal posture state.

[0092] Here, the target axis can be a coordinate axis perpendicular to a preset plane, such as the Z-axis. The position change parameter is used to represent the position change of the target mobile device along the target axis, that is, to represent the height change of the target mobile device. If the height change is gradual, it can be determined that the target mobile device is in a stable movement state; otherwise, it is determined that the target mobile device is in a bumpy state. The bumpy state is the state in which the target mobile device repeatedly switches between height increase and height decrease within a short period of time (e.g., within a preset time range). At this time, the accuracy of the two-dimensional spatial data detected by the sensing sensor is low, and it is difficult to locate the target mobile device using two-dimensional spatial data. Therefore, the first sensor can be activated for sensor joint positioning.

[0093] Method 2: Detect the tilt angle parameter of the target mobile device through a pose detection component. The tilt angle parameter is used to represent the tilt angle of the target mobile device. If the target mobile device is determined to be in a tilted state based on the tilt angle parameter, the target mobile device is determined to be in an abnormal posture state.

[0094] Here, the tilt angle can be the angle between the target mobile device and a preset plane, such as the pitch angle. The tilt angle parameter is used to represent the tilt angle of the target mobile device, that is, to represent the size of the pitch angle of the target mobile device. If the tilt angle is too large, the amount of information contained in the two-dimensional spatial data detected by the two-dimensional sensing sensor is small, making it difficult to locate the target mobile device using the two-dimensional spatial data. In this case, the first sensor can be activated for sensor joint positioning.

[0095] In this embodiment, when a mobile device is detected to be bumpy or tilted at an excessive angle, a three-dimensional sensing sensor is activated for sensor positioning, which can improve the rationality of sensor control and the accuracy of mobile device positioning.

[0096] In an exemplary embodiment, after performing feature recognition on the three-dimensional spatial data to obtain the object features of the object to be matched, the above method further includes:

[0097] S61, determine a reference object whose object features match the object features of the object to be matched from a plurality of preset reference objects, wherein the object features of the object to be matched include at least one of the following: the object type of the object to be matched, the object size of the object to be matched.

[0098] In this embodiment, object features may include multiple features, including but not limited to at least one of the following: object type, object size. Correspondingly, the object features of the object to be matched include at least one of the following: object type of the object to be matched, object size of the object to be matched. Here, object type can be a variety of preset object types, each object type corresponding to a category of objects. For example, object types include but are not limited to furniture, flower pots, doors and windows in a home, etc.

[0099] Multiple reference object features can be preset on the target area map or target feature map. After obtaining the object features of the object to be matched, the object features of the object to be matched can be matched with the object features of each reference object to obtain the matching degree between the object to be matched and each reference object. Then, based on the matching degree between the object to be matched and each reference object, the target object is determined. The target object can be: among the multiple preset reference objects, the reference object whose object features match the object features of the object to be matched.

[0100] Here, the object that matches the object to be matched can be a reference object with a matching degree greater than a preset matching degree threshold, or the reference object with the highest matching degree, or other methods of determining the matching object. This embodiment does not limit this.

[0101] This embodiment improves the convenience and accuracy of object matching by performing feature matching based on object type and size.

[0102] In an exemplary embodiment, after performing feature recognition on the three-dimensional spatial data to obtain the object features of the object to be matched, the above method further includes:

[0103] S71, when the object to be matched contains multiple candidate objects, the multiple candidate objects are matched with multiple reference objects based on the object features of each candidate object, the positional relationship between the multiple candidate objects, the object features of the preset multiple reference objects, and the positional relationship between the multiple reference objects.

[0104] S72, if the object features of multiple candidate objects match the object features of multiple target objects among multiple reference objects, and the positional relationship between multiple candidate objects matches the positional relationship between multiple target objects, then the multiple target objects are determined as matching objects that match the multiple candidate objects.

[0105] If there is only one object to be matched, object feature matching can be performed in a manner similar to that described in the previous embodiments to determine the reference object corresponding to the object to be matched. This has already been explained and will not be repeated here.

[0106] If the object to be matched contains multiple candidate objects, one candidate object can be selected as the object to be matched, and object feature matching can be performed in a manner similar to that in the previous embodiments to determine the reference object corresponding to the object to be matched. The selection method can be to select a candidate object of the target type as the object to be matched, or it can be random selection, or other selection methods.

[0107] In this embodiment, in order to improve the accuracy of object feature matching, multiple candidate objects can be matched with multiple reference objects based on the object features of each candidate object, the positional relationship between multiple candidate objects, the object features of multiple preset reference objects, and the positional relationship between multiple reference objects. That is, in addition to judging whether the object features match, it also judges whether the positional relationship between adjacent features is correct. For example, multiple doors are repeatedly observed at the current position.

[0108] If the object features of multiple candidate objects match the object features of multiple target objects among multiple reference objects, and the positional relationship between multiple candidate objects matches the positional relationship between multiple target objects, then the multiple target objects can be identified as matching objects that match the multiple candidate objects.

[0109] When locating a target mobile device, one target object can be selected from multiple target objects. The target mobile device is located based on the position of the selected target object and the relative position between the target mobile device and the selected target object, thereby improving the convenience of mobile device location.

[0110] This embodiment improves the accuracy of mobile device positioning by locating the mobile device based on object features and the positional relationships of adjacent features.

[0111] The positioning method for the mobile device in this embodiment will be explained below with reference to an optional example. In this optional example, the target mobile device is a mobile robot, the first sensor is a TOF sensor, and the second sensor is an LDS sensor.

[0112] This optional example provides a SLAM localization scheme that uses a ToF sensor for loop closure to assist LDS point cloud localization, such as... Figure 3 As shown, the process of locating a mobile device in this optional example may include the following steps:

[0113] Step S302: Construct a regional feature map using a TOF sensor and an LDS sensor.

[0114] A TOF sensor is used to perceive three-dimensional environmental information and identify features such as furniture and doors. These features are then matched one-to-one with the LDS SLAM localization results at the same time to construct a map based on these features, namely, a regional feature map.

[0115] Step S304: Use the constructed regional feature map to locate the mobile robot.

[0116] A kdtree (k-dimensional tree) is constructed from the regional feature map. Using the LDS SLAM localization results, object features within a certain range of the current location can be detected on the kdtree. It is then determined whether the object features have been observed repeatedly. If all conditions are met, it proves that the robot has been to this location, thus establishing a closed-loop (loop-closure) constraint for localization of the mobile robot.

[0117] Here, the methods for determining whether an object's features have been observed repeatedly can include:

[0118] 1) Are they all of the same type of feature, for example, are they all doors or sofas?

[0119] 2) Are the dimensions of the features consistent? For example, some doors in a house are wide and some are narrow. Determine if the door dimensions are consistent.

[0120] This example demonstrates how to improve the stability of LDS sensors in positioning when the robot is bumpy or tilted, making them more robust and accurate when the robot is traversing slopes.

[0121] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0123] According to another aspect of the embodiments of this application, a positioning device for a mobile device that implements the positioning method of the above-described mobile device is also provided. Figure 4 This is a structural block diagram of an optional positioning device for a mobile device according to an embodiment of this application, such as... Figure 4 As shown, the device may include:

[0124] The first acquisition unit 402 is used to acquire three-dimensional spatial data of the space to be measured through the first sensor;

[0125] The identification unit 404 is connected to the first acquisition unit 402 and is used to perform feature recognition on three-dimensional spatial data to obtain the object features of the object to be matched.

[0126] The first determining unit 406, connected to the identification unit 404, is used to determine the relative position between the object to be matched and the target mobile device based on two-dimensional spatial data when the object features of the object to be matched match the object features of the preset target object.

[0127] The first positioning unit 408 is connected to the first determining unit 406 and is used to locate the target mobile device based on the relative position between the object to be matched and the target mobile device and the preset object position of the target object, wherein the preset object position is the position of the target object in the target area map.

[0128] It should be noted that the first acquisition unit 402 in this embodiment can be used to perform the above step S202, the identification unit 404 in this embodiment can be used to perform the above step S204, the first determination unit 406 in this embodiment can be used to perform the above step S206, and the first positioning unit 408 in this embodiment can be used to perform the above step S208.

[0129] Through the above modules, three-dimensional spatial data of the space to be measured is acquired through the first sensor; feature recognition is performed on the three-dimensional spatial data to obtain the object features of the object to be matched; if the object features of the object to be matched match the object features of the preset target object, the relative position between the object to be matched and the target mobile device is determined; based on the relative position between the object to be matched and the target mobile device and the preset object position of the target object, the target mobile device is located, wherein the preset object position is the position of the target object within the target area map. This solves the problem of low accuracy in mobile device positioning due to poor stability of sensor data in related technologies, and improves the accuracy of device positioning.

[0130] In one exemplary embodiment, the above-described apparatus further includes:

[0131] The control unit is used to control the first sensor to switch from a closed state to an open state before acquiring three-dimensional spatial data of the space to be measured through the first sensor, and if it is determined that the target mobile device is in an abnormal posture state.

[0132] In one exemplary embodiment, the above-described apparatus further includes:

[0133] The detection unit is used to detect the pose reference parameters of the target mobile device through the pose detection component, wherein the pose reference parameters include at least one of the following: a position change parameter for representing the position change of the target mobile device along the target axis, and a tilt angle parameter for representing the tilt angle of the target mobile device.

[0134] The second determining unit is used to determine that the target mobile device is in an abnormal posture state when the target mobile device is determined to be in a bumpy state based on the position change parameter or to be in a tilted state based on the tilt angle parameter.

[0135] In one exemplary embodiment, the above-described apparatus further includes:

[0136] The third determining unit is used to determine, after performing feature recognition on the three-dimensional spatial data and obtaining the object features of the object to be matched, a set of preset reference objects whose object features match the object features of the object to be matched, wherein the object features of the object to be matched include at least one of the following: the object type of the object to be matched, and the object size of the object to be matched.

[0137] In one exemplary embodiment, the above-described apparatus further includes:

[0138] The matching unit is used to match multiple candidate objects with multiple reference objects after performing feature recognition on the three-dimensional spatial data to obtain the object features of the object to be matched, when the object to be matched contains multiple candidate objects, based on the object features of each candidate object, the positional relationship between the multiple candidate objects, the object features of multiple preset reference objects, and the positional relationship between the multiple reference objects.

[0139] The fourth determining unit is used to determine multiple target objects as matching objects that match multiple candidate objects when the object features of multiple candidate objects match the object features of multiple target objects among multiple reference objects, and the positional relationship between multiple candidate objects matches the positional relationship between multiple target objects.

[0140] In one exemplary embodiment, the first determining unit includes:

[0141] The determination module is used to determine the relative position between the object to be matched and the target mobile device based on the two-dimensional spatial data acquired by the second sensor.

[0142] In one exemplary embodiment, the first acquisition unit includes an acquisition module, and the apparatus further includes a second acquisition unit, wherein...

[0143] The acquisition module is used to acquire three-dimensional point cloud data of the space to be measured through a time-of-flight sensor, wherein the first sensor is a time-of-flight sensor and the three-dimensional spatial data is three-dimensional point cloud data;

[0144] The second acquisition unit is used to acquire laser point cloud data of the space to be measured through a laser sensor, wherein the second sensor is a laser sensor and the two-dimensional spatial data is laser point cloud data.

[0145] In one exemplary embodiment, the above-described apparatus further includes:

[0146] The second positioning unit is used to locate the target mobile device using the second sensor and obtain the target location of the target mobile device in the target area map.

[0147] The third acquisition unit is used to acquire the object features of the target object within the area containing the target location through the first sensor;

[0148] The fifth determining unit is used to determine the preset object position based on the target position and the relative position between the target mobile device and the target object;

[0149] The storage unit is used to store the preset object positions and target object features that have a corresponding relationship. It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can operate in environments such as... Figure 1 The hardware environment shown can be implemented through software or hardware, and the hardware environment includes the network environment.

[0150] According to another aspect of the embodiments of this application, a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to execute program code for the positioning method of any of the mobile devices described in the embodiments of this application.

[0151] Optionally, in this embodiment, the storage medium may be located on at least one of the network devices in the network shown in the above embodiment.

[0152] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

[0153] S1, acquire three-dimensional spatial data of the space to be measured through the first sensor;

[0154] S2, perform feature recognition on the three-dimensional spatial data to obtain the object features of the object to be matched;

[0155] S3, if the object features of the object to be matched match the object features of the preset target object, determine the relative position between the object to be matched and the target mobile device based on the two-dimensional spatial data.

[0156] S4. Based on the relative position between the object to be matched and the target mobile device and the preset object position of the target object, the target mobile device is located. The preset object position is the position of the target object within the target area map.

[0157] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated in this embodiment.

[0158] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0159] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described positioning method for mobile devices is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0160] Figure 5 This is a structural block diagram of an optional electronic device according to an embodiment of this application, such as... Figure 5 As shown, it includes a processor 502, a communication interface 504, a memory 506, and a communication bus 508. The processor 502, communication interface 504, and memory 506 communicate with each other via the communication bus 508.

[0161] Memory 506 is used to store computer programs;

[0162] When processor 502 executes a computer program stored in memory 506, it performs the following steps:

[0163] S1, acquire three-dimensional spatial data of the space to be measured through the first sensor;

[0164] S2, perform feature recognition on the three-dimensional spatial data to obtain the object features of the object to be matched;

[0165] S3, if the object features of the object to be matched match the object features of the preset target object, determine the relative position between the object to be matched and the target mobile device based on the two-dimensional spatial data.

[0166] S4. Based on the relative position between the object to be matched and the target mobile device and the preset object position of the target object, the target mobile device is located. The preset object position is the position of the target object within the target area map.

[0167] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 5 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic device and other devices.

[0168] The aforementioned memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0169] As an example, the memory 506 described above may include, but is not limited to, the first acquisition unit 402, the identification unit 404, the first determination unit 406, and the first positioning unit 408 from the control device of the aforementioned device. Furthermore, it may include, but is not limited to, other module units from the control device of the aforementioned device, which will not be elaborated upon in this example.

[0170] The processors mentioned above can be general-purpose processors, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; they can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0171] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0172] Those skilled in the art will understand that Figure 5The structure shown is for illustrative purposes only. The device implementing the above-described mobile device positioning method can be a terminal device, such as a smartphone (e.g., Android phone, iOS phone), tablet computer, PDA, mobile internet device (MID), PAD, etc. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, the electronic device may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0173] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0174] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0175] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0176] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0178] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the solution provided in this embodiment, depending on actual needs.

[0179] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0180] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A positioning method for a mobile device, characterized in that, include: The three-dimensional spatial data of the space to be measured is acquired through the first sensor; The three-dimensional spatial data is used to perform feature recognition to obtain the object features of the object to be matched; If the object features of the object to be matched match the object features of the preset target object, the relative position between the object to be matched and the target mobile device is determined based on two-dimensional spatial data. The target mobile device is located based on the relative position between the object to be matched and the target mobile device and the preset object position of the target object, wherein the preset object position is the position of the target object within the target area map; Prior to acquiring the three-dimensional spatial data of the space to be measured via the first sensor, the method further includes: If it is determined that the target mobile device is in an abnormal posture state, the first sensor is controlled to be switched from the off state to the on state.

2. The method as described in claim 1, characterized in that, The method further includes: The pose reference parameters of the target mobile device are detected by the pose detection component, wherein the pose reference parameters include at least one of the following: a position change parameter for representing the position change of the target mobile device along the target axis, and a tilt angle parameter for representing the tilt angle of the target mobile device; If the target mobile device is determined to be in a bumpy state based on the position change parameter or in a tilted state based on the tilt angle parameter, then the target mobile device is determined to be in the abnormal posture state.

3. The method as described in claim 1, characterized in that, After performing feature recognition on the three-dimensional spatial data to obtain the object features of the object to be matched, the method further includes: Among a plurality of preset reference objects, a reference object whose object features match the object features of the object to be matched is identified, wherein the object features of the object to be matched include at least one of the following: the object type of the object to be matched, and the object size of the object to be matched.

4. The method as described in claim 1, characterized in that, After performing feature recognition on the three-dimensional spatial data to obtain the object features of the object to be matched, the method further includes: When the object to be matched contains multiple candidate objects, the multiple candidate objects are matched with the multiple reference objects based on the object features of each candidate object, the positional relationship between the multiple candidate objects, the object features of a preset multiple reference objects, and the positional relationship between the multiple reference objects. If the object features of the multiple candidate objects match the object features of the multiple target objects among the multiple reference objects, and the positional relationship between the multiple candidate objects matches the positional relationship between the multiple target objects, then the multiple target objects are determined as matching objects that match the multiple candidate objects.

5. The method according to any one of claims 1 to 4, characterized in that, Determining the relative position between the object to be matched and the target mobile device based on two-dimensional spatial data includes: The relative position between the object to be matched and the target mobile device is determined based on the two-dimensional spatial data obtained by the second sensor.

6. The method as described in claim 5, characterized in that, The step of acquiring three-dimensional spatial data of the space to be measured through the first sensor includes: acquiring three-dimensional point cloud data of the space to be measured through a time-of-flight sensor, wherein the first sensor is the time-of-flight sensor, and the three-dimensional spatial data is the three-dimensional point cloud data; The method further includes: acquiring laser point cloud data of the space to be measured through a laser sensor, wherein the second sensor is the laser sensor, and the two-dimensional spatial data is the laser point cloud data.

7. The method as described in claim 5, characterized in that, The method further includes: The target mobile device is located using the second sensor to obtain its target location on the target area map. The first sensor acquires the object features of the target object within a region containing the target location; The preset object position is determined based on the target location and the relative position between the target mobile device and the target object; The preset object positions and the object features of the target object, which have a corresponding relationship, are saved.

8. A positioning device for a mobile device, characterized in that, include: The first acquisition unit is used to acquire three-dimensional spatial data of the space to be measured through the first sensor; The identification unit is used to perform feature identification on the three-dimensional spatial data to obtain the object features of the object to be matched. The first determining unit is used to determine the relative position between the object to be matched and the target mobile device based on two-dimensional spatial data when the object features of the object to be matched match the object features of the preset target object. The first positioning unit is used to locate the target mobile device based on the relative position between the object to be matched and the target mobile device and the preset object position of the target object, wherein the preset object position is the position of the target object within the target area map; The control unit is used to control the first sensor to switch from a closed state to an open state before acquiring three-dimensional spatial data of the space to be measured through the first sensor, and if it is determined that the target mobile device is in an abnormal posture state.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 7 through the computer program.

Images