Method and device for determining state of object, medium and equipment

Abstract

The embodiment of the invention discloses a method and device for determining the state of an object, a medium and equipment, and the method comprises the steps: determining first object information, obtained by observing the object through a first sensor at a first moment, of autonomous mobile equipment, and second object information, obtained by observing the object through a second sensor, of the autonomous mobile equipment; determining a topological graph about the position, velocity and acceleration; determining a target function based on the topological graph; determining an initial state of a variable node in the topological graph based on third object information of the object at the second moment; based on the first object information, the second object information, the initial states of the variable nodes and the target function, target states corresponding to the variable nodes are obtained through iterative updating; and determining the target object information of the object at the first moment based on the target state so as to reduce the calculation amount and reduce the implementation difficulty on the basis of improving the state precision of the object.

Description

Technical Field

[0001] The present disclosure relates to computer vision technology, and more particularly to a method, apparatus, medium, and device for determining the state of an object. Background Art

[0002] In an Advanced Driving Assistance System (ADAS), various sensors installed on the vehicle are usually used to perceive objects (or obstacles) in the surrounding environment. By tracking and measuring the state of the objects, obstacle information is provided to the driver, allowing the driver to quickly detect potential dangers in a complex traffic environment and take corresponding countermeasures in a timely manner to ensure the safety of vehicle driving. In the related art, in the field of forward-looking obstacle state estimation, the state of the obstacle is usually determined based on monocular camera ranging and speed measurement or multi-camera ranging and speed measurement. However, the measurement accuracy of a monocular camera is low, and it is difficult to cope with complex traffic scenes. The computational complexity of a binocular camera is high, and in order to ensure stereo matching between the two cameras, the calibration requirements between the two lenses are very strict, which brings great challenges to the implementation of binocular camera ranging and speed measurement. Summary of the Invention

[0003] The embodiments of the present disclosure provide a method, apparatus, medium, and device for determining the state of an object, so as to avoid using a binocular camera, reduce the amount of calculation, and lower the difficulty of implementation while improving the accuracy of the object's state.

[0004] A first aspect of an embodiment of the present disclosure provides a method for determining the state of an object, comprising: determining first object information obtained by an autonomous mobile device observing the object through a first sensor at a first moment; determining second object information obtained by the autonomous mobile device observing the object through a second sensor at the first moment; determining a topological graph regarding the position, velocity, and acceleration of the object; the topological graph comprising variable nodes and edges representing constraint relationships between the variable nodes; determining an objective function based on the topological graph; determining an initial state of the variable node in the topological graph based on third object information of the object at the second moment; the second moment being before the first moment; iteratively updating the state of the variable node based on the first object information, the second object information, the initial state of the variable node, and the objective function to obtain a target state corresponding to the variable node; and determining the target object information of the object at the first moment based on the target state corresponding to the variable node.

[0005] According to a second aspect of an embodiment of the present disclosure, there is provided an apparatus for determining a state of an object, comprising: a first processing module for determining first object information obtained by an autonomous mobile device observing the object through a first sensor at a first moment; a second processing module for determining second object information obtained by the autonomous mobile device observing the object through a second sensor at the first moment; a third processing module for determining a topological graph regarding the position, velocity, and acceleration of the object; the topological graph comprising variable nodes and edges representing constraint relationships between the variable nodes; a fourth processing module for determining an objective function based on the topological graph; a fifth processing module for determining an initial state of the variable node in the topological graph based on the third object information of the object at the second moment; the second moment being before the first moment; a sixth processing module for iteratively updating the state of the variable node based on the first object information, the second object information, the initial state of the variable node, and the objective function to obtain a target state corresponding to the variable node; and a seventh processing module for determining target object information of the object at the first moment based on the target state corresponding to the variable node.

[0006] According to a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method for determining the state of an object as described in any of the above embodiments of the present disclosure.

[0007] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and the processor for reading the executable instructions from the memory and executing the instructions to implement the method for determining the state of an object as described in any of the above embodiments of the present disclosure.

[0008] A fifth aspect of the embodiments of the present disclosure provides a computer program product. When instructions in the computer program product are executed by a processor, the method for determining the state of an object provided by any of the above embodiments of the present disclosure is executed.

[0009] Based on the method, device, medium and equipment for determining the state of an object provided in the above-mentioned embodiments of the present disclosure, the object is observed respectively by a first sensor and a second sensor, and based on the object information obtained by the observations of the two sensors and a topological graph of the position, velocity and acceleration of the object, the states of the variable nodes of the topological graph are iteratively updated by a graph optimization algorithm to obtain the optimized target state of the variable nodes, and then the target object information (or target object state) of the object is determined based on the target state. Since the observation amount of the graph optimization process is determined by combining the observation results of the object by the first sensor and the second sensor, the characteristics of the first sensor and the second sensor can be taken into account, thereby improving the accuracy and effectiveness of the observation amount. Further, combined with the third object information of the object at the second moment, the state of the variable node is optimized by the graph optimization algorithm. Since the topological graph represents the constraint relationship between the state of the object at the second moment and the first moment, as well as the position, velocity, and acceleration of the object, and the object information at the second moment is accurate and effective object information that has been optimized before the first moment, the target object information of the target object at the first moment is solved by minimizing the objective function, so that the target object information of the object obtained by optimization at the first moment satisfies the actual driving conditions of the automatic mobile device and the observation conditions of the object by the first sensor and the second sensor, effectively improving the accuracy and effectiveness of the state of the object, and avoiding the use of a binocular camera to measure the distance and speed of the object, solving the problem that the strict calibration requirements between the two lenses of the binocular camera make it difficult to implement. In addition, compared with a monocular camera, the embodiment of the present disclosure can improve the observation accuracy of the observation results by using two sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is an exemplary application scenario of the method for determining the state of an object provided by the present disclosure;

[0011] Figure 2 is a flowchart of a method for determining the state of an object provided by an exemplary embodiment of the present disclosure;

[0012] Figure 3 is a flowchart of a method for determining a state of an object provided by another exemplary embodiment of the present disclosure;

[0013] Figure 4 is a schematic diagram of a principle of similar triangles provided by an exemplary embodiment of the present disclosure;

[0014] Figure 5 is a flowchart of a method for determining a state of an object provided by yet another exemplary embodiment of the present disclosure;

[0015] Figure 6 is a schematic diagram of a topology diagram provided by an exemplary embodiment of the present disclosure;

[0016] Figure 7 is a flowchart of a method for determining a state of an object provided by another exemplary embodiment of the present disclosure;

[0017] Figure 8 is a flowchart of a method for determining a state of an object provided by yet another exemplary embodiment of the present disclosure;

[0018] Figure 9 is a schematic diagram of a flow chart of determining the initial state of a variable node provided by an exemplary embodiment of the present disclosure;

[0019] Figure 10 is a schematic diagram of a flow chart of determining a first fusion observation state of an object provided by an exemplary embodiment of the present disclosure;

[0020] Figure 11 is a flowchart of a method for determining a state of an object provided by another exemplary embodiment of the present disclosure;

[0021] Figure 12 is a schematic diagram of a process for determining observation hysteresis indication information provided by an exemplary embodiment of the present disclosure;

[0022] Figure 13 is a structural diagram of an apparatus for determining a state of an object provided by an exemplary embodiment of the present disclosure;

[0023] Figure 14 is a structural diagram of an apparatus for determining a state of an object provided by another exemplary embodiment of the present disclosure;

[0024] Figure 15 This is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0026] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0027] Overview of the Disclosure

[0028] In the process of realizing the present disclosure, the inventors discovered that in the field of forward-looking obstacle state estimation, in the related technologies of object (or obstacle) perception, the obstacle state is usually determined based on the ranging and speed measurement of a monocular camera or a multi-camera ranging and speed measurement. However, the measurement accuracy of a monocular camera is low, making it difficult to cope with complex traffic scenarios. The computational complexity of a binocular camera is high, and in order to ensure stereo matching between the two cameras, the calibration requirements between the two lenses are very strict, which poses a great challenge to the implementation of binocular camera ranging and speed measurement.

[0029] Exemplary Overview

[0030] Figure 1 This is an exemplary application scenario of the method for determining the state of an object provided by the present disclosure. Figure 1 As shown, the autonomous mobile device takes a vehicle 11 as an example, and a first sensor 12 and a second sensor 13 are provided on the vehicle 11. The first sensor 12 and the second sensor 13 may be different types of sensors. For example, the first sensor 12 may be but is not limited to a camera, and the second sensor 13 may be but is not limited to a lidar. Alternatively, the first sensor 12 and the second sensor 13 may be cameras with different fields of view, and xoy represents the local coordinate system of the vehicle 11, or the self-vehicle coordinate system. While the vehicle 11 is driving on the road, the first sensor 12 and the second sensor 13 may be used to observe objects within the forward viewing range. The objects may include dynamic objects and / or static objects. In the figure, objects 14 and 15 are used as examples. The method for determining the state of an object according to an embodiment of the present disclosure is used to determine the target object information (or target object state) of the object at the first moment (for example, the current moment). Specifically, the method can determine first object information obtained by an autonomous mobile device observing an object via a first sensor at a first moment; determine second object information obtained by the autonomous mobile device observing the object via a second sensor at a first moment; determine a topological graph of the object's position, velocity, and acceleration; the topological graph includes variable nodes and edges representing constraint relationships between the variable nodes; determine an objective function based on the topological graph; determine the initial states of the variable nodes in the topological graph based on the third object information of the object at the second moment; the second moment is before the first moment; iteratively update the states of the variable nodes based on the first object information, the second object information, the initial states of the variable nodes, and the objective function to obtain the target states corresponding to the variable nodes; and determine the target object information of the object at the first moment based on the target states corresponding to the variable nodes. If there are multiple objects within the observation range, the target object information of any object can be determined according to the above steps.

[0031] Since the observation quantity of the graph optimization process is determined by combining the observation results of the first sensor and the second sensor on the object, the characteristics of the first sensor and the second sensor can be taken into account, and the accuracy and effectiveness of the observation quantity can be improved. Further combined with the third object information of the object at the second moment, the state of the variable node is optimized by the graph optimization algorithm. Since the topological graph represents the constraint relationship between the state of the object at the second moment and the first moment, as well as the position, velocity, and acceleration of the object, and the object information at the second moment is the accurate and effective object information optimized before the first moment, the target object information of the target object at the first moment is solved by minimizing the objective function, so that the target object information of the object obtained by optimization at the first moment meets the actual driving conditions of the automatic mobile device and the observation conditions of the object by the first sensor and the second sensor, effectively improving the accuracy and effectiveness of the object state, and avoiding the use of binocular cameras to measure the distance and speed of the object, solving the problem that the strict calibration requirements between the two lenses of the binocular camera make it difficult to implement. Compared with the monocular camera, the embodiment of the present disclosure can improve the observation accuracy of the observation results by using two sensors.

[0032] Exemplary Methods

[0033] Figure 2 This is a flow chart of a method for determining the state of an object provided by an exemplary embodiment of the present disclosure. The method for determining the state of an object provided by an embodiment of the present disclosure can be applied to electronic devices, such as edge devices of a computing platform (or terminal) of an autonomous mobile device. This embodiment can also be applied to a server, such as a cloud server, by communicating with the autonomous mobile device, obtaining sensor data from the autonomous mobile device, determining the state of the object on the server side, and then transmitting the target object information to the autonomous mobile device. Figure 2 As shown, the method for determining the state of an object according to an embodiment of the present disclosure may include the following steps:

[0034] Step 210 : Determine first object information obtained by the autonomous mobile device observing the object through a first sensor at a first moment.

[0035] The first moment can be any moment, for example, the current moment or any historical moment. In an intelligent driving scenario, the first moment is, for example, the current moment during the driving process of an autonomous mobile device (or intelligent agent). Autonomous mobile devices may include vehicles, robots, drones, etc. The first sensor can be any sensor configured on the autonomous mobile device as needed, for example, a camera, a lidar, or other sensors. Objects can include dynamic objects and / or static objects. Dynamic objects may include, but are not limited to, vehicles, cyclists, pedestrians, etc. Static objects may include, but are not limited to, cones, fences, traffic signs, etc. The first object information includes state information of the observed object, which may include, for example, one or more of the object's identification, type, location, size, direction, speed, and other information. The first object information of a dynamic object may be referred to as first dynamic object information, and the first object information of a static object may be referred to as first static object information. The number of objects observed by the first sensor at the first moment is determined by the environment in which the autonomous mobile device is located.

[0036] In the embodiment of the present disclosure, the method of the present disclosure may be described in detail by taking a dynamic object as an example, but is not limited to dynamic objects in practical applications.

[0037] In some optional embodiments, the first object information may be determined based on sensor data collected by the first sensor through a preconfigured perception algorithm model.

[0038] Step 220 : Determine second object information obtained by the autonomous mobile device observing the object through the second sensor at the first moment.

[0039] The second sensor is a sensor that has at least one different attribute from the first sensor. These different attributes may include, but are not limited to, different sensor types and different fields of view, resulting in different perception capabilities between the second and first sensors. For example, the first sensor may be a camera and the second sensor may be a lidar. The first sensor may have higher perception accuracy at close range, while the lidar may have stronger penetration in inclement weather. For another example, the first and second sensors may both be cameras, and the first sensor's field of view (FOV) may be larger than the second sensor's. In other words, the first sensor may be a wide-angle camera and the second sensor may be a narrow-angle camera. Wide-angle cameras may have higher perception accuracy for close-range objects, while narrow-angle cameras may have higher perception accuracy for distant objects. The second object information includes state information of objects observed by the second sensor. This state information may include, for example, one or more of the object's identity, type, location, size, direction, and speed. The second object information for dynamic objects may be referred to as second dynamic object information, while the second object information for static objects may be referred to as second static object information.

[0040] In some optional embodiments, the second object information may be determined based on sensor data collected by the second sensor through a preconfigured perception algorithm model.

[0041] In step 230 , a topological map of the position, velocity, and acceleration of the object is determined.

[0042] The topological graph includes variable nodes and edges representing the constraint relationships between variable nodes.

[0043] In some optional embodiments, the topology graph may include variable nodes corresponding to multiple moments, and the variable nodes may include position variable nodes, velocity variable nodes, and acceleration variable nodes. For example, the first moment is T m time, m is an integer greater than 1, and the topology graph includes time T0, time T1, ..., T m The position variable node, velocity variable node and acceleration variable node corresponding to each moment. i The position variable node, velocity variable node and acceleration variable node corresponding to the time are used to represent the object at T i The position variables, velocity variables, and acceleration variables to be optimized at each moment. The constraints between variable nodes must satisfy kinematics and state continuity at adjacent moments.

[0044] In some optional embodiments, a topological map of the object's position, velocity, and acceleration can be constructed based on the object's motion characteristics. For example, for a dynamic object, a topological map of the object's position, velocity, and acceleration can be constructed based on the object's motion characteristics. For a static object, the states of the velocity variable nodes and acceleration variable nodes in the constructed topological map can be preset or set to invalid nodes based on the static object's motion characteristics. The specific setting can be based on actual needs.

[0045] Step 240: Determine the objective function based on the topology graph.

[0046] Among them, the edges in the topological graph represent the constraint relationship between variable nodes. According to the constraint relationship, the residual term (or error term) is constructed, and the objective function is determined based on the error term.

[0047] Step 250: Determine the initial state of the variable node in the topology graph based on the third object information of the object at the second moment.

[0048] The second moment is before the first moment. For example, if the first moment is the current moment, the second moment is a historical moment. The third object information of the object at the second moment is the state information of the object at the second moment.

[0049] In some optional embodiments, the number of the second moments is multiple, and the specific number can be determined according to the number of moments when the object is actually observed. m At this moment, m is an integer greater than 1. m-1 Time, T m-2 Time, ..., T m-n If the object is observed at all moments, the number of the second moments can be n or a value less than n.

[0050] In some optional embodiments, the state of the object can only be optimized through the topological map when the object is observed at at least two times. When the initial state of the variable node is determined during the first optimization process, the third object information at the second time can be the state information of the object obtained based on the perception algorithm model or other means. In a non-first optimization process, the third object information can be the state information of the object after the previous optimization. The third object information can include the position, velocity, and acceleration of the object at the second time.

[0051] In some optional embodiments, for a variable node corresponding to the second moment, the initial state at the second moment can be determined based on the position, velocity, and acceleration in the third object information at the second moment. For a variable node corresponding to the first moment, the initial state at the first moment can be predicted based on the third object information at the second moment.

[0052] Step 260 , based on the first object information, the second object information, the initial state of the variable node and the objective function, iteratively update the state of the variable node to obtain the target state corresponding to the variable node.

[0053] Among them, the first object information and the second object information are used to determine the observed quantity in the iterative optimization process, and the initial state of the variable node is used to determine the state quantity in the iterative optimization process. The residual term in the objective function is a function of the state quantity and the observed quantity. By minimizing the objective function, a pre-configured optimization algorithm is used to iteratively update the state quantity to achieve iterative update of the state of the variable node until the iteration end condition is met, and the target state corresponding to the variable node is obtained. The target state corresponding to the position variable node is the target position of the object, the target state corresponding to the velocity variable node is the target velocity of the object, and the target state corresponding to the acceleration variable node is the target acceleration. The optimization algorithm may include but is not limited to the Gauss-Newton method, the stochastic gradient descent algorithm, the batch gradient descent algorithm, etc.

[0054] Step 270: Determine the target object information of the object at the first moment based on the target state corresponding to the variable node.

[0055] The variable nodes include the variable nodes corresponding to the second moment and the variable nodes corresponding to the first moment. The target state of the variable node corresponding to the first moment can be obtained from the target state corresponding to the variable node, and the target object information of the object at the first moment can be determined based on the target state at the first moment. For example, based on the target position of the position variable node, the target velocity of the velocity variable node, and the target acceleration of the acceleration variable node corresponding to the first moment obtained by graph optimization, the target position, target velocity, and target acceleration of the object at the first moment can be obtained.

[0056] The method for determining the state of an object provided by the embodiments of the present disclosure combines the observation results of the object by the first sensor and the second sensor to determine the observation quantity of the graph optimization process. This can take into account the characteristics of the first sensor and the second sensor, thereby improving the accuracy and effectiveness of the observation quantity. Further, the third object information of the object at the second moment is combined to optimize the state of the variable node through a graph optimization algorithm. Since the topological graph represents the constraint relationship between the state of the object at the second moment and the first moment, as well as the position, velocity, and acceleration of the object, and the object information at the second moment is the accurate and effective object information optimized before the first moment, the target object information of the target object at the first moment is solved by minimizing the objective function. The optimized target object information of the object at the first moment satisfies the actual driving conditions of the automatic mobile device and the observation conditions of the object by the first sensor and the second sensor, effectively improving the accuracy and effectiveness of the object state, avoiding the use of binocular cameras to measure the distance and speed of the object, and solving the problem that the strict calibration requirements between the two lenses of the binocular camera make it difficult to implement. Compared with a monocular camera, the embodiments of the present disclosure can improve the observation accuracy of the observation results by using two sensors.

[0057] Figure 3 FIG. 4 is a flowchart of a method for determining a state of an object provided by another exemplary embodiment of the present disclosure.

[0058] In some optional embodiments, in the above Figure 2 Based on the embodiment shown, Figure 3 As shown, determining the first object information obtained by the autonomous mobile device observing the object through the first sensor at the first moment in step 210 may include:

[0059] Step 2110: Determine a first observation position obtained by observing the object through a first sensor at a first moment.

[0060] The first observation position is the position of the object observed by the first sensor. The first observation position may include the horizontal coordinate and the vertical coordinate of the object, for example, may be expressed as (x1, y1).

[0061] In some optional embodiments, the first observation position may be a position in a local coordinate system or a global coordinate system. The local coordinate system may be the local coordinate system of the autonomous mobile device at the first moment, that is, a coordinate system with a preset position on the autonomous mobile device as the origin. The global coordinate system may be the world coordinate system or the coordinate system of the autonomous mobile device at the starting moment. Different coordinate systems may be converted according to corresponding conversion relationships. In order to more conveniently describe the relative relationship between the object and the autonomous mobile device, the embodiment of the present disclosure is described using the local coordinate system at the first moment as an example. The state of the object at other moments may be converted to the local coordinate system at the first moment so as to perform the relevant steps of the embodiment of the present disclosure in a unified coordinate system.

[0062] Step 2120 : Determine a first observed velocity of the object at the first moment based on a third observed position of the object obtained by the first sensor at the second moment.

[0063] The third observation position is obtained by observing the first sensor at the second moment.

[0064] In some optional embodiments, there are multiple second moments, and the first observed velocity of the object at the first moment can be obtained by fitting the third observed positions of the object at the multiple second moments.

[0065] In some optional embodiments, the observed positions of the object at different second moments in time can be unified into the same coordinate system to obtain a third observed position. For example, if the observed position observed at any moment is a position in a local coordinate system, the observed position of the object observed at the second moment in time is converted to the local coordinate system at the first moment in time to obtain the third observed position at the second moment in time. If the observed position is a position in a global coordinate system, no coordinate system conversion is required.

[0066] In some optional embodiments, the first observed velocity of the object at the first moment may be determined based on the third observed position of the object at the second moment and the first observed position of the object at the first moment. For example, the first observed velocity of the object at the first moment may be obtained by fitting the third observed positions of the object at multiple second moments and the first observed position of the object at the first moment.

[0067] Step 2130: Determine first object information based on the first observation position and the first observation speed.

[0068] After obtaining the first observation position and the first observation speed of the object at the first moment, the first observation position and the first observation speed may be determined as the first object information.

[0069] In an embodiment of the present disclosure, a first observation position of an object at a first moment is obtained by observing with a first sensor, and a first observation speed of the object at the first moment is obtained by fitting with a third observation position of the object at a second moment, which helps to improve the stability of the observation speed.

[0070] In some optional embodiments, determining the second object information obtained by the autonomous mobile device observing the object through the second sensor at the first moment in step 220 may include:

[0071] Step 2210: Determine a second observation position obtained by observing the object through a second sensor at the first moment.

[0072] Step 2220 : Determine a second observed velocity of the object at the first moment based on a fourth observed position of the object obtained by the second sensor at the second moment.

[0073] Step 2230: Determine second object information based on the second observation position and the second observation speed.

[0074] Among them, the specific operations of step 2210 to step 2230 are similar to the specific operations of the above-mentioned step 2110 to step 2130, and are not repeated here.

[0075] In an embodiment of the present disclosure, the second observation position of the object at the first moment is obtained by observing with the second sensor, and the second observation speed of the object at the first moment is obtained by fitting the fourth observation position of the object at the second moment, which helps to improve the stability of the observation speed.

[0076] In some optional embodiments, the first sensor is a first camera with a first field of view; the number at the second moment is one or more; the third observation position at the second moment includes observation positions determined based on at least one element of the object in the first image, at least one element includes a first height and / or first width of the area box of the object in the first image, and the first image is an image captured by the first camera at the second moment.

[0077] The first camera may also be referred to as the first camera head. Based on the first image captured by the first camera, an area frame of the object in the first image is obtained through object detection. Based on any element of the area frame and the physical size of the object, the distance between the object and the autonomous mobile device can be determined using similar triangles. For objects within the forward viewing range of the autonomous mobile device, the longitudinal coordinates of the object can be determined based on this distance. Based on the pixel distance of the area frame from the center point in the first image, the lateral coordinates of the object can be determined using similar triangles. Based on the longitudinal and lateral coordinates, a third observation position of the object is obtained.

[0078] In some optional embodiments, the longitudinal coordinate of the object may be determined based on the first height or the first width by similar triangles, and then the third observation position of the object may be determined based on the longitudinal coordinate and the transverse coordinate.

[0079] In some optional embodiments, the longitudinal coordinates of the object may be determined based on the first height and the first width, respectively, to obtain the longitudinal coordinates corresponding to the first height and the longitudinal coordinates corresponding to the first width. The longitudinal coordinates corresponding to the first height and the longitudinal coordinates corresponding to the first width are then fused to obtain the longitudinal coordinates of the object, and a third observation position of the object is determined based on the longitudinal coordinates and the transverse coordinates. Alternatively, one third observation position of the object is determined based on the longitudinal coordinates and the transverse coordinates corresponding to the first height, and another third observation position of the object is determined based on the longitudinal coordinates and the transverse coordinates corresponding to the first width.

[0080] In some optional embodiments, the first observation position of the object at the first moment can be determined based on at least one element of the object in the third image, which is similar to the third observation position described above and is not described in detail here. The third image is an image captured by the first camera at the first moment.

[0081] In some optional embodiments, Figure 4 FIG. 1 is a schematic diagram of a similar triangle provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, H represents the physical height of the object, h represents the height of the object in the image, such as the first height of the area frame at the second moment or the height of the area frame at the first moment, f represents the focal length of the camera, which is a pre-calibrated parameter, and d represents the distance between the object and the camera. According to h / H=f / d, the distance d between the object and the camera can be determined, and based on the external parameters of the camera relative to the autonomous mobile device, the distance between the object and the autonomous mobile device can be determined. The physical height of the object can be obtained by filtering or other means, which is not limited in the embodiments of the present disclosure.

[0082] In some optional embodiments, taking the third image captured at the first moment as an example, based on the third image, the area frame of the object in the third image is detected, and based on the height h of the area frame and the physical height H of the object, combined with the focal length f of the camera, an observation position pos of the object relative to the autonomous mobile device can be obtained. h Similarly, based on the width w of the area frame and the physical width W of the object, combined with the focal length f, another observation position pos of the object relative to the autonomous mobile device can be obtained. w . w and pos h Fusion, get the first observation position of the object.

[0083] Determining the first observed velocity of the object at the first moment based on the third observed position of the object obtained by the first sensor at the second moment in step 2120 may include:

[0084] Based on the third observation position at the second moment, determine the third observation position sequence corresponding to at least one element; based on the third observation position sequence corresponding to the element, determine the observation speed corresponding to the element; based on the observation speed corresponding to the element, determine the first observation speed of the object at the first moment.

[0085] Based on the third observation position at the second moment, the third observation position of any element at the second moment can be determined. The third observation positions of the same element at different moments form a third observation position sequence corresponding to that element. This third observation position sequence represents the change in the object's observation position over time. Therefore, based on this third observation position sequence, the observed velocity corresponding to that element can be fitted. The observed velocities corresponding to at least one element are fused to obtain the object's first observed velocity at the first moment. This fusion method can be weighted fusion or other fusion methods.

[0086] In some optional embodiments, a third observation position sequence corresponding to at least one element may be determined based on the third observation position at the second moment and the first observation position at the first moment. Specifically, the third observation position at the second moment and the first observation position at the first moment for the same element form a third observation position sequence, which is used to determine the first observed velocity of the object at the first moment.

[0087] In an embodiment of the present disclosure, a first observation speed of the object at the first moment is determined by integrating a third observation position sequence obtained by different elements of the area frame of the object in the image. Since different elements of the area frame represent the observation characteristics of the object in different directions, for example, the observability in the height direction and the width direction may be different, the observation results of multiple elements can complement each other, which helps to improve the accuracy and stability of the observation speed.

[0088] In some optional embodiments, determining the observed speed corresponding to the element based on the third observed position sequence corresponding to the element may include:

[0089] Based on the third observation position sequence corresponding to the element, a first straight line between the observation position and time corresponding to the element is fitted; based on the third observation position sequence and the first straight line, the third observation position sequence is filtered to obtain a filtered fifth observation position sequence; based on the fifth observation position sequence, a second straight line between the observation position and time corresponding to the element is fitted; based on the second straight line, the observation speed corresponding to the element is determined.

[0090] For any element, the third observation position sequence corresponding to that element represents the change in the object's observation position over time, and the slope of the first straight line between the observation position and time obtained by fitting the third observation position sequence represents the object's speed. After initially fitting the first straight line, the third observation position sequence can be further filtered based on the relationship between the third observation position sequence and the first straight line. For example, the third observation position in the third observation position sequence can be traversed to determine the deviation between the traversed third observation position and the first straight line. Outliers with deviations greater than a deviation threshold can be removed to obtain a fifth observation position sequence. The deviation threshold can be the average of the deviations between the third observation position in the third observation position sequence and the first straight line. Based on the filtered fifth observation position sequence, a second straight line between the observation position and time corresponding to the element is refitted. Based on the slope of the second straight line, the observation speed corresponding to the element is determined. Based on this, the observation speed corresponding to any element can be obtained. The observation speed corresponding to any element represents the speed of the object observed based on that element at the first moment.

[0091] In some optional embodiments, the fitting method of the first straight line and the second straight line may be least square fitting, and the fitting formula is expressed as follows:

[0092]

[0093] In formula (1), k is the slope of the fitted line, that is, the fitted observation speed, b is the intercept of the fitted line, n is the length of the fitting window, that is, the number of the second moment, t i is the time difference between the i-th second moment and the i-1-th second moment, pos i is the i-th third observation position.

[0094] In the embodiment of the present disclosure, the observation speed is obtained by fitting the third observation position sequence of the same element at multiple second moments, which can ensure the stability of the observation speed. Moreover, after the initial fitting, the third observation position sequence is filtered based on the fitted straight line to reduce the adverse effects of outliers on the observation speed, which can further improve the effectiveness of the observation position sequence and thus further improve the effectiveness of the observation speed.

[0095] In some optional embodiments, determining a first observed velocity of the object at a first moment based on the observed velocity corresponding to the element may include:

[0096] Based on the observed speed corresponding to the element and the third speed of the object at the third moment, determine the first speed residual of the observed speed corresponding to the element and the third speed; the third moment is before the first moment; based on the first speed residual, determine the first speed observation confidence corresponding to the element; based on the observed speed corresponding to the element and the first speed observation confidence, determine the first observed speed of the object at the first moment.

[0097] The third moment is a moment before the first moment, that is, the moment closest to the first moment. The third velocity of the object at the third moment is the velocity optimized during the iterative optimization process at the third moment. The observed velocity corresponding to any element represents the velocity of the object observed based on that element. There may be a certain deviation between the observed velocity and the third velocity, and the absolute value or square of this deviation is used as the first velocity residual. For any element, the first velocity residual corresponding to that element can be determined based on the difference between the observed velocity and the third velocity.

[0098] In some optional embodiments, the first speed observation confidence corresponding to any element represents the credibility of the observed speed obtained based on the observation of the element. The first speed observation confidence corresponding to the element can be determined based on the mapping relationship between the preset speed observation confidence and the first speed residual. The setting principle of the mapping relationship is that the larger the residual, the lower the first speed observation confidence. Exemplarily, taking the two elements of height and width as examples, the first speed residual corresponding to the height is expressed as σ1, and the first speed residual corresponding to the width is expressed as σ2, then the first speed observation confidence f1 corresponding to the height and the first speed observation confidence f2 corresponding to the width can be expressed as follows:

[0099]

[0100] In formula (2), if the first velocity residual corresponding to the height is Greater than the first velocity residual corresponding to the width Then the first velocity observation confidence level corresponding to the height is less than the first velocity observation confidence level corresponding to the width.

[0101] In some optional embodiments, after obtaining the first speed observation confidence corresponding to at least one element, the first observed speed of the object at the first moment can be obtained through weighted fusion based on the observed speed corresponding to the at least one element and the first speed observation confidence.

[0102] In some optional embodiments, after obtaining the first velocity observation confidence corresponding to at least one element, the first observed velocity of the object at the first moment can be obtained by weighted fusion based on the observed velocity corresponding to at least one element and the first velocity observation confidence. For example, the observed velocity corresponding to the height is represented by V h1 , the observed velocity corresponding to the width is expressed as V w1 , then the first observed velocity V1 of the object at the first moment can be expressed as follows:

[0103] V1=f1*V h1 +f2*V w1 Formula (3)

[0104] In formula (3), f1 represents the first velocity observation confidence level corresponding to the height, and f2 represents the first velocity observation confidence level corresponding to the width.

[0105] In an embodiment of the present disclosure, the speed observation confidences corresponding to different elements are determined by the residuals between the observed speeds corresponding to different elements and the third speed at the third moment, which is used to perform weighted fusion of the observed speeds corresponding to different elements. Since the third speed at the third moment is the optimized speed of the previous moment, it has higher accuracy, and the time interval between two adjacent moments is short, the speed change of the object is small. Taking the third speed as a reference can ensure the validity of the confidences of the observed speeds corresponding to different elements, and then the observed speeds corresponding to the elements are fused by the speed observation confidences corresponding to different elements to obtain the first observed speed of the object at the first moment, which can ensure the validity of the first observed speed of the object at the first moment.

[0106] In some optional embodiments, the second sensor is a second camera with a second field of view (FOV); the second FOV is smaller than the first FOV corresponding to the first sensor. The fourth observation position at any second moment includes observation positions determined based on at least one element of the object in the second image, where the at least one element includes a second height and / or a second width of a region frame of the object in the second image, and the second image is an image captured by the second camera at the second moment.

[0107] Among them, the second field of view angle of the second camera is smaller than the first field of view angle of the first sensor (i.e., the first camera), which means that relative to the second camera, the first camera is a wide-angle camera (or a large-eye camera) and the second camera is a narrow-angle camera (or a small-eye camera). Due to the different focal lengths of the large and small-eye cameras, for objects at close range, the wide-angle camera images more clearly, the area frame of the detected object (or the detection frame) has higher accuracy and stability, and the observed observation position has higher confidence. For objects at a distance, the narrow-angle camera images more clearly, the area frame of the detected object has higher accuracy and stability, and the observed observation position has higher confidence. The combination of wide-angle cameras and narrow-angle cameras ensures that the observation position of the object at different distances can have a reliable source, thereby ensuring the accuracy and effectiveness of the observation position over the entire distance range.

[0108] The method of obtaining the fourth observation position is similar to that of the third observation position, and will not be described in detail here.

[0109] In some optional embodiments, the specific operation of determining the second observation position in step 2210 can refer to the specific operation of determining the first observation position by similar triangles in the aforementioned embodiment, and will not be repeated here.

[0110] Determining the second observed velocity of the object at the first moment based on the fourth observed position of the object obtained by the second sensor at the second moment in step 2220 may include:

[0111] Based on the fourth observation position at the second moment, determine the fourth observation position sequence corresponding to at least one element; based on the fourth observation position sequence corresponding to the element, determine the observation speed corresponding to the element; based on the observation speed corresponding to the element, determine the second observation speed of the object at the first moment.

[0112] It should be noted that the specific operation of determining the second observed velocity of the object at the first moment in this step is similar to the specific operation of determining the first observed velocity of the object at the first moment in the above embodiment. The operation of determining the second observed velocity of the object at the first moment can be achieved by replacing the third observation position in the above embodiment with the fourth observation position, and replacing the third observation position sequence with the fourth observation position sequence. A detailed description is omitted here.

[0113] In some optional embodiments, a fourth observation position sequence corresponding to at least one element may be determined based on the fourth observation position at the second moment and the second observation position at the first moment.

[0114] In an embodiment of the present disclosure, a second observation speed of the object at the first moment is determined by integrating a fourth observation position sequence obtained by integrating different elements of the area frame of the object in the image. Since different elements of the area frame represent the observation characteristics of the object in different directions, for example, the observability in the height direction and the width direction may be different, the observation results of multiple elements can complement each other, which helps to improve the accuracy and stability of the observation speed.

[0115] In some optional embodiments, determining the observed speed corresponding to the element based on the fourth observed position sequence corresponding to the element may include:

[0116] Based on the fourth observation position sequence corresponding to the element, a third straight line of the observation position and time corresponding to the element is fitted; based on the fourth observation position sequence and the third straight line, the fourth observation position sequence is filtered to obtain a filtered sixth observation position sequence; based on the sixth observation position sequence, a fourth straight line of the observation position and time corresponding to the element is fitted; based on the fourth straight line, the observation speed corresponding to the element is determined.

[0117] The specific operation of this step is similar to the specific operation of determining the observation speed corresponding to the element based on the third observation position sequence corresponding to the element in the above embodiment. Replacing the third observation position sequence, the first straight line, the fifth observation position sequence, and the second straight line in the above embodiment with the fourth observation position sequence, the third straight line, the sixth observation position sequence, and the fourth straight line, respectively, can determine the observation speed corresponding to the element observed by the second camera. For details, please refer to the above embodiment and will not be repeated here.

[0118] In the embodiment of the present disclosure, the observation speed is obtained by fitting the fourth observation position sequence of the same element at multiple second moments, which can ensure the stability of the observation speed. Moreover, after the initial fitting, the fourth observation position sequence is filtered based on the fitted straight line to reduce the adverse effects of outliers on the observation speed, which can further improve the effectiveness of the observation position sequence and thus further improve the effectiveness of the observation speed.

[0119] In some optional embodiments, determining the second observed velocity of the object at the first moment based on the observed velocity corresponding to the element may include:

[0120] Based on the observed speed corresponding to the element and the third speed of the object at the third moment, determine the second speed residual of the observed speed corresponding to the element and the third speed; the third moment is before the first moment; based on the second speed residual, determine the second speed observation confidence corresponding to the element; based on the observed speed corresponding to the element and the second speed observation confidence, determine the second observed speed of the object at the first moment.

[0121] The specific operation for determining the second observed velocity of the object at the first moment in this step is similar to that of the aforementioned embodiment. The first velocity residual and the first velocity observation confidence level in the aforementioned embodiment can be replaced with the second velocity residual, the second velocity observation confidence level, and the second observed velocity to determine the second observed velocity of the object at the first moment. Furthermore, unlike the aforementioned embodiment, the observed velocity corresponding to the element here is the observed velocity corresponding to the element in the area frame observed by the second camera.

[0122] In an embodiment of the present disclosure, the speed observation confidence corresponding to different elements is determined by the residual between the observed speed corresponding to different elements and the third speed at the third moment, which is used to perform weighted fusion of the observed speeds corresponding to the elements. Since the third speed at the third moment is the optimized speed of the previous moment, it has higher accuracy, and the time interval between two adjacent moments is short, the speed change of the object is small. Taking the third speed as a reference can ensure the validity of the confidence of the observed speed corresponding to different elements, and then the observed speeds corresponding to the elements are fused by the speed observation confidence corresponding to different elements to obtain the second observed speed of the object at the first moment, which can ensure the validity of the second observed speed of the object at the first moment.

[0123] Figure 5 FIG. 4 is a flowchart of a method for determining a state of an object provided by yet another exemplary embodiment of the present disclosure.

[0124] In some optional embodiments, based on any of the above embodiments, Figure 5 As shown, the first sensor is a first camera with a first field of view; the second sensor is a second camera with a second field of view; and the first field of view is greater than the second field of view.

[0125] Determining the first object information obtained by the autonomous mobile device observing the object through the first sensor at the first moment in step 210 may include:

[0126] Step 2101: Determine a first area frame of an object in a first image captured by a first camera.

[0127] The first region frame is a boundary frame of the region occupied by the object in the first image. Object detection can be performed on the first image using a target detection algorithm to obtain the first region frame of the object in the first image.

[0128] In some optional embodiments, if the object is a vehicle, the first area frame may include at least one of the observed vehicle's full frame, rear frame, and side frame, and any area frame may include at least one element of height and width.

[0129] Step 2102: Determine first object information based on the first area frame and first parameters of the first camera.

[0130] The first parameter includes an intrinsic parameter and an extrinsic parameter of the first camera. The first object information may include, but is not limited to, a first observation position and a first observation speed of the object.

[0131] In some optional embodiments, a first observed velocity of the object at the first moment can be determined based on a first observed position of the object obtained by the first camera and a third observed position of the object obtained by the first sensor at the second moment. For specific operations, see the aforementioned embodiments. First object information is determined based on the first observed position and the first observed velocity.

[0132] In some optional embodiments, determining the second object information obtained by the autonomous mobile device observing the object through the second sensor at the first moment in step 220 may include:

[0133] Step 2201: Determine a second area frame of the object in a second image captured by a second camera.

[0134] Step 2202 : Determine second object information based on the second area frame and second parameters of the second camera.

[0135] The second parameters include the intrinsic and extrinsic parameters of the second camera. The specific operations of steps 2201 to 2202 are similar to those of steps 2101 to 2102 in the aforementioned embodiment. Replacing the first camera, first image, first region frame, first parameter, and first object information in the aforementioned embodiment with the corresponding second camera, second image, second region frame, second parameter, and second object, respectively, can implement the specific operations for determining the second object information. These operations are not further described here.

[0136] In the embodiments of the present disclosure, objects are observed separately by cameras with different fields of view to obtain first and second object information corresponding to the objects. Since cameras with large fields of view have higher accuracy and stability for close-range objects, and cameras with small fields of view have higher accuracy and stability for distant objects, the combined use of cameras with different fields of view provides a reliable source for position observation of objects at different distances, thereby ensuring the accuracy and effectiveness of the observed position over the entire range of distances. Compared to monocular cameras, this can effectively improve the observation accuracy of objects. Compared to binocular cameras, this can avoid the challenges posed by the calibration requirements between the two lenses in achieving distance and speed measurement, and can effectively reduce computational complexity.

[0137] In some optional embodiments, determining the first object information based on the first area frame and the first parameter of the first camera in step 2102 may include:

[0138] Determine a first distance of the object based on a first height and a first parameter of the first area box; determine a second distance of the object based on a first width and the first parameter of the first area box; determine a first horizontal coordinate of the object based on a pixel distance between the first area box and a center point of the first image, and the first parameter; and determine first object information based on the first distance, the second distance, and the first horizontal coordinate.

[0139] The first distance is the distance between the object corresponding to the first height and the first camera, and the second distance is the distance between the object corresponding to the first width and the first camera. Since the characteristics of the object in the height direction and the width direction may be different, the obtained first distance and second distance may be different.

[0140] In some optional embodiments, the first distance of the object can be determined based on the first height and the first parameter of the first area box according to the principle of similar triangles. Similarly, the second distance of the object can be determined based on the first width and the first parameter of the first area box according to the principle of similar triangles. The pixel distance between the first area box and the center point of the first image can characterize the situation where the object deviates from the optical axis of the first camera. Therefore, based on the pixel distance and the first parameter, the first horizontal coordinate of the object in the camera coordinate system is determined by the principle of similar triangles. Furthermore, based on the first distance, the second distance and the first horizontal coordinate, combined with the external parameters of the first camera relative to the autonomous mobile device, the longitudinal coordinate and the horizontal coordinate of the object in the local coordinate system of the autonomous mobile device can be determined, and the longitudinal coordinate and the horizontal coordinate can be used as the first observation position of the object.

[0141] In some optional embodiments, the observation confidences corresponding to the first distance and the second distance can be determined based on the variances corresponding to the height and width of the area box in the image of the object at the second moment, respectively. Based on the observation confidences corresponding to the first distance and the second distance, respectively, the first distance and the second distance are weightedly fused to obtain the fused distance, and then the first observation position of the object is determined based on the fused distance and the first horizontal coordinate.

[0142] In some optional embodiments, the second time includes T m-n Time, T m-n+1 Time..., T m-2 Time, T m-1 moment, the first moment is T m At the n second moments, in the images captured by the first camera, the heights of the object's area frame are represented by h1, h2, ..., h n , determine the height mean based on the heights of n second moments, and determine the height variance h based on the heights of n second moments and the height mean std Similarly, the width variance w can be determined based on the width of the area frame at n second moments. std Based on the height variance and width variance, the observation confidence corresponding to the height and the observation confidence corresponding to the width are determined. The observation confidence corresponding to the height is used as the observation confidence corresponding to the first distance, and the observation confidence corresponding to the width is used as the observation confidence corresponding to the second distance. The principle of determining the observation confidence is that the larger the variance, the smaller the observation confidence.

[0143] In some optional embodiments, the fusion distance represents the longitudinal coordinate of the object in the camera coordinate system of the first camera. The longitudinal coordinate in the camera coordinate system can be determined based on the fusion distance, and the longitudinal coordinate and the first lateral coordinate in the camera coordinate system can be converted to the local coordinate system of the autonomous mobile device to obtain the first observation position of the object, and the first object information is determined based on the first observation position.

[0144] In some optional embodiments, when determining the first distance and the second distance, taking the first distance as an example, the first area frame can be projected onto the camera coordinate system of the first camera based on the first parameter of the first camera to obtain an initial distance. Based on the initial distance, the initial physical height of the object can be determined using similar triangles. The initial physical height is then filtered, such as by a Kalman filter, to obtain a stable physical height H of the object. The first distance can then be determined using similar triangles based on the physical height H, the first height, and the focal length f in the first parameter. Alternatively, after obtaining the distance using similar triangles, the distance can be further filtered to obtain a more stable first distance.

[0145] In some optional embodiments, determining the second object information based on the second area frame and the second parameter of the second camera in step 2202 may include:

[0146] Determine a third distance of the object based on the second height and second parameter of the second area box; determine a fourth distance of the object based on the second width and second parameter of the second area box; determine a second horizontal coordinate of the object based on the pixel distance between the second area box and the center point of the second image, and the second parameter; and determine second object information based on the third distance, the fourth distance, and the second horizontal coordinate.

[0147] Among them, the specific operations of determining the third distance, the fourth distance and the second horizontal coordinate are similar to the specific operations of determining the first distance, the second distance and the first horizontal coordinate. Furthermore, based on the third distance, the fourth distance and the second horizontal coordinate, the specific operations of determining the second object information are similar to the specific operations of determining the first object information based on the first distance, the second distance and the first horizontal coordinate. They will not be repeated here.

[0148] In the embodiments of the present disclosure, the observation distance of the object is determined separately by different elements of the area frame of the object in the image, namely the height and width of the area frame. Then, the observation distances corresponding to the different elements can be fused in combination with the observation confidence of the different elements, which helps to further improve the accuracy and effectiveness of the object information observed by any camera.

[0149] In some optional embodiments, Figure 6 Schematic diagram of a topological diagram provided by an exemplary embodiment of the present disclosure. Figure 6As shown, taking a dynamic object as an example, the topology graph includes a position variable node 71, a velocity variable node 72, and an acceleration variable node 73, as well as edges between the variable nodes. Figure 6 The circular nodes in the figure, any type of variable node includes nodes at multiple moments. The edges between variable nodes may include edges between speed variable nodes at different moments, edges between acceleration variable nodes at different moments, edges between speed variable nodes and acceleration variable nodes, etc. The edges represent the constraint relationship between variable nodes. In the figure, taking moments T0, T1, ..., T4 as examples, the gray square pointed to by 74 represents the speed after the last optimization, and the gray square pointed to by 75 represents the acceleration after the last optimization. Taking the optimization process at moment T4 as an example, at moment T4, there is no speed and acceleration after the last optimization. The first camera observes the object to obtain the observation position sequence corresponding to the height of the area frame and the observation position sequence corresponding to the width of the area frame. The second camera observes to obtain the observation position sequence corresponding to the height of the area frame and the observation position sequence corresponding to the width of the area frame. The field of view of the first camera is greater than the field of view of the second camera. For any of the first and second cameras, based on the observation position sequence corresponding to the two elements of the area frame height and area frame width, the observation velocity corresponding to at least one element can be obtained through least squares fitting. Then, the observation velocity corresponding to at least one element can be fused to obtain the observation velocity of the object under the camera, such as the first observation velocity and the second observation velocity mentioned above. The first observation position, the first observation velocity, the second observation position, and the second observation velocity can be used to determine the observation quantity, which can be used as a constraint in the optimization process and is expressed as Figure 6 b1 in , i.e., the fusion observation velocity constraint, is used to construct some residual terms in the objective function.

[0150] In some optional embodiments, based on any of the above embodiments, Figure 6 As shown, the constraint relationship between variable nodes includes but is not limited to at least one of the following sub-constraint relationships:

[0151] The first sub-constraint relationship between the velocity variable node and the corresponding fused observation velocity; the second sub-constraint relationship between the velocity variable node at any moment and the velocity variable node and acceleration variable node at the previous moment that satisfies the preset motion model; the third sub-constraint relationship between the position variable node at any moment and the position variable node at the previous moment, the first optimized velocity of the velocity variable node at the previous moment, and the first optimized acceleration of the acceleration variable node at the previous moment; the first optimized velocity of the velocity variable node and the first optimized acceleration of the acceleration variable node at any moment are the velocity and acceleration of the velocity variable node and the acceleration variable node after the previous graph optimization; the fourth sub-constraint relationship that the accelerations of adjacent acceleration variable nodes are equal; the fifth sub-constraint relationship that the velocities of adjacent velocity variable nodes are equal, which is used to suppress the amplitude of acceleration; the sixth sub-constraint relationship that the velocity variable node is equal to the first optimized velocity of the velocity variable node; the seventh sub-constraint relationship that the acceleration variable node is equal to the first optimized acceleration of the acceleration variable node.

[0152] The preset motion model may be, but is not limited to, a uniform acceleration model, a variable acceleration model, etc. The first optimization speed refers to the speed after the last (or previous) optimization process, such as Figure 6 74. The previous optimization process refers to the optimization process when the state of the object is determined at a time (e.g., time T3) before the first time (e.g., time T4 in the figure). The first optimized acceleration refers to the acceleration after the previous optimization process.

[0153] In some optional embodiments, for any moment, the first sub-constraint relationship is the sub-constraint relationship between the velocity variable node at that moment and the fused observation velocity at that moment, see Figure 6 b1 in , that is, the deviation between the state of the velocity variable node and the fused observation velocity should be small, based on which the residual term corresponding to the first sub-constraint relationship can be constructed.

[0154] The second sub-constraint relationship is the sub-constraint relationship between the velocity variable node at any moment and the velocity variable node and acceleration variable node at the previous moment, which should satisfy the preset motion model, or the sub-constraint relationship between the velocity change corresponding to the velocity variable node at any moment and the acceleration variable node at the previous moment, see Figure 6 b2 in the figure. The constraint between the velocity variable node at time T1 and the velocity variable node and acceleration variable node at time T0 that satisfies the preset motion model means that the velocity at time T1 (which can be called the estimated velocity) is determined based on the state of the velocity variable node at time T0, the state of the acceleration variable node, and time, according to the relationship between velocity and acceleration in the preset motion model. The deviation between the estimated velocity at time T1 and the state of the velocity variable node at time T1 should be small. Based on this, the residual term corresponding to the second sub-constraint relationship can be constructed.

[0155] The third sub-constraint is the sub-constraint between the state of the position variable node at any moment and the previously optimized velocity and acceleration at the previous moment, see Figure 6 b3 in the figure, for the position variable node at time T1, based on the state of the position variable node at time T0, the speed and acceleration after the previous optimization, and according to the relationship between position, speed and acceleration in the preset motion model, the estimated position at time T1 can be determined, that is, the position change can be determined based on the first optimized speed and the first optimized acceleration at time T0, and the estimated position at time T1 is determined based on the state and position change of the position variable node at time T0. The deviation between the estimated position and the state of the position variable node at time T1 should be small, based on which the residual term corresponding to the third sub-constraint relationship can be constructed.

[0156] The fourth sub-constraint is the sub-constraint that the states of the acceleration variable nodes at two adjacent moments are equal, see Figure 6 In b4, since the time interval between two adjacent moments is small, in the optimization process, it is considered that T i+1 The state of the acceleration variable node at time T i The states of the acceleration variable nodes at the time instant are equal, i = 1, 2, .... Based on this, the residual term corresponding to the fourth sub-constraint relationship can be constructed.

[0157] The fifth sub-constraint is the sub-constraint that the states of the velocity variable nodes at two adjacent moments are equal, see Figure 6 b5 in [ 1 ] indicates that since the time interval between two adjacent moments is small, the velocity change should not be too large. Based on this, the residual term corresponding to the fifth sub-constraint can be constructed. It should be noted that this fifth sub-constraint is used to suppress the amplitude of acceleration, not to ensure that the states of two adjacent velocity variable nodes are equal. This can be achieved by setting a smaller weight for the residual term corresponding to the fifth sub-constraint. That is, this residual term has a smaller weight in the objective function, thus suppressing the acceleration amplitude.

[0158] The sixth sub-constraint is the sub-constraint that the speed variable node is equal to the speed after the previous optimization corresponding to the speed variable node. Figure 6 b6 in , i.e., the state of the speed variable node should have a small change relative to the speed after the last optimization. Based on this, the residual term corresponding to the sixth sub-constraint relationship can be constructed.

[0159] The seventh sub-constraint is the sub-constraint that the acceleration variable node is equal to the acceleration after the previous optimization corresponding to the acceleration variable node. Figure 6b7 in , that is, at any moment when the acceleration and velocity after the previous optimization exist, such as any moment from T0 to T3, the deviation of the state of the velocity variable node at that moment relative to the velocity after the previous optimization corresponding to that moment (i.e., the first optimized velocity) should be small, and the residual term corresponding to the seventh sub-constraint relationship is constructed based on this moment.

[0160] The state of any variable node starts from the initial state and is continuously updated during the iterative optimization process until the value of the objective function meets the iteration end condition.

[0161] It should be noted that, when constructing the residual term, for the sub-constraint relationship involving the last optimized speed and / or the last optimized acceleration, if the variable node at a certain moment does not have the last optimized speed and acceleration, the state of the variable node at that moment does not participate in constructing the residual term, such as Figure 6 b6, b7, the velocity variable node and the acceleration variable node at time T4 do not participate in constructing the residual terms corresponding to b6 and b7.

[0162] In the embodiments of the present disclosure, multiple sub-constraint relationships are established through the state after the previous optimization, the current state to be optimized, the observed state, etc. corresponding to the variable node, which are used to construct the residual term of the objective function. This allows the state of the object at the first moment to be solved through graph optimization, so that the state of the object satisfies the constraints of the preset motion model, observation state, etc., thereby ensuring the validity of the object's state.

[0163] Figure 7 FIG. 4 is a flowchart of a method for determining a state of an object provided by another exemplary embodiment of the present disclosure.

[0164] In some optional embodiments, based on any of the above embodiments, Figure 7 As shown, determining the objective function based on the topological graph in step 240 may include:

[0165] Step 2410: Based on the sub-constraint relationship in the constraint relationship between the variable nodes, determine the residual term corresponding to the sub-constraint relationship.

[0166] The sub-constraints may include one or more of the seven aforementioned sub-constraints. To further ensure the validity of the object's state, the seven sub-constraints may be comprehensively considered to determine the residual terms corresponding to each of the seven sub-constraints. The residual term corresponding to any sub-constraint may include residual sub-terms corresponding to multiple time instants.

[0167] In some optional examples, for the first sub-constraint relationship, taking the example that the time window of the residual term includes four time moments T1, T2, T3, and T4, the residual term corresponding to the first sub-constraint relationship may include residual sub-terms corresponding to the multiple time moments, respectively. The residual term e1 corresponding to the first sub-constraint relationship may be expressed as follows:

[0168]

[0169] In formula (4), j represents T j Moment, ω j Indicates T j The confidence weight corresponding to the moment, v j (var) represents T j The state of the velocity variable node at time v j (fus) means T j The fused observation speed at the time instant, that is, var represents the variable and fus represents the fused observation. The time window of the residual term is not limited to this example.

[0170] In some optional examples, the residual term e2 corresponding to the second sub-constraint relationship can be expressed as follows:

[0171]

[0172] In formula (5), j represents T j Moment, ω j Indicates T j The confidence weight corresponding to the moment, v j (var) represents T j The state of the velocity variable node at time a j-1 (var) represents T j-1 The state of the acceleration variable node at time t j Indicates T j Moment and T j-1 The time interval between moments.

[0173] In some optional examples, the residual term e3 corresponding to the third sub-constraint relationship can be expressed as follows:

[0174]

[0175] In formula (6), j represents T j Moment, ω j Indicates T j The confidence weight corresponding to the moment, p j (var) represents T j The state of the position variable node at the moment, v j-1 (opt) means T j-1The first optimization speed corresponding to the moment, a j-1 (opt) means T j-1 The first optimized acceleration corresponding to the moment, t j Indicates T j Moment and T j-1 The time interval between moments, p j (est) indicates that the j-1 The first optimized speed, first optimized acceleration and time interval determined by T j Estimated location at the moment. p j-1 (var) represents T j-1 The state of the position variable node at the moment.

[0176] Optionally, in determining T j The estimated position at time p j (est), you can j-1 (opt) and v j The average value of (opt) replaces v j-1 (opt), a j-1 (opt) and a j The average value of (opt) replaces a j-1 (opt), to determine T j The estimated position at time p j (est).

[0177] In some optional examples, the residual term e4 corresponding to the fourth sub-constraint relationship can be expressed as follows:

[0178]

[0179] In formula (7), j represents T j Moment, ω j Indicates T j The confidence weight corresponding to the moment, a j (var) represents T j The state of the acceleration variable node at the moment, a j-1 (var) represents T j-1 The state of the acceleration variable node at time instant.

[0180] In some optional examples, the residual term e5 corresponding to the fifth sub-constraint relationship can be expressed as follows:

[0181]

[0182] In formula (8), j represents T j Moment, ω j Indicates T j The confidence weight corresponding to the moment, vj (var) represents T j The state of the velocity variable node at time v j-1 (var) represents T j-1 The state of the velocity variable node at time instant.

[0183] In some optional embodiments, the residual term e6 corresponding to the sixth sub-constraint relationship can be expressed as follows:

[0184]

[0185] In formula (9), j represents T j Moment, ω j Indicates T j The confidence weight corresponding to the moment, v j (var) represents T j The state of the velocity variable node at time v j (opt) means T j The first optimized speed corresponding to the moment.

[0186] In some optional examples, the residual term e7 corresponding to the seventh sub-constraint relationship can be expressed as follows:

[0187]

[0188] In formula (10), j represents T j Moment, ω j Indicates T j The confidence weight corresponding to the moment, a j (var) represents T j The state of the acceleration variable node at the moment, a j (opt) means T j The first optimized acceleration corresponding to the moment.

[0189] It should be noted that in the residual terms corresponding to different sub-constraint relationships, the confidence weight ω j They can be set separately according to actual needs. For example, the confidence weight in e1 can be determined based on the speed observation confidence of the aforementioned embodiment. The confidence weights in e2 to e5 can be preset empirical values. Among them, the confidence weight in e2 can be set to a larger value; the confidence weight in e5 can be set to a smaller value to suppress the amplitude of acceleration; the confidence weights in e6 and e7 can initially be preset empirical values ​​and can be adjusted during the optimization process.

[0190] Step 2420: Determine the objective function based on the residual term.

[0191] Wherein, at least one residual term is used as a part of the objective function, and the sum of at least one residual term is used as the objective function.

[0192] J=e1+e2+e3+e4+e5+e6+e7 Formula (11)

[0193] In formula (11), e i This is only an example of the objective function. In practical applications, the objective function may include one or more of the seven residual terms mentioned above. Optionally, other residual terms may be expanded according to actual needs.

[0194] In the embodiment of the present disclosure, the residual term of the objective function is determined according to the sub-constraint relationship to guide the optimization of the state of the variable node, so as to facilitate solving the optimal state through the optimization algorithm.

[0195] Figure 8 FIG. 4 is a flowchart of a method for determining a state of an object provided by yet another exemplary embodiment of the present disclosure.

[0196] In some optional embodiments, based on any of the above embodiments, Figure 8 As shown, step 250 of determining the initial state of the variable node in the topology graph based on the third object information of the object at the second moment may include:

[0197] Step 2510: predict a first predicted state of the object at a first moment based on the third object information.

[0198] Among them, the third object information represents the optimized state of the object at the second moment with high accuracy. Based on the third object information, a preconfigured prediction algorithm or model can be used to predict the first predicted state of the object at the first moment. The first predicted state may include one or more of the predicted position, predicted speed and predicted acceleration of the object at the first moment. For example, the first predicted state may include the first predicted position and the first predicted speed. The predicted acceleration may be the same as the optimized acceleration at the second moment, or the predicted acceleration may be a preset value, such as 0 or other values.

[0199] Step 2520: Determine the initial state of the variable node based on the third object information and the first predicted state.

[0200] The state of the object in the third object information can be used as the initial state of the variable node at the second moment, and the first predicted state can be used as the initial state of the variable node at the first moment.

[0201] In some optional embodiments, there are multiple second moments, and the third object information of the object at multiple second moments can represent the movement trend of the object. Predicting the state of the first moment based on the third object information can ensure that the first predicted state has a certain degree of accuracy.

[0202] In an embodiment of the present disclosure, predicting the predicted state of an object at a first moment based on the third object information at a second moment can be used to determine the initial state at the first moment. Since the third object information represents the optimized state of the object at the second moment with high accuracy, the predicted state at the first moment has a certain accuracy, thereby improving the effectiveness of the initial state of the variable node.

[0203] Figure 9 4 is a flowchart of determining the initial state of a variable node provided by an exemplary embodiment of the present disclosure.

[0204] In some optional embodiments, such as Figure 9 As shown, step 2520 of determining the initial state of the variable node based on the third object information and the first predicted state may include:

[0205] Step 25210: Determine a first position, a first velocity, and a first acceleration of the object at a second moment based on the third object information.

[0206] The third object information includes the optimized position, velocity, and acceleration of the object at the second moment. Therefore, the optimized position, velocity, and acceleration can be obtained from the third object information as the first position, first velocity, and first acceleration, respectively.

[0207] Step 25220: Determine a first predicted position and a first predicted velocity of the object at a first moment based on the first predicted state.

[0208] The first predicted state may include a first predicted position and a first predicted speed. Therefore, the first predicted position and the first predicted speed may be acquired from the first predicted state.

[0209] Step 25230, based on the first position of the object at the second moment and the first predicted position of the object at the first moment, determine the initial positions of the position variable nodes corresponding to the second moment and the first moment respectively.

[0210] Among them, the first position of the object at any second moment is determined as the initial position (or initial state) of the position variable node corresponding to the second moment, and the first predicted position of the object at the first moment is determined as the initial position of the position variable node corresponding to the first moment, thereby obtaining the initial states of the position variable nodes corresponding to the second moment and the first moment respectively.

[0211] Step 25240: Based on the first speed of the object at the second moment and the first predicted speed of the object at the first moment, determine the initial speeds of the speed variable nodes corresponding to the second moment and the first moment respectively.

[0212] Among them, the first speed of the object at any second moment is determined as the initial speed (or initial state) of the speed variable node corresponding to the second moment, and the first predicted speed of the object at the first moment is determined as the initial speed of the speed variable node corresponding to the first moment, thereby obtaining the initial states of the speed variable nodes corresponding to the second moment and the first moment respectively.

[0213] Step 25250: Determine the second acceleration of the object at the first moment based on the first acceleration of the object at the second moment or a preset acceleration.

[0214] In some optional embodiments, the first acceleration of the object at a second moment closest to the first moment may be used as the second acceleration of the object at the first moment.

[0215] In some optional embodiments, the preset acceleration may be 0 or another value, and the preset acceleration is used as the second acceleration of the object at the first moment.

[0216] Step 25260: Based on the first acceleration of the object at the second moment and the second acceleration of the object at the first moment, determine the initial accelerations of the acceleration variable nodes corresponding to the second moment and the first moment respectively.

[0217] Among them, the first acceleration of the object at any second moment is determined as the initial acceleration (or initial state) of the acceleration variable node corresponding to the second moment, and the second acceleration of the object at the first moment is determined as the initial acceleration of the acceleration variable node corresponding to the first moment, thereby obtaining the initial states of the acceleration variable nodes corresponding to the second moment and the first moment respectively.

[0218] Step 25270, based on the initial position of the position variable node, the initial velocity of the velocity variable node, and the initial acceleration of the acceleration variable node, obtain the initial state of the variable node.

[0219] Among them, the initial position of the position variable node, the initial velocity of the velocity variable node, and the initial acceleration of the acceleration variable node form the initial state of the variable node.

[0220] In the embodiments of the present disclosure, by using the optimized state at the second moment as the initial state of the variable node at the second moment, and using the predicted state of the object at the first moment as the initial state of the variable node at the first moment, it can be ensured that the initial state of the variable node has a certain degree of accuracy, which is conducive to reducing the number of iterations. Even by updating the state in one iteration, the target state that meets the iteration end condition can be obtained, thereby improving the optimization efficiency.

[0221] In some optional embodiments, based on any of the above embodiments, Figure 8As shown, step 260 iteratively updates the state of the variable node based on the first object information, the second object information, the initial state of the variable node, and the objective function to obtain the target state corresponding to the variable node, which may include:

[0222] Step 2610: Determine a first fusion observation state of the object based on the first object information and the second object information.

[0223] The first fusion observation state is the fusion observation state of the object at the first moment, and the first fusion observation state may include a first fusion observation position and / or a first fusion observation speed.

[0224] In some optional embodiments, the first observation state in the first object information and the second observation state in the second object information may be fused to obtain a first fused observation state of the object. Fusion methods include, but are not limited to, averaging, weighted summation according to different weights, and the like, and are not specifically limited.

[0225] Step 2620, based on the first fused observation state, the initial state of the variable node and the objective function, iteratively update the state of the variable node to obtain the target state corresponding to the variable node.

[0226] Among them, the first fused observation state and the initial state of the variable node are used in the relevant residual terms in the objective function to determine the value of the objective function, the Jacobian moment corresponding to the objective function, etc. during the solution process. The Jacobian matrix includes the partial derivatives of the residual term of the objective function with respect to the state variable. Then, by constructing the Hessian matrix (Hessian) or its approximation (such as the Gauss-Newton matrix), the state update amount of the variable node is solved. The specific algorithm for solving the state update amount is not limited. Based on the state update amount, the state of the variable node is updated to obtain the updated state, and it is judged whether the iteration end condition is met, such as reaching the maximum number of iterations or the convergence of the objective function value. If the iteration end condition is not met, based on the first fused observation state, the updated state of the variable node and the objective function, the state is further updated according to the above process, and so on, until the iteration end condition is met and the target state corresponding to the variable node is obtained.

[0227] In some optional embodiments, the state of the variable node can be iteratively updated based on the first fused observation state, the second fused observation state of the object at the second moment, the initial state of the variable node, and the objective function to obtain the target state corresponding to the variable node. The second fused observation state of the object at the second moment is used for the first sub-constraint relationship in the objective function. The second fused observation state of the object at the second moment can be obtained and stored during the optimization process at the second moment in a manner consistent with the first fused observation state, and will not be further described here.

[0228] In an embodiment of the present disclosure, a first fused observation state of an object is determined by first object information and second object information, and then the state of the variable node is updated in combination with the initial state and objective function of the variable node, so that the optimal target state of the variable node is solved based on the graph optimization algorithm, effectively improving the accuracy and effectiveness of the target state, and the first object information and the second object information are respectively obtained through observations of sensors with different attributes, so that the observation states of objects in different distance segments can have higher observation accuracy, further improving the accuracy and effectiveness of the target state.

[0229] Figure 10 It is a flowchart of determining a first fusion observation state of an object provided by an exemplary embodiment of the present disclosure.

[0230] In some optional embodiments, such as Figure 10 As shown, step 2610 of determining the first fusion observation state of the object based on the first object information and the second object information may include the following steps 26110 to 26140 and / or steps 26101 to 26104, and step 26150.

[0231] Step 26110: Determine a first observation position of the object based on the first object information.

[0232] The first object information includes a first observation position of the object at a first moment, so the first observation position of the object can be acquired from the first object information.

[0233] Step 26120: Determine a second observation position of the object based on the second object information.

[0234] The second object information includes the second observation position of the object at the first moment, so the second observation position of the object can be acquired from the second object information.

[0235] Step 26130: Determine a first position weight corresponding to the first observation position and a second position weight corresponding to the second observation position.

[0236] The first position weight (or first position observation confidence) and the second position weight (or second position observation confidence) respectively represent the observation confidence of the first observation position and the second observation position.

[0237] In some optional embodiments, the first position weight and the second position weight may be determined based on a first position residual between the first observed position and the optimized position at the previous moment, and a second position residual between the second observed position and the optimized position at the previous moment. Similar to f1 and f2 in the aforementioned embodiment, detailed description is omitted here.

[0238] Step 26140: Based on the first position weight and the second position weight, the first observation position and the second observation position are fused to obtain a first fused observation position.

[0239] The first fused observation position P1 is obtained by summing the product of the first position weight and the first observation position and the product of the second position weight and the second observation position, as shown below:

[0240] P1=β1*P 11 +β2*P 12 Formula (12)

[0241] In formula (12), β1 and β2 represent the first position weight and the second position weight respectively. 11 and P 12 Represent the first observation position and the second observation position respectively.

[0242] Step 26101: Determine a first observed velocity of the object based on first object information.

[0243] The first observed velocity of the object may be acquired from the first object information.

[0244] Step 26102: Determine a second observed velocity of the object based on the second object information.

[0245] The second observed velocity of the object may be acquired from the second object information.

[0246] Step 26103: Determine a first speed weight corresponding to the first observed speed and a second speed weight corresponding to the second observed speed.

[0247] Among them, the first speed weight and the second speed weight can be determined according to the residual between the first observed speed and the optimized speed at the previous moment, and the residual between the second observed speed and the optimized speed at the previous moment. The specific principle is similar to the first position weight and the second position weight, and will not be repeated here.

[0248] Step 26104: Based on the first velocity weight and the second velocity weight, the first observation velocity and the second observation velocity are fused to obtain a first fused observation velocity.

[0249] The first observation speed and the second observation speed are weighted and summed based on the first speed weight and the second speed weight to obtain the first fused observation speed.

[0250] Step 26150: Determine a first fused observation state based on the first fused observation position and / or the first fused observation speed.

[0251] In some optional embodiments, the first fused observation position may be used as the first fused observation state.

[0252] In some optional embodiments, the first fused observation speed may be used as the first fused observation state.

[0253] In some optional embodiments, the first fused observation position and the first fused observation speed may be used as the first fused observation state.

[0254] In some optional embodiments, the first fused observation state is used for the residual term corresponding to the first sub-constraint relationship in the objective function.

[0255] In some optional embodiments, the constraint relationship may further include an eighth sub-constraint relationship between the position variable node at any time and the fused observation position at the corresponding time, that is, the deviation between the state of the position variable node and the corresponding fused observation position should be small, and the corresponding residual term can be expressed as Where j represents T j Moment, ω j Indicates T j The confidence weight corresponding to the moment, p j (var) represents T j The state of the position variable node at the moment, p j (fus) means T j The fused observation position at time . Then the first fused observation position can be used for the residual term.

[0256] In the embodiments of the present disclosure, by weightedly fusing the first observation position of the object observed by the first sensor with the second observation position of the object observed by the second sensor, and weighted fusing the first observation speed with the second observation speed, the first fused observation position and the first fused observation speed of the object at the first moment are obtained. The observation performance of different sensors on the object can be integrated to effectively improve the observation accuracy of the fused observation state of objects in different distance segments.

[0257] In some optional embodiments, determining the first position weight corresponding to the first observation position and the second position weight corresponding to the second observation position in step 26130 may include:

[0258] Based on the first observation position and the third position of the object at the third moment, determine the first position residual; the third moment is before the first moment; based on the second observation position and the third position of the object at the third moment, determine the second position residual; based on the first position residual and the second position residual, determine the first position weight and the second position weight.

[0259] The third position of the object at the third moment is the position optimized by the optimization process before the first moment. For example, if the first moment is T4 and the third moment is T3, the third position is the position after the previous optimization corresponding to T3. The first position residual is the absolute value of the difference between the first observed position and the third position, and the second position residual is the absolute value of the difference between the second observed position and the third position. Based on the first position residual and the second position residual, the first position weight and the second position weight are determined according to the rule that the larger the residual, the smaller the weight.

[0260] In some optional embodiments, for any second moment, since the previously optimized position already exists, the large-eye observation position residual can be determined based on the large-eye observation position and the optimized position of the object at the second moment. The large-eye observation position refers to the observation position of the object observed by the first camera; the small-eye observation position residual can be determined based on the small-eye observation position and the optimized position of the object at the second moment. The small-eye observation position refers to the observation position of the object observed by the second camera. Based on the large-eye observation position residual and the small-eye observation position residual, the large-eye position weight and the small-eye position weight corresponding to the large-eye observation position are determined. Then, based on the large-eye position weight and the small-eye position weight, the large-eye observation position and the small-eye observation position are weightedly fused to obtain the second fused observation position at the second moment. Figure 6 As shown, at the second moment within a certain time window, the state of its variable node may have undergone optimization processes at multiple moments. For example, the variable node at time T2 undergoes an optimization process during the optimization process at time T2, with time T2 being the first moment. During the optimization process at time T3, the variable node at time T2 undergoes another optimization process, with time T3 being the first moment and time T2 being the second moment. During the optimization process at time T4, the variable node at time T2 and the variable node at time T3 undergo another optimization process, with time T4 being the first moment and both times T2 and T3 being the second moments. This continues until time T2 slides out of the time window. After any optimization, the position change after optimization may cause changes in the large eye position weight and the small eye position weight. Therefore, the large eye position weight and the small eye position weight, which are used as the second moment, can be updated after the optimization. When fusing the large and small eye observation positions, the fusion is performed based on the latest position weights to ensure the validity of the fused observation positions. For the eighth sub-constraint relationship mentioned above, the fused observation position required at the second moment may be the latest fused observation position.

[0261] In the embodiments of the present disclosure, the position weights of different sensors are determined by the size of the residuals between the observation positions observed by different sensors and the optimized positions. Since the size of the residuals can characterize the credibility of the sensor observation results, the position weights of sensors with high credibility are larger, and the position weights of sensors with low credibility are smaller, thereby increasing the contribution of observation positions with high credibility to the fused observation positions and reducing the contribution of observation positions with low credibility to the fused observation positions, thereby ensuring the reliability and effectiveness of the fused observation positions.

[0262] In some optional embodiments, after completing the optimization process at the first moment, the optimized states of the variable nodes at the first moment are obtained. Based on the large eye observation position and small eye observation position at the first moment and the optimized positions at the first moment, the large eye position weight and small eye position weight at the first moment can be updated for use in subsequent optimization processes.

[0263] Figure 11 FIG. 4 is a flowchart of a method for determining a state of an object provided by another exemplary embodiment of the present disclosure.

[0264] In some optional embodiments, based on any of the above embodiments, Figure 11 As shown, before iteratively updating the state of the variable node based on the first object information, the second object information, the initial state of the variable node and the objective function in step 260 to obtain the target state corresponding to the variable node, the following steps are also included:

[0265] Step 310 : Determine observation lag indication information based on the first velocity of the object at the second moment and the second fused observation velocity of the object at the second moment.

[0266] The observation lag indication information indicates whether the observed velocity is lagged or not. The first velocity at the second moment is the previously optimized velocity corresponding to the velocity variable node at the second moment. If there are multiple second moments, the multiple first velocities form an optimized velocity sequence. The second fused observed velocity of the object at the second moment is derived based on the observed velocity of the object observed by the first sensor and the observed velocity of the object observed by the second sensor.

[0267] In some optional embodiments, for any second moment, the observed speed observed by the first camera and the observed speed observed by the second camera can be weightedly fused based on the first fusion confidence weight corresponding to the first camera and the second fusion confidence weight corresponding to the second camera to obtain a second fused observed speed. The first fusion confidence weight can be determined based on the speed observation confidence corresponding to at least one element observed by the first camera. The second fusion confidence weight can be determined based on the speed observation confidence corresponding to at least one element observed by the second camera. The speed observation confidence corresponding to at least one element observed by any camera can be determined based on the residual between the observed speed corresponding to the element and the optimized speed at the second moment, with the larger the residual, the smaller the speed observation confidence. For example, the first fusion confidence weight and the second fusion confidence weight can be determined based on the sum of the speed observation confidences corresponding to multiple elements observed by the first camera and the sum of the speed observation confidences corresponding to multiple elements observed by the second camera. Optionally, the sum of the speed observation confidences corresponding to at least one element observed by the first camera can be used as the first fusion confidence weight, and the sum of the speed observation confidences corresponding to at least one element observed by the second camera can be used as the second fusion confidence weight. Alternatively, the sum of the speed observation confidences corresponding to at least one element observed by the first camera and the sum of the speed observation confidences corresponding to at least one element observed by the second camera can be normalized to obtain the first fusion confidence weight and the second fusion confidence weight.

[0268] In some optional embodiments, the observation lag indication information may be determined based on a relationship between a cumulative value of a speed deviation of the second fused observation speed relative to the first speed at the second moment and a threshold.

[0269] Step 320 : In response to the observation lag indication information indicating that the observation speed is lagging, adjust the confidence level of the residual term corresponding to the seventh sub-constraint relationship in the objective function.

[0270] If the observation lag indication information indicates that the observation speed is lagging, the confidence in the residual term corresponding to the seventh sub-constraint relationship in the objective function is reduced, that is, the confidence weight ω of the residual term in e7 j , to speed up the response. If the observation lag indication information indicates that the observation speed is not lagging, the confidence weight in the residual term corresponding to the seventh sub-constraint relationship in the objective function can be increased to improve stability.

[0271] In the embodiment of the present disclosure, by adjusting the confidence weight of the residual term corresponding to the seventh sub-constraint relationship in the objective function to optimize the speed response sensitivity, the adverse effects of observation lag can be reduced, and the adaptability of the method of the embodiment of the present disclosure to different driving scenarios can be improved, ensuring a rapid response during sudden acceleration and deceleration, and maintaining the stability of the autonomous mobile device when the speed is stable, thereby effectively improving the performance of the autonomous mobile device.

[0272] Figure 12 It is a flowchart of determining observation lag indication information provided by an exemplary embodiment of the present disclosure.

[0273] In some optional embodiments, such as Figure 12 As shown, determining the observation lag indication information based on the first velocity of the object at the second moment and the second fused observation velocity of the object at the second moment in step 310 may include:

[0274] Step 3110: Determine the first position cumulative change based on the first speed at the second moment.

[0275] The first cumulative position change is a cumulative result of sequentially accumulating position changes from a first second moment to a last second moment, at the first speed at each of the second moments. The first cumulative position change can be determined based on the first speed at each second moment, the time interval between adjacent second moments, and the relationship between position change, speed, and time.

[0276] Step 3120: Determine a second position cumulative change based on the second fused observation speed at the second moment.

[0277] The second cumulative position change is the cumulative result of sequentially accumulating position changes according to the second fused observed speed at at least one second moment. The second cumulative position change can be determined based on the second fused observed speed at at least one second moment, the time interval between the second moments, and the relationship between position change, speed, and time.

[0278] Step 3130: Determine the position cumulative change residual based on the first position cumulative change and the second position cumulative change.

[0279] The position cumulative change residual refers to the difference between the first position cumulative change and the second position cumulative change or the absolute value of the difference.

[0280] Step 3140: Determine the velocity residual based on the position cumulative change residual and the duration of the time window where the second moment is located.

[0281] The velocity residual can be obtained by dividing the cumulative position change residual by the length of the time window (or residual window).

[0282] In some optional embodiments, the time window may be a preset value.

[0283] In some optional embodiments, the time window can be adjusted according to different scenarios.

[0284] Step 3150: Determine observation lag indication information based on the velocity residual.

[0285] Among them, the observation lag indication information can be determined based on the speed residual and the residual threshold. The residual threshold can be determined based on the distance of the object and the motion scene. The motion scene refers to whether the object brakes suddenly, and whether the object brakes suddenly can be judged based on the acceleration of the object. The rule for determining the residual threshold is that the longer the distance, the larger the residual threshold, and the closer the distance, the smaller the residual threshold. The residual threshold is smaller in the sudden braking scene, and larger in the non-sudden braking scene. If the speed residual is greater than the residual threshold, the observation lag indication information is determined to indicate that the observed speed is lagging. If the speed residual is less than the residual threshold, the observation lag indication information is determined to indicate that the observed speed is not lagging.

[0286] In some optional examples, the velocity residual vel res It can be expressed as follows:

[0287]

[0288] In formula (13), n represents the length of the time window, that is, the number of moments included. i represents the optimized speed corresponding to the i-th moment in the time window, that is, the first speed, v i Indicates the second fused observation velocity corresponding to the i-th moment in the time window. i It represents the time interval between the i-th moment and the i+1-th moment, and dt represents the total duration between the first moment and the last moment in the time window.

[0289] In the embodiment of the present disclosure, the speed residual is determined by the optimized speed and the fused observation speed at the moment within the residual window, which is used to judge the observation hysteresis, thereby providing an effective reference basis for adjusting the confidence weight of the relevant residual term.

[0290] In some optional embodiments, adjusting the confidence of the residual term corresponding to the seventh sub-constraint relationship in the objective function in step 320 may include:

[0291] Based on the velocity residual, the confidence of the residual term corresponding to the seventh sub-constraint relationship in the objective function is adjusted.

[0292] The adjustment rule based on the speed residual is: the larger the speed residual, the smaller the confidence of the residual term corresponding to the seventh sub-constraint relationship; the smaller the speed residual, the greater the confidence of the residual term corresponding to the seventh sub-constraint relationship.

[0293] It should be noted that the operation of adjusting the residual item confidence from step 310 to step 320 can be performed before any optimization, or after the previous optimization is completed, to ensure that the confidence of the residual item used in the optimization is the latest confidence.

[0294] In the embodiments of the present disclosure, since the size of the speed residual represents the degree of speed observation lag, the confidence level of the residual term corresponding to the seventh sub-constraint relationship is adjusted based on the speed residual to optimize the speed response sensitivity, ensure a rapid response when the object accelerates or decelerates suddenly, avoid the adverse effects of observation lag, and maintain the stability of speed observation when the object speed is stable, thereby effectively improving the driving performance of the autonomous mobile device.

[0295] In some optional embodiments, before determining the velocity residual based on the cumulative position change residual and the duration of the time window at the second moment in step 3140, the method further includes: determining a motion scene of the object and adjusting the duration of the time window according to the motion scene.

[0296] The object's motion scene includes scenes with drastic speed changes and scenes with relatively smooth speed changes. The object's motion scene can be determined based on the object's acceleration.

[0297] In some optional embodiments, for scenes where the object speed changes dramatically, a smaller time window is used to determine the speed residual to ensure the timeliness of the speed. For scenes where the object speed changes more smoothly, a larger time window is used to ensure the stability of the speed.

[0298] The embodiments of the present disclosure provide a method for determining the state of an object. A short-focal-length, wide-angle camera (large-eye camera) is used to focus on identifying nearby objects, and a long-focal-length, narrow-angle camera (small-eye camera) is used to identify distant objects. By combining cameras of different focal lengths, the limitations of monocular ranging in detection distance and coverage can be overcome, and the complex pixel matching problem in binocular ranging technology can be avoided. This effectively reduces the amount of computation, reduces system complexity, improves ranging accuracy and robustness, and solves the problem of the contradiction between detection distance and coverage in complex traffic environments, thereby improving the system reliability and practicality of autonomous mobile devices. Secondly, before optimization, the confidence level of the historically optimized speed is adjusted through a speed observation lag adjustment mechanism to optimize the response sensitivity of the relevant systems of the autonomous mobile device to the speed of the object, improve the system's adaptability in different driving scenarios, ensure rapid response when the object accelerates or decelerates suddenly, and maintain system stability when the object's speed is stable, effectively improving system performance. Thirdly, by combining observation state sequences determined by multiple regional frames and multiple factors with weighted fusion of observation confidences to generate a fused observation state, the robustness of the observation state is enhanced, the accuracy and stability of the system's observed velocity are strengthened, the error and instability introduced by a single observation source are reduced, and the overall system performance is effectively improved. Furthermore, by adjusting the length of the residual window according to the different motion scenarios of the object, using a small window to improve the system's response speed in scenarios with rapid acceleration and deceleration, and a large window in scenarios with stable velocity, the system's stability is improved. This enables the autonomous mobile device system to flexibly adapt to different driving scenarios, ensuring optimal performance under various operating conditions. Furthermore, by fitting the velocity based on the observed positions at multiple moments when determining the first observed velocity, and after the initial fitting, removing outliers with large deviations to improve the validity of the observed positions, the velocity is then refitted to improve the accuracy and reliability of the observed velocity, reducing the adverse effects of abnormal data on the system, and providing effective technical support for achieving high-precision velocity observation. In addition, graph optimization algorithms are used to process the distance (or position) and velocity estimation of objects, comprehensively consider the relationship between the observed states and state variables at multiple moments, reduce noise and cumulative errors, and improve the adaptability of the autonomous mobile device system to highly nonlinear scenarios and observation models. The accuracy of the object's state is improved through iterative optimization algorithms to ensure the robustness and stability of the system in complex environments.

[0299] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information in the technical solutions disclosed herein comply with relevant laws and regulations and do not violate public order and good morals. Furthermore, the collection and use of user personal information in the technical solutions disclosed herein are conducted with the user's knowledge and authorization, and do not involve any unauthorized collection or use of user personal information.

[0300] The above-mentioned embodiments of the present disclosure may be implemented individually or in any combination without conflict. The specific configuration may be based on actual needs and is not limited by the present disclosure.

[0301] Any method for determining the state of an object provided in the embodiments of the present disclosure can be performed by any appropriate electronic device with data processing capabilities, including but not limited to: electronic devices such as terminal devices and servers. Alternatively, any method for determining the state of an object provided in the embodiments of the present disclosure can be performed by a processor, such as a processor executing any method for determining the state of an object mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be repeated below.

[0302] Exemplary devices

[0303] Figure 13 Schematic diagram of a device for determining the state of an object provided by an exemplary embodiment of the present disclosure. The device of the embodiment of the present disclosure can be used to implement the corresponding method embodiment of the present disclosure, such as Figure 13 The device shown may include: a first processing module 61 , a second processing module 62 , a third processing module 63 , a fourth processing module 64 , a fifth processing module 65 , a sixth processing module 66 and a seventh processing module 67 .

[0304] A first processing module 61 is configured to determine first object information obtained by the autonomous mobile device observing the object through a first sensor at a first moment;

[0305] A second processing module 62 is configured to determine second object information obtained by the autonomous mobile device observing the object through a second sensor at a first moment;

[0306] The third processing module 63 is used to determine a topological graph of the position, velocity, and acceleration of the object; the topological graph includes position variable nodes, velocity variable nodes, acceleration variable nodes, and edges representing constraint relationships between the variable nodes;

[0307] A fourth processing module 64 is configured to determine an objective function based on the topological graph;

[0308] A fifth processing module 65 is configured to determine an initial state of a variable node in the topology graph based on third object information of the object at a second moment; the second moment is before the first moment;

[0309] A sixth processing module 66 is configured to iteratively update the state of the variable node based on the first object information, the second object information, the initial state of the variable node, and the objective function to obtain a target state corresponding to the variable node;

[0310] The seventh processing module 67 is configured to determine target object information of the object at the first moment based on the target state corresponding to the variable node.

[0311] Figure 14 3 is a schematic structural diagram of an apparatus for determining a state of an object provided by another exemplary embodiment of the present disclosure.

[0312] In some optional embodiments, in the above Figure 13 Based on the embodiment shown, Figure 14 As shown, the first processing module 61 may include: a first determining unit 611 , a second determining unit 612 and a third determining unit 613 .

[0313] The first determining unit 611 is configured to determine a first observation position obtained by observing the object through a first sensor at a first moment.

[0314] The second determining unit 612 is configured to determine a first observed velocity of the object at the first moment based on a third observed position of the object obtained by the first sensor at the second moment.

[0315] The third determining unit 613 is configured to determine first object information based on the first observation position and the first observation speed.

[0316] In some optional embodiments, the second processing module 62 may include: a fourth determining unit 621 , a fifth determining unit 622 , and a sixth determining unit 623 .

[0317] The fourth determining unit 621 is configured to determine a second observation position obtained by observing the object through the second sensor at the first moment.

[0318] The fifth determining unit 622 is configured to determine a second observed speed of the object at the first moment based on a fourth observed position of the object obtained by the second sensor at the second moment.

[0319] The sixth determining unit 623 is configured to determine second object information based on the second observation position and the second observation speed.

[0320] In some optional embodiments, the first sensor is a first camera with a first field of view; the number of the second moments is one or more; the third observation position at any second moment includes observation positions determined based on at least one element of the object in the first image, at least one element includes a first height and / or first width of the area box of the object in the first image, and the first image is an image captured by the first camera at the second moment.

[0321] The second determining unit 612 is specifically configured to:

[0322] Based on the third observation position at the second moment, determine the third observation position sequence corresponding to at least one element; based on the third observation position sequence corresponding to the element, determine the observation speed corresponding to the element; based on the observation speed corresponding to the element, determine the first observation speed of the object at the first moment.

[0323] In some optional embodiments, the second determining unit 612 is specifically configured to:

[0324] Based on the third observation position sequence corresponding to the element, a first straight line between the observation position and time corresponding to the element is fitted; based on the third observation position sequence and the first straight line, the third observation position sequence is filtered to obtain a filtered fifth observation position sequence; based on the fifth observation position sequence, a second straight line between the observation position and time corresponding to the element is fitted; based on the second straight line, the observation speed corresponding to the element is determined.

[0325] In some optional embodiments, the second determining unit 612 is specifically configured to:

[0326] Based on the observed speed corresponding to the element and the third speed of the object at the third moment, determine the first speed residual of the observed speed corresponding to the element and the third speed; the third moment is before the first moment; based on the first speed residual, determine the first speed observation confidence corresponding to the element; based on the observed speed corresponding to the element and the first speed observation confidence, determine the first observed speed of the object at the first moment.

[0327] In some optional embodiments, the second sensor is a second camera with a second field of view (FOV); the second FOV is smaller than the first FOV corresponding to the first sensor. The fourth observation position at any second moment includes observation positions determined based on at least one element of the object in the second image, where the at least one element includes a second height and / or a second width of a region frame of the object in the second image, and the second image is an image captured by the second camera at the second moment.

[0328] The fifth determining unit 622 is specifically configured to:

[0329] Based on the fourth observation position at the second moment, determine the fourth observation position sequence corresponding to at least one element; based on the fourth observation position sequence corresponding to the element, determine the observation speed corresponding to the element; based on the observation speed corresponding to the element, determine the second observation speed of the object at the first moment.

[0330] In some optional embodiments, the fifth determining unit 622 is specifically configured to:

[0331] Based on the fourth observation position sequence corresponding to the element, a third straight line of the observation position and time corresponding to the element is fitted; based on the fourth observation position sequence and the third straight line, the fourth observation position sequence is filtered to obtain a filtered sixth observation position sequence; based on the sixth observation position sequence, a fourth straight line of the observation position and time corresponding to the element is fitted; based on the fourth straight line, the observation speed corresponding to the element is determined.

[0332] In some optional embodiments, the fifth determining unit 622 is specifically configured to:

[0333] Based on the observed speed corresponding to the element and the third speed of the object at the third moment, determine the second speed residual of the observed speed corresponding to the element and the third speed; the third moment is before the first moment; based on the second speed residual, determine the second speed observation confidence corresponding to the element; based on the observed speed corresponding to the element and the second speed observation confidence, determine the second observed speed of the object at the first moment.

[0334] In some optional embodiments, based on any of the above embodiments, the first sensor is a first camera with a first field of view; the second sensor is a second camera with a second field of view; and the first field of view is greater than the second field of view. The first processing module 61 or the first determination unit 611 is configured to determine a first region frame of the object in a first image captured by the first camera. The first object information is determined based on the first region frame and first parameters of the first camera.

[0335] In some optional embodiments, the second processing module 62 or the fourth determining unit 621 is configured to: determine a second area frame of the object in the second image captured by the second camera, and determine second object information based on the second area frame and second parameters of the second camera.

[0336] In some optional embodiments, the first processing module 61 or the first determining unit 611 is specifically configured to:

[0337] Determine a first distance of the object based on a first height and a first parameter of the first area box; determine a second distance of the object based on a first width and the first parameter of the first area box; determine a first horizontal coordinate of the object based on a pixel distance between the first area box and a center point of the first image, and the first parameter; and determine first object information based on the first distance, the second distance, and the first horizontal coordinate.

[0338] In some optional embodiments, the second processing module 62 or the fourth determining unit 621 is specifically configured to:

[0339] Determine a third distance of the object based on the second height and second parameter of the second area box; determine a fourth distance of the object based on the second width and second parameter of the second area box; determine a second horizontal coordinate of the object based on the pixel distance between the second area box and the center point of the second image, and the second parameter; and determine second object information based on the third distance, the fourth distance, and the second horizontal coordinate.

[0340] In some optional embodiments, based on any of the above embodiments, the constraint relationship between the variable nodes includes at least one of the following:

[0341] The first sub-constraint relationship between the velocity variable node and the corresponding fused observation velocity; the second sub-constraint relationship between the velocity variable node at any moment and the velocity variable node and acceleration variable node at the previous moment that satisfies the preset motion model; the third sub-constraint relationship between the position variable node at any moment and the position variable node at the previous moment, the first optimized velocity of the velocity variable node at the previous moment, and the first optimized acceleration of the acceleration variable node at the previous moment; the first optimized velocity of the velocity variable node and the first optimized acceleration of the acceleration variable node at any moment are the velocity and acceleration of the velocity variable node and the acceleration variable node after the previous graph optimization; the fourth sub-constraint relationship that the accelerations of adjacent acceleration variable nodes are equal; the fifth sub-constraint relationship that the velocities of adjacent velocity variable nodes are equal, which is used to suppress the amplitude of acceleration; the sixth sub-constraint relationship that the velocity variable node is equal to the first optimized velocity of the velocity variable node; the seventh sub-constraint relationship that the acceleration variable node is equal to the first optimized acceleration of the acceleration variable node.

[0342] In some optional embodiments, based on any of the above embodiments, Figure 14 As shown, the fourth processing module 64 may include: a first processing unit 641 and a second processing unit 642 .

[0343] The first processing unit 641 is configured to determine a residual term corresponding to a sub-constraint relationship based on a sub-constraint relationship in a constraint relationship between variable nodes.

[0344] The second processing unit 642 is configured to determine an objective function based on the residual term.

[0345] In some optional embodiments, based on any of the above embodiments, Figure 14 As shown, the fifth processing module 65 may include: a third processing unit 651 and a fourth processing unit 652 .

[0346] The third processing unit 651 is configured to predict a first predicted state of the object at a first moment based on the third object information.

[0347] The fourth processing unit 652 is configured to determine an initial state of the variable node based on the third object information and the first predicted state.

[0348] In some optional embodiments, the fourth processing unit 652 is specifically configured to:

[0349] Based on the third object information, the first position, first velocity and first acceleration of the object at the second moment are determined. Based on the first predicted state, the first predicted position and first predicted velocity of the object at the first moment are determined. Based on the first position of the object at the second moment and the first predicted position of the object at the first moment, the initial positions of the position variable nodes corresponding to the second moment and the first moment are determined. Based on the first velocity of the object at the second moment and the first predicted velocity of the object at the first moment, the initial velocities of the velocity variable nodes corresponding to the second moment and the first moment are determined. Based on the first acceleration of the object at the second moment or the preset acceleration, the second acceleration of the object at the first moment is determined. Based on the first acceleration of the object at the second moment and the second acceleration of the object at the first moment, the initial accelerations of the acceleration variable nodes corresponding to the second moment and the first moment are determined. Based on the initial position of the position variable node, the initial velocity of the velocity variable node and the initial acceleration of the acceleration variable node, the initial state of the variable node is obtained.

[0350] In some optional embodiments, based on any of the above embodiments, Figure 14 As shown, the sixth processing module 66 may include: a fifth processing unit 661 and a sixth processing unit 662 .

[0351] The fifth processing unit 661 is configured to determine a first fusion observation state of the object based on the first object information and the second object information.

[0352] The sixth processing unit 662 is used to iteratively update the state of the variable node based on the first fused observation state, the initial state of the variable node and the objective function, and obtain the target state corresponding to the variable node.

[0353] In some optional embodiments, the fifth processing unit 661 is specifically configured to: determine a first observed position of the object based on the first object information; determine a second observed position of the object based on the second object information; determine a first position weight corresponding to the first observed position and a second position weight corresponding to the second observed position; and fuse the first observed position and the second observed position based on the first position weight and the second position weight to obtain a first fused observed position. And / or, determine a first observed velocity of the object based on the first object information; determine a second observed velocity of the object based on the second object information; determine a first velocity weight corresponding to the first observed velocity and a second velocity weight corresponding to the second observed velocity; and fuse the first observed velocity and the second observed velocity based on the first velocity weight and the second velocity weight to obtain a first fused observed velocity. Determine a first fused observation state based on the first fused observed position and / or the first fused observed velocity.

[0354] In some optional embodiments, the fifth processing unit 661 is specifically configured to:

[0355] Based on the first observation position and the third position of the object at the third moment, determine the first position residual; the third moment is before the first moment; based on the second observation position and the third position of the object at the third moment, determine the second position residual; based on the first position residual and the second position residual, determine the first position weight and the second position weight.

[0356] In some optional embodiments, based on any of the above embodiments, the sixth processing unit 662 may also be configured to:

[0357] Determining observation lag indication information based on the first velocity of the object at the second moment and the second fused observation velocity of the object at the second moment. The observation lag indication information is used to indicate whether the observation velocity is lagged or not lagged. In response to the observation lag indication information indicating that the observation velocity is lagged, adjusting the confidence level of a residual term corresponding to the seventh sub-constraint relationship in the objective function.

[0358] In some optional embodiments, the sixth processing unit 662 is specifically configured to:

[0359] Determine the first cumulative position change based on the first velocity at the second moment. Determine the second cumulative position change based on the second fused observed velocity at the second moment. Determine a cumulative position change residual based on the first cumulative position change and the second cumulative position change. Determine a velocity residual based on the cumulative position change residual and the duration of the time window within which the second moment falls. Determine observation lag indication information based on the velocity residual.

[0360] In some optional embodiments, the sixth processing unit 662 is specifically configured to adjust the confidence of the residual term corresponding to the seventh sub-constraint relationship in the objective function based on the speed residual.

[0361] In some optional embodiments, the sixth processing unit 662 may be further configured to: determine a motion scene of the object, and adjust the duration of the time window according to the motion scene.

[0362] The above-mentioned embodiments of the present disclosure may be implemented individually or in any combination without conflict. The specific configuration may be based on actual needs and is not limited by the present disclosure.

[0363] The beneficial technical effects corresponding to the exemplary embodiment of this device can be found in the corresponding beneficial technical effects of the above exemplary method part, which will not be repeated here.

[0364] Exemplary electronic devices

[0365] Figure 15 is a structural diagram of an electronic device provided by an embodiment of the present disclosure, including at least one processor 91 and a memory 92.

[0366] The processor 91 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.

[0367] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute the one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of the present disclosure described above.

[0368] In one example, the electronic device 90 may further include an input device 93 and an output device 94 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0369] The input device 93 may also include, for example, a touch screen, a microphone, various sensors, and the like. Sensors may include, for example, image sensors (e.g., cameras, etc.), laser radars, millimeter-wave radars, ultrasonic radars, positioning sensors, pressure sensors, air quality sensors, temperature sensors, and the like. Image sensors, laser radars, millimeter-wave radars, ultrasonic radars, and the like may be used to perceive the surrounding environment, i.e., to detect dynamic and static objects in the surrounding environment. Dynamic and static objects may include, for example, static objects such as lane lines, curbs, arrows, signs, trees, and buildings, as well as dynamic objects such as surrounding vehicles, pedestrians, and cyclists. Positioning sensors are used to locate the movable device (e.g., a vehicle, robot, etc.) in which the electronic device is located. Positioning sensors may include, for example, an inertial measurement unit (IMU), a global positioning system (GPS), and the like. Pressure sensors may be used to detect seat pressure. Temperature sensors may be used to detect the temperature in the vehicle cabin. Air quality sensors may be used to detect the air quality in the vehicle cabin.

[0370] The output device 94 can output various information to the outside, and may include, for example, a display, a speaker, a communication network and a remote output device connected thereto, and the like.

[0371] Of course, to simplify, Figure 15 Only some of the components related to the present disclosure in the electronic device 90 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 90 may further include any other appropriate components according to specific application scenarios.

[0372] Exemplary computer program products and computer-readable storage media

[0373] In addition to the above methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to perform the steps in the methods of various embodiments of the present disclosure described in the above "Exemplary Method" section.

[0374] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0375] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps in the method of various embodiments of the present disclosure described in the above “Exemplary Method” section.

[0376] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or component comprising electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0377] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be considered as essential to each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0378] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A method for determining a state of an object, comprising: Determine first object information obtained by the autonomous mobile device observing the object through a first sensor at a first moment; Determining second object information obtained by the autonomous mobile device observing the object through a second sensor at the first moment; determining a topological map of position, velocity, and acceleration of the object; The topological graph includes variable nodes and edges representing constraint relationships between the variable nodes; Based on the topological graph, determining an objective function; determining an initial state of the variable node in the topological graph based on third object information of the object at the second moment; The second moment is before the first moment; Iteratively updating the state of the variable node based on the first object information, the second object information, the initial state of the variable node, and the objective function to obtain a target state corresponding to the variable node; Based on the target state corresponding to the variable node, target object information of the object at the first moment is determined.

2. The method according to claim 1, wherein The determining, based on the third object information of the object at the second moment, the initial state of the variable node in the topological graph includes: predicting a first predicted state of the object at the first moment based on the third object information; An initial state of the variable node is determined based on the third object information and the first predicted state.

3. The method according to claim 2, wherein: The determining the initial state of the variable node based on the third object information and the first predicted state includes: determining a first position, a first velocity, and a first acceleration of the object at the second moment based on the third object information; determining a first predicted position and a first predicted velocity of the object at the first moment based on the first predicted state; Determining initial positions of position variable nodes corresponding to the second moment and the first moment, respectively, based on the first position of the object at the second moment and the first predicted position of the object at the first moment; determining, based on the first speed of the object at the second moment and the first predicted speed of the object at the first moment, initial speeds of speed variable nodes corresponding to the second moment and the first moment respectively; determining a second acceleration of the object at the first moment based on the first acceleration or a preset acceleration of the object at the second moment; Determining initial accelerations of acceleration variable nodes corresponding to the second moment and the first moment, respectively, based on the first acceleration of the object at the second moment and the second acceleration of the object at the first moment; The initial state of the variable node is obtained based on the initial position of the position variable node, the initial velocity of the velocity variable node, and the initial acceleration of the acceleration variable node.

4. The method according to claim 1, wherein The iteratively updating the state of the variable node based on the first object information, the second object information, the initial state of the variable node, and the objective function to obtain the target state corresponding to the variable node includes: determining a first fused observation state of the object based on the first object information and the second object information; Based on the first fused observation state, the initial state of the variable node and the objective function, the state of the variable node is iteratively updated to obtain the target state corresponding to the variable node.

5. The method according to claim 4, wherein The determining a first fusion observation state of the object based on the first object information and the second object information includes: determining a first observation position of the object based on the first object information; determining a second observation position of the object based on the second object information; Determining a first position weight corresponding to the first observation position and a second position weight corresponding to the second observation position; fusing the first observation position and the second observation position based on the first position weight and the second position weight to obtain a first fused observation position; and / or, determining a first observed velocity of the object based on the first object information; determining a second observed velocity of the object based on the second object information; determining a first speed weight corresponding to the first observed speed and a second speed weight corresponding to the second observed speed; fusing the first observed velocity and the second observed velocity based on the first velocity weight and the second velocity weight to obtain a first fused observed velocity; The first fused observation state is determined based on the first fused observation position and / or the first fused observation speed.

6. The method according to claim 5, wherein: The determining a first position weight corresponding to the first observation position and a second position weight corresponding to the second observation position includes: determining a first position residual based on the first observation position and a third position of the object at a third moment, wherein the third moment is before the first moment; determining a second position residual based on the second observation position and the third position of the object at the third moment; The first position weight and the second position weight are determined based on the first position residual and the second position residual.

7. The method according to claim 1, wherein The constraint relationship between the variable nodes includes at least one of the following: The first sub-constraint relationship between the velocity variable node and the corresponding fused observation velocity; The velocity variable node at any moment satisfies the second sub-constraint relationship of the preset motion model with the velocity variable node and acceleration variable node at the previous moment; A third sub-constraint relationship between a position variable node at any moment and the position variable node at the previous moment, the first optimized velocity of the velocity variable node at the previous moment, and the first optimized acceleration of the acceleration variable node at the previous moment; the first optimized velocity of the velocity variable node and the first optimized acceleration of the acceleration variable node at any moment are the velocity and acceleration of the velocity variable node and the acceleration variable node after the previous graph optimization; The fourth sub-constraint relationship is that the accelerations of adjacent acceleration variable nodes are equal; The fifth sub-constraint relationship of equal velocities of adjacent velocity variable nodes is used to suppress the amplitude of acceleration; a sixth sub-constraint relationship in which a speed variable node is equal to a first optimized speed of the speed variable node; a seventh sub-constraint relationship in which the acceleration variable node is equal to the first optimized acceleration of the acceleration variable node; The determining of the objective function based on the topological graph includes: Determining, based on a sub-constraint relationship in the constraint relationship between the variable nodes, a residual term corresponding to the sub-constraint relationship; The objective function is determined based on the residual term.

8. The method according to claim 7, wherein: Before iteratively updating the state of the variable node based on the first object information, the second object information, the initial state of the variable node, and the objective function to obtain the target state corresponding to the variable node, the method further includes: determining observation lag indication information based on the first velocity of the object at the second moment and the second fused observation velocity of the object at the second moment; the observation lag indication information is used to indicate whether the observation velocity is lagged or not; In response to the observation lag indication information indicating an observation speed lag, the confidence of the residual term corresponding to the seventh sub-constraint relationship in the objective function is adjusted.

9. The method according to claim 8, wherein The determining the observation lag indication information based on the first velocity of the object at the second moment and the second fused observation velocity of the object at the second moment includes: determining a first position cumulative change based on the first speed at the second moment; determining a second position cumulative change based on the second fused observation speed at the second moment; Determining a position cumulative change residual based on the first position cumulative change amount and the second position cumulative change amount; Determining a velocity residual based on the cumulative position change residual and the length of the time window in which the second moment is located; The observation lag indication information is determined based on the velocity residual.

10. The method according to claim 9, wherein: The adjusting the confidence of the residual term corresponding to the seventh sub-constraint relationship in the objective function includes: Based on the speed residual, the confidence of the residual term corresponding to the seventh sub-constraint relationship in the objective function is adjusted.

11. The method according to claim 9, wherein Also includes: determining a motion scene of the object; The duration of the time window is adjusted according to the motion scene.

12. The method according to any one of claims 1 to 11, wherein: The determining of first object information obtained by the autonomous mobile device observing the object through the first sensor at the first moment includes: determining a first observation position obtained by observing the object by the first sensor at the first moment; determining a first observed velocity of the object at the first moment based on a third observed position of the object obtained by the first sensor at the second moment; determining the first object information based on the first observation position and the first observation speed; and / or, The determining of second object information obtained by the autonomous mobile device observing the object through a second sensor at the first moment includes: determining a second observation position obtained by observing the object through the second sensor at the first moment; determining a second observed velocity of the object at the first moment based on a fourth observed position of the object obtained by the second sensor at the second moment; The second object information is determined based on the second observation position and the second observation speed.

13. The method according to claim 12, wherein: The first sensor is a first camera with a first field of view; the number at the second moment is one or more; The third observation position at the second moment includes observation positions respectively determined based on at least one element of the object in a first image, the at least one element including a first height and / or a first width of an area box of the object in the first image, and the first image is an image captured by the first camera at the second moment; Determining a first observed velocity of the object at the first moment based on a third observed position of the object obtained by the first sensor at the second moment includes: determining, based on the third observation position at the second moment, a third observation position sequence corresponding to each of the at least one element; determining an observation speed corresponding to the element based on the third observation position sequence corresponding to the element; The first observed speed of the object at the first moment is determined based on the observed speed corresponding to the element.

14. The method according to claim 13, wherein: The determining, based on the third observation position sequence corresponding to the element, the observation speed corresponding to the element, includes: Fitting a first straight line between the observation position and time corresponding to the element based on the third observation position sequence corresponding to the element; filtering the third observation position sequence based on the third observation position sequence and the first straight line to obtain a filtered fifth observation position sequence; Based on the fifth observation position sequence, a second straight line between the observation position and time corresponding to the element is obtained by fitting; Based on the second straight line, an observed speed corresponding to the element is determined.

15. The method according to claim 13, wherein The determining, based on the observed speed corresponding to the element, the first observed speed of the object at the first moment, includes: determining a first velocity residual between the observed velocity corresponding to the element and the third velocity based on the observed velocity corresponding to the element and a third velocity of the object at a third moment, wherein the third moment is before the first moment; determining a first velocity observation confidence level corresponding to the element based on the first velocity residual; The first observed speed of the object at the first moment is determined based on the observed speed corresponding to the element and the first speed observation confidence.

16. The method according to claim 12, wherein: The second sensor is a second camera with a second field of view; the second field of view is smaller than the first field of view corresponding to the first sensor; The fourth observation position at the second moment includes observation positions respectively determined based on at least one element of the object in a second image, the at least one element including a second height and / or a second width of an area box of the object in the second image, and the second image is an image captured by the second camera at the second moment; Determining a second observed velocity of the object at the first moment based on a fourth observed position of the object obtained by the second sensor at the second moment includes: determining, based on the fourth observation position at the second moment, a fourth observation position sequence corresponding to each of the at least one element; determining an observation speed corresponding to the element based on the fourth observation position sequence corresponding to the element; The second observed speed of the object at the first moment is determined based on the observed speed corresponding to the element.

17. The method according to claim 16, wherein The determining, based on the fourth observation position sequence corresponding to the element, the observation speed corresponding to the element, includes: fitting a third straight line between the observation position and time corresponding to the element based on the fourth observation position sequence corresponding to the element; filtering the fourth observation position sequence based on the fourth observation position sequence and the third straight line to obtain a filtered sixth observation position sequence; Based on the sixth observation position sequence, a fourth straight line between the observation position and time corresponding to the element is obtained by fitting; Based on the fourth straight line, an observed speed corresponding to the element is determined.

18. The method according to claim 16, wherein The determining, based on the observed speed corresponding to the element, the second observed speed of the object at the first moment, includes: determining a second velocity residual between the observed velocity corresponding to the element and the third velocity based on the observed velocity corresponding to the element and a third velocity of the object at a third moment, wherein the third moment is before the first moment; determining a second velocity observation confidence level corresponding to the element based on the second velocity residual; The second observed speed of the object at the first moment is determined based on the observed speed corresponding to the element and the second speed observation confidence.

19. The method according to any one of claims 1 to 11, wherein: The first sensor is a first camera with a first field of view; the second sensor is a second camera with a second field of view; the first field of view angle is greater than the second field of view angle; The determining of first object information obtained by the autonomous mobile device observing the object through the first sensor at the first moment includes: Determine a first area frame of the object in a first image captured by the first camera; determining the first object information based on the first area frame and a first parameter of the first camera; and / or, The determining of second object information obtained by the autonomous mobile device observing the object through a second sensor at the first moment includes: determining a second area frame of the object in a second image captured by the second camera; The second object information is determined based on the second area frame and a second parameter of the second camera.

20. The method according to claim 19, wherein The determining the first object information based on the first area frame and the first parameter of the first camera includes: determining a first distance of the object based on a first height of the first area frame and the first parameter; determining a second distance of the object based on a first width of the first area frame and the first parameter; determining a first horizontal coordinate of the object based on a pixel distance between the first area frame and a center point of the first image and the first parameter; determining the first object information based on the first distance, the second distance, and the first lateral coordinate; and / or, The determining the second object information based on the second area frame and the second parameter of the second camera includes: determining a third distance of the object based on a second height of the second area frame and the second parameter; determining a fourth distance of the object based on a second width of the second area frame and the second parameter; determining a second horizontal coordinate of the object based on a pixel distance between the second area frame and a center point of the second image and the second parameter; The second object information is determined based on the third distance, the fourth distance, and the second lateral coordinate.

21. An apparatus for determining a state of an object, comprising: A first processing module is configured to determine first object information obtained by the autonomous mobile device observing the object through a first sensor at a first moment; a second processing module, configured to determine second object information obtained by the autonomous mobile device observing the object through a second sensor at the first moment; a third processing module for determining a topological map of the position, velocity, and acceleration of the object; The topological graph includes variable nodes and edges representing constraint relationships between the variable nodes; A fourth processing module, configured to determine an objective function based on the topological graph; A fifth processing module, configured to determine an initial state of the variable node in the topological graph based on third object information of the object at the second moment; The second moment is before the first moment; a sixth processing module, configured to iteratively update the state of the variable node based on the first object information, the second object information, the initial state of the variable node, and the objective function, to obtain a target state corresponding to the variable node; A seventh processing module is configured to determine target object information of the object at the first moment based on the target state corresponding to the variable node.

22. A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to perform the method according to any one of claims 1 to 20.

23. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 20.

Images