Data processing method, control method, computing device and storage medium

Abstract

The embodiment of the invention provides a data processing method, a control method, computing equipment and a storage medium, and relates to the technical field of data processing. The data processing method comprises the following steps: acquiring first environment data and second environment data of the mobile equipment; for at least part of first data points of the first environment data, determining target data points corresponding to the first data points from a plurality of second data points of the second environment data according to first sensor parameters of the first sensor and second sensor parameters of the second sensor, taking the second object semantic information corresponding to the target data point as third object semantic information corresponding to the first data point; generating perceptible information of the first data point according to the first object semantic information and the third object semantic information; wherein the perceptible information is used for representing whether the first data point can be perceived by the second sensor.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more specifically, to a data processing method, a control method, a computing device, and a storage medium. Background Technology

[0002] For mobile devices such as vehicles, motion control can be achieved based on driving automation technology. Mobile devices equipped with driving automation capabilities can deploy one or more sensors to perceive their environment based on sensor data. This includes identifying various environmental objects such as vehicles, pedestrians, and road signs as environmental perception results, and then controlling the motion of the mobile device based on these results. Therefore, the accuracy of environmental perception is crucial for the safety of mobile device control.

[0003] In related technologies, to improve the accuracy of environmental perception, mobile devices can deploy multiple sensors and identify environmental objects by fusing data from these sensors. However, due to the inherent physical limitations of different sensors, their fields of view (FOV) vary, meaning some areas in the environment may be in blind spots of some or all sensors. When performing environmental perception tasks such as object recognition using sensor data, these undetectable areas often fail to be effectively identified and processed. When an object enters or is located in such an area, incomplete or conflicting sensing data can lead to misidentification, affecting the accuracy of environmental perception. Therefore, accurately assessing the sensor visibility of various areas in the environment is a pressing issue. Summary of the Invention

[0004] In view of this, the present disclosure proposes a new technical solution for data processing.

[0005] According to a first aspect of the present disclosure, a data processing method is provided, the method comprising: Acquire first environmental data and second environmental data of a mobile device; wherein the mobile device is equipped with a first sensor and a second sensor, the first environmental data is generated at least in part based on data sensed by the first sensor, the second environmental data is generated at least in part based on data sensed by the second sensor, the first environmental data includes a plurality of first data points and first object semantic information corresponding to each first data point, and the second environmental data includes a plurality of second data points and second object semantic information corresponding to each second data point; For at least a portion of the first data points of the first environmental data, a target data point corresponding to the first data point is determined from a plurality of second data points of the second environmental data based on the first sensor parameters of the first sensor and the second sensor parameters of the second sensor, and the second object semantic information corresponding to the target data point is used as the third object semantic information corresponding to the first data point. Based on the semantic information of the first object and the semantic information of the third object, perceptible information of the first data point is generated; wherein, the perceptible information is used to characterize whether the first data point can be perceived by the second sensor.

[0006] Optionally, the target object semantic information of the target data point includes the type and / or height of the target environment object occupying the target data point, the target data point includes a first data point and a second data point, and the target object semantic information includes first object semantic information and second object semantic information.

[0007] Optionally, when the target object semantic information includes the type of the target environment object, generating the perceptible information of the first data point based on the first object semantic information and the third object semantic information includes: When the semantic information of the first object is the same as the semantic information of the third object, the perceptible information of the first data point is set to a first value, whereby the first value indicates that the first data point can be perceived by the second sensor; or... When the semantic information of the first object is different from the semantic information of the third object, the target type of the second environmental object occupying the target data point is determined according to the semantic information of the third object, and the perceptible information of the first data point is determined according to the target type.

[0008] Optionally, determining the perceptible information of the first data point based on the target type includes: If the target type is a first preset type, the perceptible information of the first data point is determined based on the first environmental object through which the target ray passes; wherein, the target ray is a ray pointing from the target position to the first data point, the target position is the position after the position of the second sensor is switched from the second observation view to the first observation view, the first observation view is the observation view corresponding to the first environmental data, and the second observation view is the observation view corresponding to the second environmental data.

[0009] Optionally, determining the perceptible information of the first data point based on the target type includes: If the target type is a second preset type, then the perceptible information of the first data point is determined to be the first value, and the second preset type is an object type that cannot obstruct the perception field of the second sensor; or... If the target type is a third preset type, then the perceptible information of the first data point is determined to be a second value. The second value indicates that the first data point cannot be perceived by the second sensor. The third preset type is an object type that can block the field of view of the second sensor.

[0010] Optionally, determining the target data point corresponding to the first data point from a plurality of second data points of the second environmental data based on the first sensor parameters of the first sensor and the second sensor parameters of the second sensor includes: Based on the first sensor parameters of the first sensor and the second sensor parameters of the second sensor, the first data point is transformed from the first observation view to the second observation view to obtain the target coordinates of the first data point under the second observation view; wherein, the first observation view is the observation view corresponding to the first environmental data, and the second observation view is the observation view corresponding to the second environmental data. The second data point at the target coordinates is taken as the target data point corresponding to the first data point.

[0011] Optionally, the first sensor is a lidar sensor, and the second sensor is a vision sensor or an ultrasonic sensor.

[0012] Optionally, the first observation view corresponding to the first environmental data is an overhead view, and the second observation view corresponding to the second environmental data is the observation view of the second sensor.

[0013] According to a second aspect of the present disclosure, a control method is provided, the method comprising: Acquire target environment data perceived by the second sensor of a mobile device; The target environment data is input into a pre-generated target control model, and the motion control of the mobile device is performed through the target control model. The data used to train the target control model includes first environmental data and second environmental data. The first and second environmental data are generated based on historical environmental data collected by a mobile device at historical times. The mobile device is equipped with a first sensor and a second sensor when collecting the historical environmental data. The first environmental data is generated at least partially based on data perceived by the first sensor, and the second environmental data is generated at least partially based on data perceived by the second sensor. The first environmental data includes multiple first data points and first object semantic information corresponding to each first data point. The second environmental data includes multiple second data points and second object semantic information corresponding to each second data point. At least some of the first data points in the first environmental data have perceptible information, which characterizes whether the first data point can be perceived by the second sensor. The perceptible information of the first data point is generated based on the first object semantic information and third object semantic information corresponding to the first data point. The third object semantic information is the second object semantic information corresponding to the target data point. The target data point is determined from multiple second data points in the second environmental data, corresponding to the first data point, based on the first sensor parameters of the first sensor and the second sensor parameters of the second sensor.

[0014] According to a third aspect of the present disclosure, a computing device is provided, including a memory and a processor, the memory being configured to store computer instructions, and the processor being configured to invoke the computer instructions from the memory to perform the method as described in the first or second aspect.

[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first or second aspect.

[0016] Based on the data processing method provided in this disclosure, first environmental data and second environmental data acquired by different sensors are mapped at the data point level. A target data point corresponding to a first data point in the first environmental data is determined from multiple second data points in the second environmental data. Based on the semantic information of the object occupying the target data point, perceptible information of the corresponding first data point in the field of view of the second sensor is generated. Since this method integrates object semantic information obtained from data from different sensors, it may more accurately generate perceptible information at the data point level, thereby improving the accuracy of sensor perceptible information evaluation.

[0017] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the present disclosure.

[0019] Figure 1 This is a schematic diagram of an intelligent connected system to which the methods provided in the embodiments of this disclosure can be applied.

[0020] Figure 2 It is based on Figure 1 The illustrated embodiment provides a schematic diagram of a mobile device.

[0021] Figure 3 This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure.

[0022] Figure 4 This is a flowchart illustrating another data processing method provided in an embodiment of this disclosure.

[0023] Figure 5 This is a flowchart illustrating a control method provided in an embodiment of this disclosure.

[0024] Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of this disclosure. Detailed Implementation

[0025] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0026] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0027] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0028] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0029] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0030] The elements involved in the embodiments of this disclosure may represent part or all of an element. For example, the elements involved in the embodiments of this disclosure may be at least a part of an element or all of an element.

[0031] The elements involved in the embodiments of this disclosure may be one or more, such as "a", "the", "the above", "the", "the foregoing", etc., which are used to indicate that the corresponding element is mentioned for the first time or is mentioned again, and do not have the meaning of limiting the number.

[0032] It should be noted that all actions involving the collection, storage, use, processing, transmission, provision, disclosure, and deletion of data in this disclosure are carried out in accordance with the relevant data protection laws and regulations of the country or region where the data is located, and with the full authorization of the relevant data owner.

[0033] First, the application scenarios of the embodiments of this disclosure will be described.

[0034] Figure 1 This is a schematic diagram of an intelligent connected system 100 to which the methods provided in the embodiments of this disclosure can be applied. Figure 1 As shown, the intelligent connected system 100 may include: a mobile device 101, a server 102, and a user terminal 103.

[0035] In some examples, the mobile device 101 can be a mobile device such as a vehicle, robot, ship, or aircraft, for example, a vehicle, ship, or aircraft with a driving automation feature, or an autonomously moving robot (such as a cargo robot, a probe robot, or a sweeping robot).

[0036] The driving automation function can include advanced driver assistance functions (ADAS) and automated driving functions (AWD). Automated driving, also known as intelligent driving or driverless driving, refers to vehicles equipped with driving automation functions that can perform some or all of the driving tasks, such as environmental perception, decision-making, planning, and control execution. The levels of driving automation functions can refer to the vehicle intelligence classification standards established by the Society of Automotive Engineers (SAE), for example, divided into six levels from L0 to L5. L0 is emergency assistance, L1 is partial driver assistance, L2 is combined driver assistance, L3 is conditional automated driving, L4 is highly automated driving, and L5 is fully automated driving. The above classification of driving automation function levels is merely an example, and this disclosure does not limit the classification standards and levels of driving automation functions.

[0037] In some examples, server 102 can be a single server or a distributed server cluster consisting of multiple servers, and its deployment method can include local servers or cloud servers. Server 102 can communicate with mobile device 101 and / or user terminal 103 via a communication network, providing various services to mobile device 101 and / or user terminal 103. For example, the server can receive sensing data sent by mobile device 101, provide services such as high-precision maps, data analysis, and decision planning for mobile device 101, or receive query commands or control commands sent by user terminal 102, providing corresponding services to the user.

[0038] In some examples, user terminal 103 can be any form of electronic device providing services to the user, such as a personal computer, laptop, smart tablet, smartphone, smart wearable device, etc. The user can interact with the mobile device or server through the human-computer interaction terminal configured on the mobile device 101, or through user terminal 103. For example, the user can query the status and / or parameters of the mobile device, or control the mobile device to perform set tasks and / or modify configuration parameters, etc. The user terminal runs an application based on the intelligent network system to achieve interaction with the mobile device or server. This application can be a local application, a web application, or a mini-program, etc., and is not limited thereto.

[0039] In some examples, the aforementioned application running on the user's terminal can provide authentication or authorization services to the user. The user who is successfully authenticated and granted the corresponding permissions can query and / or control the mobile device within the scope of the granted permissions.

[0040] The mobile device 101, server 102, and user terminal 103 can communicate via a communication link provided by communication network 104. This communication network 104 can include one or more networks of any type, such as the Internet, Local Area Network (LAN), Wide Area Network (WAN), Virtual Private Network (VPN), Public Switched Telephone Network (PSTN), satellite communication network, Wi-Fi, 2G, 3G, 4G, 5G, 6G, NB-IoT, eMTC, infrared, Bluetooth, NFC, or a combination of these networks. The communication networks between the mobile device 101 and server 102, between the user terminal 103 and server 102, and between the user terminal 103 and mobile device 101 can be the same or different.

[0041] It should be noted that, Figure 1The structure of the intelligent connected system 100 shown is merely illustrative. The intelligent connected system in this embodiment is not limited to the above structure and may include more or fewer devices as needed, and the devices may be combined or split. For example, the intelligent connected system may not include user terminals and / or servers; as another example, user terminals and servers may be deployed together.

[0042] Figure 2 It is based on Figure 1 The illustrated embodiment provides a schematic diagram of a mobile device 101. As shown... Figure 2 As shown, the mobile device 101 may include a sensing component 1011, a computing platform 1012, an execution component 1013, etc. The sensing component 1011, the computing platform 1012, and the execution component 1013 may be connected via a bus or other means.

[0043] In some examples, the sensing component 1011 can be used to collect information about the mobile device itself or externally. The sensing component 1011 may include at least one of a visual sensing unit, radar, positioning and navigation unit, inertial measurement unit (IMU) or other sensing unit. The visual sensor unit may include one or more cameras, the radar may include at least one of lidar, millimeter-wave radar, ultrasonic radar or other radar, and the positioning and navigation unit may include at least one of a GPS system, BeiDou system or other global positioning system.

[0044] In some examples, the computing platform 1012 may include a computing-capable device for processing the sensing information collected by the sensing component 1011 to obtain control information, and sending corresponding control commands to the execution component 1013 to cause the execution component 1013 to perform corresponding actions, thereby realizing the control of the mobile device 101. For example, the computing platform 1012 can perform one or more of the following actions on the mobile device: information collection and processing, positioning, decision-making, planning, and control, thereby realizing the autonomous control of the mobile device. The computing platform 1012 may include at least one processor and at least one memory, wherein each processor can individually or jointly execute instructions stored in the memory to implement the methods provided in the embodiments of this disclosure. The processor in this disclosure embodiment may include at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Neural-network Processing Unit (NPU), Tensor Processing Unit (TPU), Data Processing Unit (DPU), Digital Signal Processor (DSP), Field Programmable Gate Array (FPGA), Programmable Logic Array (PLA), System on Chip (SOC), Application Specific Integrated Circuit (ASIC), Micro Controller Unit (MCU), or other processors. The memory may be implemented using any type of volatile or non-volatile computer-readable storage medium or a combination thereof. In addition to storing instructions, the memory may also store data, such as map data, image data, sound data, text data, configuration parameters of the mobile device, location, orientation, speed, etc. The data stored in the memory can be accessed and used by the processor.

[0045] In some examples, the computing platform of a mobile device can perform computing tasks independently or communicate with a server to complete computing tasks. For example, the computing platform of a mobile device can cooperate with a server to complete corresponding computing tasks. These computing tasks can include any task performed to achieve autonomous control of the mobile device, such as information collection and processing, positioning, decision-making, planning, or control of the mobile device.

[0046] The computing platform 1012 can be located in the mobile device 101. Some or all of the computing platform 1012 can also be located in the server corresponding to the mobile device. For example, some functions of the computing platform 1012 with high real-time requirements can be located in the mobile device, while other functions with low real-time requirements can be located in the server corresponding to the mobile device.

[0047] In some examples, the execution component 1013 is used to perform corresponding actions based on the control of the computing platform 1012, enabling the mobile device 101 to complete the movement task. The execution component 1013 may include, for example, a power component, a braking component, a transmission component, a steering component, etc.

[0048] It should be noted that, Figure 2 The structure of the mobile device 101 shown is merely illustrative. The mobile device in this embodiment is not limited to the above structure and may include more or fewer components as needed. The device may also be combined or disassembled. For example, the mobile device may not include the aforementioned computing platform. Furthermore, the mobile device may also include communication components, interface components, multimedia components, input components, output components, display components, etc.

[0049] In some embodiments of this disclosure, the mobile device may be configured with a control system, which may include some or all of the aforementioned sensing components, computing platform and execution components. The control system may, alone or in cooperation with the user, realize one or more of the functions of environmental perception, decision planning and control execution of the mobile device.

[0050] This disclosure can be applied to scenarios involving the processing of sensor data from mobile devices. For example, a mobile device can deploy multiple sensors and identify environmental objects in its surroundings by fusing data from these sensors, thus achieving environmental perception. However, due to differences in the field of view of different sensors, certain areas in the environment may be within the blind spots of some or all sensors. In related technologies, when performing environmental perception tasks such as object recognition using sensor data, these areas that cannot be perceived by the sensors are often not effectively identified and processed. When an environmental object to be identified enters or is located in such an area, misidentification may occur due to incomplete or conflicting perception data, affecting the accuracy of environmental perception.

[0051] Taking visual sensors as an example, in related technologies, after acquiring images captured by a visual sensor, semantic segmentation can be performed on the images to obtain semantic segmentation results. Based on inverse perspective mapping (IPM), the semantic segmentation results are transformed from the visual sensor's perspective to a top-down perspective, providing perceptible information about each region or location. However, the core premise of IPM is the assumption that the environment observed by the visual sensor is a perfectly flat plane. However, the real environment contains slopes, undulations, or unevenness. Any object not on this assumed plane (such as vehicles, pedestrians, etc.) may have an incorrect projection position. This causes the shape and position of these objects to be distorted in the top-down perspective, failing to accurately reflect their true geometric information. Therefore, this method is significantly affected by the constraints of the ground assumption, and the objects may undergo distortion after projection, resulting in inaccurate projection results and an inability to accurately assess the sensor's perceptibility in various areas of the environment.

[0052] To address the problems in related technologies, this disclosure provides embodiments such as Figure 3 The diagram shows a flowchart of the data processing method. This data processing method can be... Figure 1 The illustrated mobile device and / or server can execute the command, but it can also be executed by any computing device. For example... Figure 3 As shown, the data processing method of this embodiment may include the following steps S310 to S330.

[0053] Step S310: Obtain the first environmental data and the second environmental data of the mobile device.

[0054] The mobile device may be equipped with a first sensor and a second sensor. The first environmental data is generated at least in part based on data sensed by the first sensor, and the second environmental data may be generated at least in part based on data sensed by the second sensor.

[0055] For example, the data sensed by the first sensor can be first sensor data, and the data sensed by the second sensor can be second sensor data. The first environmental data can be image data or point cloud data generated solely from the parsing of the first sensor data, or it can be generated by parsing the fusion of the first sensor data and the second sensor data, or it can be generated by parsing the fusion of the first sensor data and data sensed by other sensors of the mobile device (such as device posture information). Similarly, the second environmental data can be image data or point cloud data generated solely from the parsing of the second sensor data, or it can be generated by parsing the fusion of the second sensor data and the first sensor data, or it can be generated by parsing the fusion of the second sensor data and data sensed by other sensors of the mobile device (such as device posture information).

[0056] The first and second sensors described above can be of the same type or different types. They can be deployed at the same or different locations on the mobile device. Their sensing ranges may overlap. For example, the sensing range of the first sensor may be greater than that of the second sensor; for instance, the maximum sensing distance of the first sensor may be greater than or equal to the maximum sensing distance of the second sensor; further, the sensing range of the second sensor may be a subset of the sensing range of the first sensor.

[0057] In some examples, the first sensor can be deployed on the top of the mobile device to acquire first environmental data. The mobile device can deploy multiple second sensors, which can be deployed at different positions on the front, back, left, and right sides of the main body of the mobile device to acquire second environmental data in different directions. Different second environmental data can correspond to different second observation perspectives.

[0058] In some examples, the first sensor can be a LiDAR sensor, such as one deployed on the top of a mobile device; the second sensor can be a vision sensor (e.g., a camera) or an ultrasonic sensor, such as surround-view cameras or ultrasonic sensors deployed in multiple directions (front, back, left, and right) on the main body of the mobile device. In this way, the mobile device can obtain first environmental data over a large area around the device through the first sensor, and second environmental data over smaller areas in different directions through the second sensor. The environmental information obtained from the first and second sensors can be fused to improve the reliability of environmental perception. Alternatively, confidence testing can be performed on the environmental information obtained from the second sensor based on the environmental information obtained from the first sensor to correct the perception algorithm related to the second sensor, thereby improving the accuracy of environmental perception based solely on the second sensor.

[0059] In some examples, the first observation view corresponding to the first environmental data can be a bird's-eye view (BEV), such as an aerial view with the center point of the mobile device as the origin. This first environmental data can also be called a bird's-eye view or a BEV image. The second observation view corresponding to the second environmental data can be the observation view of the second sensor, such as a sensor view with the position of the second sensor as the origin. If the second sensor is a camera, then the second observation view can be a camera view. This second observation view and the first observation view can be different.

[0060] In some examples, the first environment data may include multiple first data points and semantic information of a first object corresponding to each first data point. This semantic information may be the semantic information of the first environment object occupying that first data point, such as the type and / or height of the first environment object. The first environment data may include one or more first environment objects, and a first environment object may occupy one or more first data points. The first environment object may be, for example, a road surface, curb, fence, lane line, mobile device, or pedestrian. The first environment object may include a first static environment object and / or a first dynamic environment object.

[0061] For example, the first sensor may be a lidar sensor, the first environmental data may be radar point cloud data generated based on the data perceived by the first sensor, the first data point may be a data point in the radar point cloud, and the radar point cloud data may include one or more first environmental objects, each first environmental object occupying one or more first data points.

[0062] In some examples, the second environment data may include multiple second data points and semantic information of a second object corresponding to each second data point. This semantic information may be the semantic information of the second environment object occupying that second data point, such as the type and / or height of the first environment object. Similar to the first environment data, the second environment data may also include one or more second environment objects. A second environment object may occupy one or more second data points; for example, it may also include a second static environment object and / or a second dynamic environment object.

[0063] For example, the second sensor may be a vision sensor, the second environmental data may be image data generated based on the data perceived by the second sensor, the second data point may be a pixel in the image data, and the image data may include one or more second environmental objects, each second environmental object occupying one or more second data points (e.g., pixels).

[0064] In some examples, the first data point and the second data point mentioned above can be collectively referred to as the target data point, and the first object semantic information and the second object semantic information mentioned above can be collectively referred to as the target object semantic information. The target object semantic information of the target data point may include the type and / or height of the target environment object (e.g., the first environment object or the second environment object) that occupies the target data point.

[0065] For example, this type can include occlusion types and non-occlusion types. The occlusion type can characterize the type of target environment object that can obstruct the second sensor's field of view. This occlusion type can include objects such as buildings, vehicles, and large vegetation. The non-occlusion type can characterize the type of target environment object that cannot obstruct the second sensor's field of view. This non-occlusion type can include low-profile objects such as lane lines. Optionally, only the occlusion type can be defined, and objects of other types can be non-occlusion types. The object types included in this occlusion type can be configured by the user. The occlusion type can obstruct the second sensor's field of view, causing other data points located in the direction away from the second sensor from this data point to be undetectable by the second sensor.

[0066] The height can be a height value. If the height is greater than a set height threshold, it can be determined that the target environment object can block the perception field of the second sensor. The set height threshold can be a value preset based on engineering experience, or it can be a value determined based on the height of the second sensor. For example, the preset height threshold can be equal to the height of the second sensor.

[0067] In some examples, the semantic information of the target object of the target data point may include the type and height of the target environmental object occupying the target data point. If the type is occlusion type and the height is greater than a set height threshold, it can be determined that the target environmental object can occlude the perception field of the second sensor.

[0068] It should be noted that the semantic information of the target object may also include whether the target environment object occupying the target data point occupies the field of view of the second sensor, such as two values: occlusion or no occlusion, or occlusion probability. The value range of the occlusion probability can be 0 to 1. For example, the occlusion probability of a building can be 1, the occlusion probability of a vehicle can be 1, the occlusion probability of vegetation can be 0.6, and the occlusion probability of a pedestrian can be 0.8.

[0069] It should be noted that the first data point and the second data point can have semantic information of the same dimension or semantic information of different dimensions. For example, the first object semantic information of the first data point includes type, and the second object semantic information of the second data point includes type, height and color; or, the first object semantic information of the first data point includes type and height, and the second object semantic information of the second data point includes type.

[0070] In this way, based on the semantic information of the target object, it can be determined whether the target environmental object occupying the target data point can block the perception field of the second sensor.

[0071] Taking the target data point as a second data point as an example, the second object semantic information of the second data point can be used to determine whether the second data point can occlude the perception field of the second sensor. For example, it can be used to determine whether a second environmental object occupying the second data point can occlude the perception field of the second sensor. The second object semantic information can be determined at least partially based on the data perceived by the second sensor. The data perceived by the second sensor can be referred to as second sensor data. For example, the second object semantic information can be determined at least partially based on the second sensor data. For instance, the second sensor data (e.g., an image or point cloud) can be input into a semantic segmentation model to obtain the second object semantic information of each second data point output by the semantic segmentation model. The semantic segmentation model can be a pre-trained neural network model. The specific structure of the second semantic segmentation model can be found in the description in related technologies, which will not be repeated in this embodiment.

[0072] Step S320: For at least a portion of the first data points of the first environmental data, a target data point corresponding to the first data point is determined from a plurality of second data points of the second environmental data according to the first sensor parameters of the first sensor and the second sensor parameters of the second sensor, and the second object semantic information corresponding to the target data point is used as the third object semantic information corresponding to the first data point.

[0073] In some examples, the first sensor and the second sensor can be referred to as the target sensor. The target sensor parameters (e.g., the first sensor parameters of the first sensor or the second sensor parameters of the second sensor) can include intrinsic and / or extrinsic parameters of the target sensor. Intrinsic parameters can describe the internal properties of the target sensor. For example, if the target sensor is a camera, the intrinsic parameters can include focal length, optical center coordinates, distortion coefficients, etc. Similarly, if the target sensor is a LiDAR, the intrinsic parameters can include virtual focal length, scanning angle, horizontal offset, vertical offset, etc. Extrinsic parameters can describe the position and orientation of the target sensor in the target coordinate system. For example, they can include the rotation matrix and / or translation vector of the target sensor relative to the target coordinate system. The target coordinate system can be a device coordinate system with the center point of the mobile device as the origin or a global coordinate system. For example, the target coordinate system can be a coordinate system from a bird's-eye view, i.e., the coordinate system used in a bird's-eye view or a BEV diagram.

[0074] In some examples, the observation perspective corresponding to the first environmental data can be a first observation view (e.g., an overhead view), meaning the first environmental data is data from the first observation view. Similarly, the observation perspective corresponding to the second environmental data can be a second observation view (e.g., the observation view of the second sensor), meaning the second environmental data is data from the second observation view. Based on the first sensor parameters of the first sensor and / or the second sensor parameters of the second sensor, coordinate transformation (e.g., projection transformation) can be implemented between the first and second observation views. For example, some or all of the first data points in the first environmental data can be transformed from the first observation view to the second observation view to obtain the coordinates of the first data points from the second observation view; similarly, some or all of the second data points in the second environmental data can be transformed from the second observation view to the first observation view to obtain the coordinates of the second data points from the first observation view.

[0075] In some examples, the first data point can be transformed from a first observation view to a second observation view based on the first sensor parameters of the first sensor and the second sensor parameters of the second sensor, so as to obtain the target coordinates of the first data point under the second observation view; the second data point at the target coordinates is taken as the target data point corresponding to the first data point.

[0076] It should be noted that the specific implementation of coordinate transformation between different observation perspectives (e.g., different coordinate systems) based on sensor parameters can be found in the descriptions in relevant technologies, and will not be repeated here.

[0077] In this way, data point-level mapping can be achieved based on sensor parameters under different observation perspectives, improving the accuracy and precision of the correspondence between different environmental data, so as to generate more accurate data point-level perceptible information.

[0078] In some examples, after the first data point is transformed from the first observation view to the second observation view and the target coordinates of the first data point are obtained in the second observation view, there is no second data point at the target coordinates. That is, the projected target coordinates are outside the range of the second environmental data. Then, the target data point corresponding to the first data point can be defined as an empty data point or an invalid data point. The second environmental object occupying the target data point can be defined as an empty object. The semantic information of the second object occupying the target data point can be located as an empty attribute. This empty attribute can be used to characterize that the first data point is outside the perception field of the second sensor. Based on this empty attribute, the perceptible information of the first data point can be set as a second value. This second value can characterize that the first data point cannot be perceived by the second sensor. For example, the environmental object occupying the first data point is the first environmental object. The part of the first environmental object occupying the first data point cannot be perceived by the second sensor.

[0079] In some examples, the first observation viewpoint can be an observation viewpoint corresponding to the first environmental data, such as an overhead viewpoint, and the second observation viewpoint can be an observation viewpoint corresponding to the second environmental data, such as the sensor observation viewpoint of the second sensor. For example, the second sensor is a camera, and the sensor observation viewpoint is the camera observation viewpoint. The second environmental data can be a two-dimensional image captured by the camera. In this way, projecting the first data points from the overhead viewpoint onto the two-dimensional image can achieve a more accurate coordinate transformation.

[0080] Step S330: Generate perceptible information of the first data point based on the semantic information of the first object and the semantic information of the third object.

[0081] The perceptible information can be used to characterize whether the first data point can be perceived by the second sensor. For example, if the first data point is occupied by a first environmental object, the perceptible information can be used to characterize whether the portion of the first environmental object occupying the first data point can be perceived by the second sensor.

[0082] For example, the semantic information of a first object and the semantic information of a third object corresponding to the same first data point can be compared. If the semantic information of the first object and the semantic information of the third object are the same or have a similarity greater than a preset similarity threshold, it indicates that both the first sensor and the second sensor have perceived the same object at the location of the first data point. In this case, the perceptible information of the first data point can be set as a first value, which can be used to characterize that the first data point can be perceived by the second sensor. Conversely, if the semantic information of the first object and the semantic information of the third object are different or have a similarity less than a preset similarity threshold, it indicates that the first sensor and the second sensor have perceived different objects at the location of the first data point. Since the reliability of the first sensor is greater than that of the second sensor, the perceptible information of the first data point can be set as a second value, which can be used to characterize that the first data point cannot be perceived by the second sensor. For example, the part of the first environmental object occupying the first data point cannot be perceived by the second sensor. Furthermore, when the semantic information of the first object and the semantic information of the third object are different, it is possible to further determine whether the perception field of the second sensor is obstructed based on the semantic information of the third object, so as to determine the perceptible information of the first data point.

[0083] Using the data processing method described in steps S310 to S330 above, first environmental data and second environmental data of the mobile device are acquired; for at least a portion of the first data points of the first environmental data, a target data point corresponding to the first data point is determined from multiple second data points of the second environmental data based on the first sensor parameters of the first sensor and the second sensor parameters of the second sensor; and the second object semantic information corresponding to the target data point is used as the third object semantic information corresponding to the first data point; perceptible information of the first data point is generated based on the first object semantic information and the third object semantic information; wherein, the mobile device may be equipped with a first sensor and a second sensor, the first environmental data is generated at least partially based on the data perceived by the first sensor, the second environmental data is generated at least partially based on the data perceived by the second sensor, the first environmental data includes multiple first data points and the first object semantic information corresponding to each first data point, the second environmental data includes multiple second data points and the second object semantic information corresponding to each second data point; the perceptible information is used to characterize whether the first data point can be perceived by the second sensor. In this way, first and second environmental data acquired by different sensors are mapped at the data point level. From multiple second data points in the second environmental data, a target data point corresponding to a first data point in the first environmental data is determined. Based on the semantic information of the object occupying the target data point, perceptible information of the corresponding first data point in the second sensor's field of view is generated. Because this method integrates object semantic information obtained from data from different sensors, it may more accurately generate perceptible information at the data point level, thereby improving the accuracy of sensor perceptible information assessment.

[0084] In some embodiments of this disclosure, when the target object semantic information (e.g., first object semantic information, second object semantic information, or third object semantic information) includes the type of the target environment object, the specific method for generating the perceptible information of the first data point based on the first object semantic information and the third object semantic information in step S330 may include: When the semantic information of the first object is the same as that of the third object, the perceptible information of the first data point is set to a first value, which indicates that the first data point can be perceived by the second sensor; or, When the semantic information of the first object differs from that of the third object, the target type of the second environmental object occupying the target data point is determined based on the semantic information of the third object, and the perceptible information of the first data point is determined based on the target type. There are multiple ways to determine the perceptible information of the first data point based on the target type, such as any one of the following methods one to three.

[0085] Method 1: If the target type is the first preset type, the perceptible information of the first data point can be determined based on the first environmental object through which the target ray passes.

[0086] The target ray can be a ray pointing from the target position to the first data point. The target position can be the position after the position of the second sensor is changed from the second observation view to the first observation view. The first observation view can be the observation view corresponding to the first environmental data, such as an overhead view. The second observation view can be the observation view corresponding to the second environmental data, such as the sensor observation view of the second sensor.

[0087] For example, the target position of the target ray can be determined in the first environmental data under the first observation view. Starting from the target position, the data points are traversed one by one along the direction of executing the first data point. During the traversal, if the first environmental object occupying any data point can block the perception field of the second sensor, the perceptible information of the first data point can be set to a second value. The second value can characterize that the first data point cannot be perceived by the second sensor.

[0088] Conversely, if, after traversal, none of the data points on the ray can obstruct the field of view of the second sensor, then the perceptible information of the first data point can be set to a first value, which indicates that the first data point can be perceived by the second sensor. It should be noted that the specific method for determining the perceptible information of data points based on ray traversal can also be found in descriptions in related technologies, and this embodiment does not limit it.

[0089] The aforementioned first preset type can be an object type located at the boundary of the passable area, pre-defined based on engineering experience, such as curbs or fences. The boundary of the passable area can be determined based on this first preset type; therefore, its perceptible information needs to be more accurate. Thus, the perceptible information of the first data point can be further determined based on the target ray, thereby further improving the accuracy of the perceptible information.

[0090] Method 2: If the target type is a second preset type, then the perceptible information of the first data point can be determined as a first value. This first value indicates that the first data point can be perceived by the second sensor. The second preset type can be an object type that does not obstruct the second sensor's field of vision, such as lane lines or other non-obstructing types.

[0091] Method 3: If the target type is a third preset type, then the perceptible information of the first data point can be determined as the second value. This second value indicates that the first data point cannot be perceived by the second sensor. The third preset type can be an object type that can obstruct the field of view of the second sensor, such as buildings, vehicles, or large vegetation.

[0092] In this way, when the semantic information of the first object is different from that of the third object, the target type of the second environmental object occupying the target data point is determined based on the semantic information of the third object, and the perceptible information of the first data point is determined based on the target type, thereby obtaining more accurate and reliable perceptible information.

[0093] Figure 4 This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure. This data processing method can be... Figure 1 The illustrated mobile device and / or server can execute the command, but it can also be executed by any computing device. For example... Figure 4 As shown, the data processing method of this embodiment may include the following steps S410 to S440.

[0094] Step S410: Obtain first sensor data sensed by the first sensor and second sensor data sensed by the second sensor.

[0095] For example, the first sensor could be a lidar sensor, and the second sensor could be a vision sensor, such as a camera with different viewing angles.

[0096] Step S420: Generate first environmental data based on the first sensor data.

[0097] For example, the initial environmental data can be obtained by parsing the data from the first sensor, and then the first environmental data can be generated based on the initial environmental data.

[0098] The initial environmental data can be either manually annotated or obtained after model inference. The process of generating the first environmental data based on the initial environmental data can interpolate the initial annotation results in the initial environmental data to densify the sparse annotation results, thereby improving the accuracy of data points and facilitating the acquisition of sensor observation rays (such as camera rays) from an overhead view.

[0099] For example, based on the initial annotation results, the first environmental objects in the first environmental data can be determined, such as lane lines, road markings, and curbs. Since these environmental objects are annotated as points and lines, interpolation is needed to make them more even and continuous, thus achieving a denser annotation result.

[0100] Furthermore, for environmental objects of the first preset type, such as curbs, fences, etc., the ray from the position of the second sensor to the position of the environmental object can be calculated. Based on the intersection of the ray and the semantic segmentation result of the image, it can be used to help determine whether these environmental objects can be perceived by the second sensor.

[0101] Step S430: Generate second environmental data based on the second sensor data.

[0102] For example, the second sensor data is two-dimensional image data. The two-dimensional image data can be processed, such as image semantic segmentation. Based on the segmentation result, multiple second data points in the second environmental data and the second object semantic information of each second data point can be determined. The second object semantic information can include occlusion class and non-occlusion class.

[0103] For example, after the second sensor data, a deep learning method can be used to perform semantic segmentation on the second sensor data (e.g., an image) to obtain the type of each second data point (e.g., an image pixel). Based on the type, it is possible to define which are occlusion types and which are non-occlusion types.

[0104] Step S440: Associate the first environmental data with the second environmental data, and generate perceptible information of at least some of the first data points in the first environmental data based on the association result.

[0105] For example, a first data point in the first environmental data can be projected onto the second environmental data, semantic information can be obtained based on the projection result, and the perceptible information of the first data point can be determined based on the semantic category obtained after projection.

[0106] In some examples, for at least a portion of the first data points of the first environmental data, a target data point corresponding to the first data point can be determined from a plurality of second data points of the second environmental data based on the first sensor parameters of the first sensor and the second sensor parameters of the second sensor. The second object semantic information corresponding to the target data point is then used as the third object semantic information corresponding to the first data point. Based on the first object semantic information and the third object semantic information, perceptible information of the first data point is generated.

[0107] It should be noted that the specific implementation method for generating perceptible information can be referred to the description in the foregoing embodiments of this disclosure, and will not be repeated here.

[0108] Using this method, by combining two-dimensional image semantic segmentation with first environmental data from an overhead view, the sensor visibility of each first data point of the first environmental data under the overhead view can be accurately obtained, that is, whether each first data point can be perceived by the second sensor.

[0109] Figure 5 This is a flowchart illustrating a control method provided in an embodiment of this disclosure. The control method can be... Figure 1 The illustrated mobile device and / or server execute. For example... Figure 5 As shown, the control method of this embodiment may include the following steps S510 to S520.

[0110] Step S510: Acquire target environment data perceived by the second sensor of the mobile device.

[0111] Step S520: Input the target environment data into the pre-generated target control model, and use the target control model to perform motion control on the mobile device.

[0112] In some examples, the target control model can parse target environment data to obtain corresponding environmental information, and at least based on this environmental information, perform motion control on the mobile device. For example, it can make decisions or plans based on the environmental information and the device information of the mobile device, and then perform motion control on the mobile device. For instance, if the mobile device is a vehicle, the target control model can be a model with driving automation functions, such as a vehicle's intelligent driving model. It should be noted that the specific model structure and training method of the target control model can be found in descriptions in related technologies, and this disclosure does not limit these aspects.

[0113] It should be noted that the input to this target control model may also include other data besides the target environment data, such as the current and / or historical status of the mobile device.

[0114] In some examples, the data used to train the target control model includes first environmental data and second environmental data. The first and second environmental data are generated based on historical environmental data collected by a mobile device at historical times. The mobile device is equipped with a first sensor and a second sensor when collecting historical environmental data at historical times. The first environmental data is generated at least partially based on data perceived by the first sensor, and the second environmental data is generated at least partially based on data perceived by the second sensor. The first environmental data includes multiple first data points and first object semantic information corresponding to each first data point. The second environmental data includes multiple second data points and second object semantic information corresponding to each second data point. At least some of the first data points in the first environmental data have perceptible information, which is used to characterize whether the first data point can be perceived by the second sensor. The perceptible information of the first data point is generated based on the first object semantic information and third object semantic information corresponding to the first data point. The third object semantic information is the second object semantic information corresponding to the target data point. The target data point is a data point corresponding to the first data point determined from multiple second data points in the second environmental data based on the first sensor parameters of the first sensor and the second sensor parameters of the second sensor.

[0115] It should be noted that the specific methods for obtaining the first environmental data, the second environmental data, and the perceptible information of the first data point in this example can be found in [reference needed]. Figure 3 or Figure 4The descriptions in the illustrated embodiments will not be repeated here.

[0116] Using the above method, first and second environmental data acquired by different sensors are mapped at the data point level. A target data point corresponding to a first data point in the first environmental data is determined from multiple second data points in the second environmental data. Based on the semantic information of the object occupying the target data point, perceptible information of the corresponding first data point in the second sensor's field of view is generated. Since this method integrates object semantic information obtained from data from different sensors, it may more accurately generate perceptible information at the data point level, thereby improving the accuracy of sensor perceptible information evaluation. Furthermore, targeted training of the target control model based on the first and second environmental data with perceptible information can improve the perception accuracy and control reliability of the trained target control model. For example, during model training, targeted training can be performed based on this perceptible information; for instance, the weight of grid data that the second sensor cannot perceive can be reduced, while the weight of grid data that the second sensor can perceive can be increased, thereby improving the perception accuracy and control reliability of the trained target control model.

[0117] In some examples, the first sensor may be a lidar sensor, such as a lidar sensor deployed on the top of a mobile device, and the second sensor may be a vision sensor (such as a camera) or an ultrasonic sensor, such as a surround-view camera or ultrasonic sensor deployed in multiple directions in the front, back, left, and right directions on the main body of the mobile device.

[0118] In some examples, after the target control model of the mobile device has been trained, the first sensor may no longer need to be deployed. That is, during the execution of steps S510 to S520 above, the mobile device may no longer need to acquire environmental data perceived by the first sensor, so as to reduce the number of sensors, reduce the amount of perceived data, and improve the model's computational efficiency.

[0119] In other examples, during the execution of steps S510 to S520 above, the mobile device can acquire environmental data sensed by the first sensor and input both the environmental data sensed by the first sensor and the second sensor into the target control model to further improve the control reliability of the target control model.

[0120] Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of this disclosure. Figure 6As shown, the computing device 1000 may include a memory 1010 and a processor 1020. The memory 1010 may be used to store computer instructions, and the processor 1020 may be used to retrieve computer instructions from the memory 1010 to execute all or part of the steps of any of the methods in the foregoing embodiments of this disclosure. The processor may be one or more, and the one or more processors may execute instructions individually or jointly. Similarly, the memory may be one or more, and the one or more memories may store the aforementioned computer instructions individually or jointly.

[0121] In some examples, the computing device can be Figure 1 The computing device can be a server and / or a mobile device. In other examples, the computing device can also be any electronic device, such as a controller for a mobile device.

[0122] This disclosure also provides a mobile device that may include a memory and a processor. The memory may be used to store computer instructions, and the processor may be used to retrieve the computer instructions from the memory to perform all or part of the steps of any of the methods in the foregoing embodiments of this disclosure. The processor may be one or more processors, which may execute the instructions individually or jointly. Similarly, the memory may be one or more memories, which may store the aforementioned computer instructions individually or jointly.

[0123] The mobile device provided in this embodiment can be... Figure 1 or Figure 2 The mobile device shown is, in some examples, a vehicle that can be an electric vehicle, a hybrid vehicle, a fuel cell vehicle, or another type of vehicle. For example, the vehicle could be one equipped with autonomous driving features.

[0124] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods in the foregoing embodiments of this disclosure. Optionally, the computer-readable storage medium may be a non-transitory storage medium, but is not limited thereto, and may also be a temporary storage medium.

[0125] This disclosure also provides a chip that may include a processing unit, which can be used to execute all or part of the steps of any of the methods in the foregoing embodiments of this disclosure. The chip may be in the form of an Application-Specific Integrated Circuit (ASIC), a System-on-Chip (SOC), a Field-Programmable Gate Array (FPGA), etc., and this embodiment is not limited to this. Optionally, the chip may further include a storage unit, which can be used to store computer instructions. The processing unit can be used to retrieve the computer instructions from the storage unit to execute all or part of the steps of any of the methods in the foregoing embodiments of this disclosure.

[0126] This disclosure also provides a computer program product that may include a computer program that, when executed by a processor, can implement any of the methods described in the foregoing embodiments of this disclosure.

[0127] This disclosure may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement any of the methods in the foregoing embodiments of this disclosure.

[0128] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media may include, for example, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), compact disc-read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0129] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0130] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Smalltalk, C++, etc.) and conventional procedural programming languages ​​(e.g., the "C" language or similar programming languages). The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network (e.g., a local area network or a wide area network), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays, or programmable logic arrays, may execute computer-readable program instructions to implement various aspects of the embodiments of this disclosure by utilizing state information from the computer-readable program instructions.

[0131] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It should be noted that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are all equivalent.

[0135] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Acquire first environmental data and second environmental data of a mobile device; wherein the mobile device is equipped with a first sensor and a second sensor, the first environmental data is generated at least in part based on data sensed by the first sensor, the second environmental data is generated at least in part based on data sensed by the second sensor, the first environmental data includes a plurality of first data points and first object semantic information corresponding to each first data point, and the second environmental data includes a plurality of second data points and second object semantic information corresponding to each second data point; For at least a portion of the first data points of the first environmental data, a target data point corresponding to the first data point is determined from a plurality of second data points of the second environmental data based on the first sensor parameters of the first sensor and the second sensor parameters of the second sensor, and the second object semantic information corresponding to the target data point is used as the third object semantic information corresponding to the first data point. Based on the semantic information of the first object and the semantic information of the third object, perceptible information of the first data point is generated; wherein, the perceptible information is used to characterize whether the first data point can be perceived by the second sensor.

2. The method according to claim 1, characterized in that, The target object semantic information of the target data point includes the type and / or height of the target environment object occupying the target data point. The target data point includes a first data point and a second data point. The target object semantic information includes first object semantic information and second object semantic information.

3. The method according to claim 2, characterized in that, When the target object semantic information includes the type of the target environment object, generating the perceptible information of the first data point based on the first object semantic information and the third object semantic information includes: When the semantic information of the first object is the same as the semantic information of the third object, the perceptible information of the first data point is set to a first value, whereby the first value indicates that the first data point can be perceived by the second sensor; or... When the semantic information of the first object is different from the semantic information of the third object, the target type of the second environmental object occupying the target data point is determined according to the semantic information of the third object, and the perceptible information of the first data point is determined according to the target type.

4. The method according to claim 3, characterized in that, Determining the perceptible information of the first data point based on the target type includes: If the target type is a first preset type, the perceptible information of the first data point is determined based on the first environmental object through which the target ray passes; wherein, the target ray is a ray pointing from the target position to the first data point, the target position is the position after the position of the second sensor is switched from the second observation view to the first observation view, the first observation view is the observation view corresponding to the first environmental data, and the second observation view is the observation view corresponding to the second environmental data.

5. The method according to claim 3, characterized in that, Determining the perceptible information of the first data point based on the target type includes: If the target type is a second preset type, then the perceptible information of the first data point is determined to be the first value, and the second preset type is an object type that cannot obstruct the perception field of the second sensor; or... If the target type is a third preset type, then the perceptible information of the first data point is determined to be a second value. The second value indicates that the first data point cannot be perceived by the second sensor. The third preset type is an object type that can block the field of view of the second sensor.

6. The method according to claim 1, characterized in that, The step of determining the target data point corresponding to the first data point from a plurality of second data points of the second environmental data based on the first sensor parameters of the first sensor and the second sensor parameters of the second sensor includes: Based on the first sensor parameters of the first sensor and the second sensor parameters of the second sensor, the first data point is transformed from the first observation view to the second observation view to obtain the target coordinates of the first data point under the second observation view; wherein, the first observation view is the observation view corresponding to the first environmental data, and the second observation view is the observation view corresponding to the second environmental data. The second data point at the target coordinates is taken as the target data point corresponding to the first data point.

7. The method according to any one of claims 1 to 6, characterized in that, The first sensor is a lidar sensor, and the second sensor is a vision sensor or an ultrasonic sensor; The first observation perspective corresponding to the first environmental data is the overhead view, and the second observation perspective corresponding to the second environmental data is the observation perspective of the second sensor.

8. A control method, characterized in that, The method includes: Acquire target environment data perceived by the second sensor of a mobile device; The target environment data is input into a pre-generated target control model, and the motion control of the mobile device is performed through the target control model. The data used to train the target control model includes first environmental data and second environmental data. The first and second environmental data are generated based on historical environmental data collected by a mobile device at historical times. The mobile device is equipped with a first sensor and a second sensor when collecting the historical environmental data. The first environmental data is generated at least partially based on data perceived by the first sensor, and the second environmental data is generated at least partially based on data perceived by the second sensor. The first environmental data includes multiple first data points and first object semantic information corresponding to each first data point. The second environmental data includes multiple second data points and second object semantic information corresponding to each second data point. At least some of the first data points in the first environmental data have perceptible information, which characterizes whether the first data point can be perceived by the second sensor. The perceptible information of the first data point is generated based on the first object semantic information and third object semantic information corresponding to the first data point. The third object semantic information is the second object semantic information corresponding to the target data point. The target data point is determined from multiple second data points in the second environmental data, corresponding to the first data point, based on the first sensor parameters of the first sensor and the second sensor parameters of the second sensor.

9. A computing device, characterized in that, The method includes a memory and a processor, the memory being used to store computer instructions, and the processor being used to retrieve the computer instructions from the memory to perform the method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images