Map generation method based on semantic information and related device

Abstract

The application discloses a kind of based on semantic information map generation method, device, vehicle and computer storage medium, comprising: by in the process of vehicle movement, the image data and the initial pose data of vehicle around environment can be constantly obtained, and the two-dimensional point cloud graph with semantic class information is obtained by carrying out semantic segmentation to image data;According to initial pose data, two-dimensional point cloud graph is converted to global coordinate system to obtain local map, based on the relative transformation pose between the local map of current time and historical time and whether the relative transformation pose is judged to constitute loop by relative transformation pose;The local map of current time that constitutes loop is revised, and the global map is obtained by splicing the local map of current time after revision and the local map of historical time.The semantic class information obtained by semantic segmentation and parking space number identification is fully utilized in the application, and the accuracy of the map generation method based on semantic information can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automatic driving, and particularly relates to a map generation method and device based on semantic information, a vehicle and a computer storage medium. BACKGROUND

[0002] The field of automatic driving is developing rapidly, and automatic parking technology, as an important branch of the field of automatic driving, is also attracting more and more attention. To realize complete automatic parking function, the surrounding environment needs to be perceived through perception technology to obtain environmental data, the environmental data needs to be used to realize the positioning of the vehicle itself in the environment and the construction of the environmental map by means of SLAM (Simultaneous Localization and Mapping) technology, and finally the path planning needs to be given according to the vehicle's own pose (position and attitude) and the environmental information to control the movement of the vehicle, so as to realize automatic parking.

[0003] In the prior art, the SLAM technical solution for automatic parking in the parking lot scene usually adopts semantic segmentation to process the environmental data collected by the vehicle-mounted visual sensor and the odometry to obtain point clouds, and uses the point cloud data to realize positioning and construction of the environmental map. However, when processing the obtained environmental data, it only segments out the pixels related to the parking line, the sign and the speed bump in the collected environmental data related images and further obtains point clouds through coordinate transformation to construct the environmental map. It can be seen that the extracted semantic information is too abstract to distinguish similar scenes, and the extracted semantic information is not fully utilized, which affects the accuracy of positioning and mapping. SUMMARY

[0004] The main purpose of the present application is to provide a map generation method and device based on semantic information, a vehicle and a computer storage medium, which aims to solve the problem that the extracted semantic information is not fully utilized when generating a map based on semantic information in the prior art, thereby affecting the accuracy of the map generation method based on semantic information.

[0005] In order to achieve the above object, the first aspect of the present application provides a map generation method based on semantic information, comprising: acquiring a plurality of first image data of an environment in which a vehicle body is located collected by a collection device at a current time and initial pose data of the vehicle body collected by a pose sensor, and converting the plurality of first image data from a coordinate system of the collection device to a coordinate system of the vehicle body to obtain a plurality of second image data according to internal and external parameters of the collection device; performing semantic segmentation and point cloud conversion on the plurality of second image data respectively to obtain a panoramic two-dimensional point cloud map of the environment in which the vehicle body is located at the current time; wherein the two-dimensional point cloud map comprises semantic category information; converting the panoramic two-dimensional point cloud map in the vehicle coordinate system to a global coordinate system by using the initial pose data at the current time to obtain a local map at the current time; calculating a relative transformation pose of the local map at the current time and a local map at a historical time according to the semantic category information in the local map at the current time, judging whether the local maps constitute a loop by using the relative transformation pose, and correcting the local map at the current time based on the judgment result to splice the local maps at each time to obtain a global map.

[0006] The second aspect of the present application provides a map generation device based on semantic information, comprising: an acquisition module configured to acquire a plurality of first image data of an environment in which a vehicle body is located collected by a collection device at a current time and initial pose data of the vehicle body collected by a pose sensor, and convert the plurality of first image data from a coordinate system of the collection device to a coordinate system of the vehicle body to obtain a plurality of second image data according to internal and external parameters of the collection device; a semantic segmentation module configured to perform semantic segmentation and point cloud conversion on the plurality of second image data respectively to obtain a panoramic two-dimensional point cloud map of the environment in which the vehicle body is located at the current time; wherein the two-dimensional point cloud map comprises semantic category information; a conversion module configured to convert the panoramic two-dimensional point cloud map in the vehicle coordinate system to a global coordinate system by using the initial pose data at the current time to obtain a local map at the current time; and a map generation module configured to calculate a relative transformation pose of the local map at the current time and a local map at a historical time according to the semantic category information in the local map at the current time, judge whether the local maps constitute a loop by using the relative transformation pose, and correct the local map at the current time based on the judgment result to splice the local maps at each time to obtain a global map.

[0007] The third aspect of the present application provides a vehicle, comprising a vehicle body, a collection device, a pose sensor and a processor, wherein: the collection device is configured to collect image data of an environment in which the vehicle body is located and transmit the image data to the processor; the pose sensor is configured to acquire initial pose data of the vehicle body and transmit the initial pose data to the processor; and the processor is configured to process the image data and the initial pose data according to the above-mentioned map generation method based on semantic information to obtain a global map of the environment in which the vehicle body is located.

[0008] The fourth aspect of the present application provides a computer storage medium, wherein the computer storage medium stores a semantic information based map generation program, and the semantic information based map generation program realizes the steps of the semantic information based map generation method when executed by a processor.

[0009] Compared with the prior art in which the semantic category information obtained after extracting the image data is only used as an identification mark, the present application can continuously acquire image data of the environment around the vehicle and initial pose data of the vehicle during the movement of the vehicle, and performs semantic segmentation on the image data to obtain a two-dimensional point cloud graph with semantic category information. The two-dimensional point cloud graph is converted to a global coordinate system according to the initial pose data to obtain a local map, the relative transformation pose between the local map obtained at the current time and the local map obtained at the historical time is calculated based on the semantic category information, and it is judged whether the two constitute a loop through the relative transformation pose. The local map at the current time which constitutes the loop is corrected, and the corrected local map at the current time and the local map at the historical time are spliced to obtain a global map. Thus, the present application makes full use of the extracted semantic information, and adds a label of a parking space number in the semantic information to distinguish similar scenes in a parking lot, thereby improving the accuracy of mapping and positioning, accurately positioning a parking space to be parked, distinguishing similar scenes, and accurately completing an automatic parking task. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0011] Figure 1 is an environmental schematic diagram of a parking lot where a vehicle is located according to the present application;

[0012] Figure 2 is a flowchart of a semantic information based map generation method provided by an embodiment of the present application;

[0013] Figure 3 is a specific flowchart of a pose data correction process of a local map at a current time according to a relative transformation pose provided by an embodiment of the present application;

[0014] Figure 4 is a structural schematic diagram of a semantic information based map generation device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0015] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, technologies, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0016] It should be understood that the term "comprising" when used in this specification and the appended claims, specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0017] The technical solutions in the embodiments of the present application are clearly and completely described below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0018] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application. However, the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the concept of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0019] In recent years, the field of autonomous driving has developed rapidly, and automatic parking technology, as an important branch of autonomous driving, has also received more and more attention. Among them, path planning and motion control are key technologies in automatic parking technology, both of which can be realized on the basis of SLAM. However, the existing SLAM technical solutions for automatic parking in parking lots extract too abstract semantic information, which has limited discrimination when similar scenes exist in parking lots, and the extracted semantic information is not utilized, thereby affecting the accuracy of positioning and mapping.

[0020] In order to solve the problems existing in the prior art, in the present embodiment, a map generation method and device based on semantic information, a vehicle and a computer storage medium are proposed, which distinguish similar scenes by recognizing parking space numbers as semantic information, and utilize the extracted semantic information for semantic ICP matching to realize vehicle positioning and construction of an environment map, thereby improving the accuracy of positioning and mapping.

[0021] Figure 1Fig. 1 is a schematic diagram of an environment in which a vehicle is located according to an embodiment of the present application. In one embodiment, the vehicle comprises a vehicle body 10, a collection device 11, a pose sensor 12 and a processor (not shown), wherein the collection device 11 is configured to collect image data of an environment in which the vehicle body is located and transmit the image data to the processor; the pose sensor 12 is configured to obtain initial pose data of the vehicle body and transmit the initial pose data to the processor; and the processor is configured to process the image data and the initial pose data of the vehicle body according to a map generation method provided by one or more embodiments of the present application to obtain a global map of the environment in which the vehicle body 10 is located. Further, the processor can provide path planning according to the environment information of the vehicle to control the movement of the vehicle body 10, thereby achieving automatic parking.

[0022] In some embodiments, the collection device 11 comprises at least one fisheye camera configured to collect image data of the surrounding environment of the vehicle. When the collection device 11 comprises at least one fisheye camera, the fisheye camera can be rotatably connected to the vehicle body 10, and the fisheye camera can collect multiple frames of image data of different local scenes in the surrounding environment by rotating and transmit the multiple frames of image data to the processor. When the collection device 11 comprises multiple fisheye cameras, the fisheye cameras can be arranged at four directions (front, back, left and right) of the vehicle body 10, and the multiple fisheye cameras can collect multiple frames of image data of the environment in which the vehicle body 10 is located without rotating and transmit the multiple frames of image data to the processor. It should be noted that the collection device 11 can further comprise a depth camera, a laser radar or a three-dimensional scanner, etc. for assisting in obtaining environment data, which is not limited herein.

[0023] In one embodiment, the pose sensor 12 comprises an inertial measurement unit (IMU) configured to obtain acceleration and angular velocity of the vehicle body 10 and transmit the acceleration and angular velocity to the processor, and the processor is configured to integrate the acceleration and angular velocity to obtain the initial pose data of the vehicle body 10. In another embodiment, the pose sensor 12 comprises a wheel odometry configured to directly obtain the initial pose data of the vehicle body 10. In yet another embodiment, the pose sensor 12 comprises an IMU and a wheel odometry, and the initial pose data of the vehicle body 10 can be obtained by fusing the pose data obtained by the IMU and the pose data obtained by the wheel odometry. It should be noted that the pose sensor 12 can further comprise a GPS, WiFi or other sensors configured to obtain the initial pose data of the vehicle body, and the initial pose data of the vehicle body 10 can be obtained by fusing the pose data obtained by the above-mentioned sensors, which is not limited herein.

[0024] In an embodiment, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), neural network chips, or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0025] In an embodiment, the vehicle further comprises a memory for storing computer programs, such as a computer program for generating a map based on semantic information, for execution by the processor. The memory can be an internal storage unit, such as a hard disk or a memory, and can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory can include both an internal storage unit and an external storage device, and can also be used to temporarily store data that has been output or is to be output. It should be noted that when the processor is a neural network chip, the vehicle can not include a memory, and whether the vehicle needs to use the memory to store the corresponding computer program depends on the type of the processor.

[0026] Exemplary method

[0027] Figure 2 To provide a map generation method based on semantic information according to an embodiment of the present application, the method comprises the following steps:

[0028] In step S100, a plurality of first image data of an environment in which a vehicle body is located and initial pose data of the vehicle body collected by a pose sensor at a current time are acquired, and the plurality of first image data is converted from a coordinate system of a collection device to a coordinate system of the vehicle body to obtain a plurality of second image data according to internal and external parameters of the collection device.

[0029] Specifically, pixels on each first image data are inversely projected to a camera coordinate system of the collection device according to a camera model using the internal parameters of the collection device, and then converted to a ground surface in the vehicle coordinate system using the external parameters of the collection device to obtain each second image data.

[0030] In step S200, the multi-frame second image data is respectively subjected to semantic segmentation and point cloud conversion to obtain a panoramic two-dimensional point cloud map of the environment where the vehicle body is located at the current time; the two-dimensional point cloud map comprises semantic category information, and the semantic category information comprises category information such as parking spaces, parking space numbers, lane lines and road markings, which can distinguish the parking lot scene.

[0031] In one embodiment, before the multi-frame second image data is subjected to semantic segmentation, the multi-frame second image data can be spliced to obtain a panoramic image of the environment where the vehicle body is located, and then the panoramic image is subjected to semantic segmentation and point cloud conversion to obtain a panoramic two-dimensional point cloud map of the environment where the vehicle body is located at the current time; the panoramic two-dimensional point cloud map is accompanied by semantic category information.

[0032] In some other embodiments, the multi-frame second image data can also be respectively subjected to semantic segmentation and point cloud conversion to respectively obtain multi-frame two-dimensional point cloud maps accompanied by semantic category information, and the multi-frame two-dimensional point cloud maps are spliced to obtain a panoramic two-dimensional point cloud map of the environment where the vehicle body is located at the current time; the two-dimensional point cloud map is accompanied by semantic category information.

[0033] In one embodiment, splicing the multi-frame second image data or the multi-frame two-dimensional point cloud map to obtain a panoramic two-dimensional point cloud map at the current time comprises: splicing the multi-frame second image data or the multi-frame two-dimensional point cloud map according to the angle of rotation of the collection device or the relative extrinsic parameters between the cameras in the collection device to obtain a panoramic two-dimensional point cloud map at the current time. Specifically, when the collection device needs to collect multi-frame first image data of the environment where the vehicle body is located by rotating, the pose sensor needs to record the angle of rotation of the collection device at the current time when the collection device collects each frame of image, and the multi-frame second image data or the multi-frame two-dimensional point cloud map can be spliced according to the angle of rotation recorded by the pose sensor to obtain a global two-dimensional point cloud map; when the collection device does not need to rotate to collect multi-frame first data of the environment where the vehicle body is located, the relative extrinsic parameters between the multiple cameras in the collection device can be used to splice the multi-frame second image data or the multi-frame two-dimensional point cloud map to obtain a panoramic two-dimensional point cloud map at the current time.

[0034] In an embodiment, the semantic segmentation of the multiple frames of second image data or the panoramic image respectively comprises: performing semantic segmentation on the second image data or the panoramic image by a deep neural network such as a semantic segmentation network (U-Net), a fully convolutional network (FCN), SegNet, PSPNet, and DeepLab to obtain semantic category information such as parking spaces, lane lines, and road signs; performing parking space number semantic recognition on the second image data or the panoramic image by a convolutional neural network (CNN) to recognize semantic category information about parking space numbers and associate the parking spaces in the semantic category information with the parking space numbers one by one, thereby distinguishing similar scenes in the parking lot according to the association information of the parking spaces and the parking space numbers. It should be noted that some parking lots are not marked with parking spaces by pure numbers, but may have some other characters such as letters, Chinese characters, symbols, numbers, and combinations of these characters. When performing parking space number recognition on the second image data, other characters and character combinations can also be recognized.

[0035] In step S300, the panoramic two-dimensional point cloud map in the vehicle coordinate system is converted to the global coordinate system by using the initial pose data at the current time to obtain a local map at the current time including semantic category information. The initial pose data is obtained by fusing the pose data obtained by the inertial measurement unit and the pose data obtained by the wheel odometry.

[0036] It should be noted that, in order to ensure the accuracy of mapping, the acquisition device in this embodiment can continuously acquire multiple frames of first image data of the same local scene at the current time to obtain a local map. That is, for different local scenes, the acquisition device correspondingly acquires multiple frames of first image data, and the accuracy of the acquired image data is improved by fusing the multiple frames of first image data of the current local scene, thereby ensuring the accuracy of mapping.

[0037] In step S400, the relative transformation pose of the local map at the current time and the local map at the historical time is calculated according to the semantic category information in the local map at the current time, it is judged whether the local maps constitute a loop by using the relative transformation pose, and the local map at the current time is corrected based on the judgment result, and the local maps at different times are spliced to obtain a global map.

[0038] In an embodiment, if a loop is constituted, the local map at the current time is corrected in pose by using the relative transformation pose, and the global map is obtained by splicing the corrected local map at the current time and the local map at the historical time. If a loop is not constituted, the initial pose does not need to be corrected, and the global map is obtained by directly splicing the local map at the current time and the local map at the historical time. It should be noted that the historical time includes all times corresponding to the processed local map or the previous time corresponding to the current time.

[0039] Specifically, a pre-defined semantic ICP algorithm is used to solve the relative transformation poses between local maps. The matching score between the local map obtained at the current time and the local maps obtained at previous times is calculated based on the relative transformation poses. The matching score is then used to determine whether the local map obtained at the current time forms a loop with the local map obtained at previous times. If the matching score is less than the pre-defined matching score, it indicates that the local map obtained at the current time forms a loop with the local map obtained at previous times. In this case, the local map at the current time needs to be corrected, and then the corrected local map is stitched together with the local map obtained at previous times to obtain the global map. If the matching score of the relative transformation pose is greater than or equal to the pre-defined matching score, it indicates that the local map obtained at the current time does not form a loop with the local map obtained at previous times. In this case, there is no need to correct the initial pose; the local map at the current time is directly stitched together with the local map obtained at previous times to obtain the global map.

[0040] In one embodiment, a preset semantic ICP algorithm is used to solve the relative transformation poses between local maps. The matching score between local maps is calculated based on the relative transformation poses. If the matching score is less than a preset matching score, a loop is formed, and the local map at the current moment needs to be corrected. Then, the corrected local map is stitched together with the local maps from previous moments to accumulate and stitch together the local maps from each moment to obtain the global map. More specifically, this includes... Figure 3 The steps shown are as follows:

[0041] Step S301: Obtain the initial relative transformation pose T of the point cloud of the local map and the target point cloud at the current moment; wherein, the target point cloud is a stored map point cloud, which is obtained by stitching together the local map at a historical moment; the initial relative transformation pose T can be obtained by fusing the pose data of the current moment and the historical moment recorded by the IMU and the wheel odometry.

[0042] Step S302: Traverse all semantic category information in the point cloud of the local map at the current moment, and determine whether the number of points corresponding to each category information in all semantic category information is greater than the preset threshold and whether they exist in the target point cloud.

[0043] Step S303: If the number of points corresponding to any category of information is greater than a preset threshold and exists in the target point cloud, then the coordinate transformation of the point cloud of that category in the local map at the current moment is performed using the initial relative transformation pose to obtain the source point cloud at the current moment in the same coordinate system as the target point cloud.

[0044] Specifically, for the corresponding various category information in the point cloud of the local map at the current moment, such as parking space, lane line, road sign, and parking space number, if the number of points of any category information in the point cloud of the local map at the current moment is greater than the preset number of points, and the target point cloud contains the category information, the initial relative transformation pose is used to perform coordinate conversion on the point cloud of the category in the local map at the current moment, to obtain the number of points of any category information in the current source point cloud in the same coordinate system of the target point cloud being less than or equal to the preset number of points, and then other category semantic category information in the point cloud of the current local map is reselected; or if the target point cloud does not contain the same semantic category information, other category semantic category information in the point cloud of the current local map is reselected.

[0045] In step S304, the current relative transformation pose T between the current source point cloud and the target point cloud is calculated according to the category information. * The current relative transformation pose is compared with the initial relative transformation pose to calculate a matching score of the source point cloud and the target point cloud, and whether the point clouds constitute a loop is determined based on the matching score.

[0046] Specifically, assuming that the current source point cloud contains a plurality of source points (denoted as a source point set s), for any source point k, nearest neighbor search is performed from the points of the same semantic category information in the target point cloud according to the semantic category information corresponding to the source point, to obtain N matching points; a target matching point (denoted as a target matching point set t) with a distance less than a preset distance from the source point is searched from the N matching points, and a loss function of the target matching point and the source point to be matched is calculated to solve the current relative transformation pose T * .

[0047] In one embodiment, searching for a target matching point with a distance less than a preset distance from the source point from the t matching points includes: respectively calculating the Euclidean distances of the t matching points and the source point, if the Euclidean distance corresponding to the current matching point is less than the preset distance, the current matching point is regarded as the target matching point, and the target matching point and the corresponding source point are regarded as a pair of matching points.

[0048] In one embodiment, the loss function of the target matching point and the source point is calculated to solve the current relative transformation pose, including: calculating the loss function of the target matching point and the source point and minimizing the loss function, and inversely solving the current relative transformation pose T * from the minimized loss function. Specifically, the loss function of a pair of matching points is represented as wherein p α (*) represents a Cauchy kernel function, represents a weight, N represents the number of matching points, and ||*| 2 represents an L2 norm, represents the coordinates of the kth point in the target matching point set t, This indicates that the source point is transformed to the coordinate system of the target matching point using an initial relative transformation pose, where R represents the rotation matrix and p represents the translation matrix. This represents the coordinates of the k-th point in the source point set s. Furthermore, the current relative transformation pose T is solved in reverse by minimizing the loss function. * This includes: traversing all matching point pairs, calculating the minimum value of the loss function, and inversely solving for the current relative transformation pose T. * Specifically, it is expressed as:

[0049]

[0050] Wherein, the SE(3) manifold represents a special Euclidean group, and n represents n pairs of matching points.

[0051] Furthermore, compare the current relative transformation pose T * If the difference between the current relative transformation pose T and the initial relative transformation pose T on the SE(3) manifold is less than a preset difference, then the result converges. If it is less than the preset difference, then the current relative transformation pose T is used. * Calculate the matching score between the source point cloud and the target point cloud. If the score is less than the preset matching score, it indicates that the source point cloud and the target point cloud form a loop at the current time. Otherwise, they do not form a loop.

[0052] If the current relative transformation pose T * If the difference between the current relative transformation pose T and the initial relative transformation pose T on the SE(3) manifold is greater than or equal to a preset difference, then the current relative transformation pose T will be changed. * Assign an initial relative transformation pose, and repeat steps S301-S304 to adjust the current relative transformation pose T. * Iterate through the calculations until the results converge.

[0053] In step S305, if a loop is formed, the pose of the source point cloud at the current time is corrected using the current relative transformation pose, and then the corrected source point cloud is stitched together with the target point cloud to obtain a global map by accumulating and stitching.

[0054] In step S306, if no loop is formed, there is no need to correct the pose; the source point cloud and the target point cloud are directly accumulated and stitched together to obtain the global map.

[0055] In one embodiment, after obtaining the global map of the parking lot, when a vehicle re-enters the same parking lot, the point cloud of the local map obtained at the current moment is matched with the point cloud of the global map using a preset semantic ICP algorithm, so that the vehicle's corresponding pose in the parking lot can be accurately located.

[0056] In some embodiments, since the initial pose data used in the embodiments is obtained by fusing the pose data acquired by the IMU and the wheel odometry, after the local map at the current time is corrected according to the relative transformation pose, the local map after the correction can also be fused with the pose data obtained by other sensors (such as GPS, WIFI, etc.) through extended Kalman filtering or weighted averaging, so as to obtain a more accurate global map.

[0057] Exemplary device

[0058] Figure 4 To provide a map generation device based on semantic information according to the embodiments of the present application, the device comprises: an acquisition module 401 configured to acquire a plurality of frames of first image data collected by a collection device at a current time and initial pose data of a vehicle body collected by a pose sensor, and convert the plurality of frames of first image data from a coordinate system of the collection device to a coordinate system of the vehicle body to obtain a plurality of frames of second image data according to internal and external parameters of the collection device; a semantic segmentation module 402 configured to perform semantic segmentation and point cloud conversion on the plurality of frames of second image data respectively, and obtain a panoramic two-dimensional point cloud map of an environment in which the vehicle body is located at the current time; wherein the two-dimensional point cloud map comprises semantic category information, and the semantic category information comprises category information such as parking spaces, parking space numbers, lane lines and road markings, which can distinguish the parking lot scene; a conversion module 403 configured to convert the panoramic two-dimensional point cloud map in the vehicle coordinate system to a global coordinate system using the initial pose data at the current time, and obtain a local map at the current time comprising semantic category information; wherein the initial pose data is obtained by fusing the pose data acquired by an inertial measurement unit and the pose data acquired by a wheel odometry; and a map generation module 404 configured to calculate a relative transformation pose of the local map at the current time and a local map at a historical time according to the semantic category information in the local map at the current time, determine whether the local maps constitute a loop using the relative transformation pose, and correct the local map at the current time based on the determination result, and splice the local maps at different times to obtain a global map.

[0059] It should be noted that the specific functions of the map generation device based on semantic information and each module thereof can also refer to the corresponding description in the above map generation method based on semantic information, and will not be described here.

[0060] The embodiments of the present application also provide a computer storage medium, and the computer storage medium stores a map generation program based on semantic information, and the map generation program based on semantic information implements the steps of any one of the map generation methods based on semantic information provided by the embodiments of the present application when executed by a processor.

[0061] It should be understood that the sequence of the steps in the above-described embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0062] Those skilled in the art can appreciate that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0063] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other ways. For example, the above-described apparatus / terminal device embodiments are merely schematic, for example, the division of the above-described modules or units is merely a logical function division, and an actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0064] The integrated modules / units described above, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the computer program can instruct related hardware to complete all or part of the processes in the above-described embodiments, and the computer program can be stored in a computer readable storage medium. The computer program can be executed by a processor to implement the steps of each method embodiment. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer readable storage medium contains content which can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0065] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that; it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements are not the essence of the corresponding technical solutions deviating from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A map generation method based on semantic information, characterized by, Applied to vehicles, the vehicle includes a vehicle body, a data acquisition device, a pose sensor, and a processor, wherein the map generation method is executed by the processor, and the method includes: The system acquires multiple frames of first image data of the environment in which the vehicle body is located, which are acquired by the acquisition device at the current moment, and the initial pose data of the vehicle body acquired by the pose sensor. Based on the intrinsic and extrinsic parameters of the acquisition device, the system transforms the multiple frames of first image data from the coordinate system of the acquisition device to the coordinate system of the vehicle body to obtain multiple frames of second image data. Semantic segmentation and point cloud transformation are performed on the multiple frames of second image data respectively to obtain a panoramic two-dimensional point cloud map of the environment in which the vehicle body is located at the current moment; wherein, the panoramic two-dimensional point cloud map includes semantic category information, which includes parking space, parking space number, lane line and road markings; Using the initial pose data at the current moment, the panoramic two-dimensional point cloud map in the vehicle coordinate system is transformed to the global coordinate system to obtain the local map at the current moment; The relative transformation pose of the current local map and the local maps at historical times is calculated based on the semantic category information in the current local map. The relative transformation pose is used to determine whether loops exist between the local maps. Based on the determination result, the current local map is corrected. The global map is obtained by stitching together the local maps from each time point, including: Obtain the initial relative transformation pose of the point cloud of the local map and the target point cloud at the current moment; wherein, the target point cloud is the stored map point cloud; Traverse all semantic category information in the point cloud of the local map at the current moment, and determine whether the number of points corresponding to each category information in all semantic category information is greater than a preset threshold and whether they exist in the target point cloud; If the number of points corresponding to any category of information is greater than a preset threshold and exists in the target point cloud, then the coordinate transformation of the point cloud of that category in the local map at the current moment is performed using the initial relative transformation pose to obtain the source point cloud at the current moment in the same coordinate system as the target point cloud. The current relative transformation pose between the source point cloud and the target point cloud at the current moment is calculated based on the category information. The difference between the current relative transformation pose and the initial relative transformation pose on a special Euclidean group manifold is compared to see if it is less than a preset difference. If it is less than the preset difference, the matching score between the source point cloud and the target point cloud is calculated based on the current relative transformation pose, and the matching score is used to determine whether a loop is formed between the point clouds. If a loop is formed, the current relative transformation pose is used to correct the pose of the source point cloud at the current moment, and then the corrected source point cloud is stitched together with the target point cloud to obtain a global map; if no loop is formed, the source point cloud and the target point cloud are stitched together to obtain a global map.

2. The map generation method based on semantic information according to claim 1, characterized in that, The step of performing semantic segmentation and point cloud transformation on the multiple frames of second image data to obtain a panoramic two-dimensional point cloud map of the environment in which the vehicle body is located at the current moment includes: The multiple frames of second image data are stitched together to obtain a panoramic view of the environment in which the vehicle body is located. The panoramic image is semantically segmented and converted into point cloud data to obtain a panoramic two-dimensional point cloud image of the environment in which the vehicle is located at the current moment.

3. The map generation method based on semantic information according to claim 1, characterized in that, The step of performing semantic segmentation and point cloud transformation on the multiple frames of second image data to obtain a panoramic two-dimensional point cloud map of the environment in which the vehicle body is located at the current moment includes: Semantic segmentation and point cloud transformation are performed on the multiple frames of second image data respectively to obtain multiple frames of two-dimensional point cloud maps with semantic category information; By stitching together the multiple frames of two-dimensional point cloud images, a panoramic two-dimensional point cloud image of the environment in which the vehicle body is located at the current moment is obtained.

4. The map generation method based on semantic information according to claim 2 or 3, characterized in that, The semantic segmentation of the panoramic image or the semantic segmentation of the multi-frame second image data includes: Semantic segmentation of the second image data or panoramic image is performed using a deep neural network to obtain semantic category information including parking spaces, lane lines, and road markings; Based on the convolutional neural network, the parking space number in the panoramic image or the multi-frame second image data is identified, and semantic category information about the parking space number is obtained; The parking spaces in the semantic category information are associated with the parking space numbers one by one in order to distinguish similar scenarios.

5. The map generation method based on semantic information according to claim 1, characterized in that, The step of calculating the relative transformation pose of the local map at the current time and the local map at a historical time based on the semantic category information in the local map at the current time, and using the relative transformation pose to determine whether a loop is formed between the local maps, includes: The relative transformation poses of the local map at the current time and the local map at a historical time are solved using a preset semantic ICP algorithm. The matching score between the local maps is calculated using the relative transformation poses, and it is determined whether the matching score is less than a preset matching score to determine whether the local map at the current time and the local map at the historical time form a loop.

6. The map generation method based on semantic information according to claim 5, characterized in that, The step of determining whether the matching score is less than a preset matching score to determine whether the local map at the current time and the local map at the historical time form a loop includes: If the matching score is less than the preset matching score, the local map at the current moment and the local map at the historical moment form a loop. In this case, the local map at the current moment needs to be corrected, and then the corrected local map is stitched together with the local map at the historical moment to obtain the global map. If the matching score of the relative transformation pose is greater than or equal to the preset matching score, then the local maps do not form a loop, and the global map is obtained by stitching the local map at the current moment with the local map at the historical moment.

7. The map generation method based on semantic information according to claim 1, characterized in that, After obtaining the global map, the map generation method further includes: By using a preset semantic ICP algorithm, the point cloud of the local map obtained at the current moment is matched with the point cloud of the global map to locate the corresponding pose of the vehicle in the parking lot.

8. A map generation device based on semantic information, characterized in that, Applied to vehicles, the vehicle includes a vehicle body, a data acquisition device, a pose sensor, and a processor; the map generation device includes: The acquisition module is used to acquire multiple frames of first image data of the environment in which the vehicle body is located at the current moment, which are acquired by the acquisition device and the initial pose data of the vehicle body acquired by the pose sensor, and to transform the multiple frames of first image data from the coordinate system of the acquisition device to the coordinate system of the vehicle body according to the intrinsic and extrinsic parameters of the acquisition device to obtain multiple frames of second image data. The semantic segmentation module is used to perform semantic segmentation and point cloud conversion on the multiple frames of second image data respectively to obtain a panoramic two-dimensional point cloud map of the environment in which the vehicle body is located at the current moment; wherein, the panoramic two-dimensional point cloud map includes semantic category information, which includes parking space, parking space number, lane line and road markings. The conversion module is used to convert the panoramic two-dimensional point cloud map in the vehicle coordinate system to the global coordinate system using the initial pose data at the current moment, so as to obtain the local map at the current moment. The map generation module is used to calculate the relative transformation pose of the local map at the current time and the local map at the historical time based on the semantic category information in the local map at the current time, use the relative transformation pose to determine whether there is a loop between the local maps, and correct the local map at the current time based on the judgment result, and stitch the local maps at each time to obtain the global map. The map generation module is specifically used to: obtain the initial relative transformation pose of the point cloud of the local map at the current moment and the target point cloud; wherein, the target point cloud is a stored map point cloud; Traverse all semantic category information in the point cloud of the local map at the current moment, and determine whether the number of points corresponding to each category information in all semantic category information is greater than a preset threshold and whether they exist in the target point cloud; If the number of points corresponding to any category of information is greater than a preset threshold and exists in the target point cloud, then the coordinate transformation of the point cloud of that category in the local map at the current moment is performed using the initial relative transformation pose to obtain the source point cloud at the current moment in the same coordinate system as the target point cloud. The current relative transformation pose between the source point cloud and the target point cloud at the current moment is calculated based on the category information. The difference between the current relative transformation pose and the initial relative transformation pose on a special Euclidean group manifold is compared to see if it is less than a preset difference. If it is less than the preset difference, the matching score between the source point cloud and the target point cloud is calculated based on the current relative transformation pose, and the matching score is used to determine whether a loop is formed between the point clouds. If a loop is formed, the current relative transformation pose is used to correct the pose of the source point cloud at the current moment, and then the corrected source point cloud is stitched together with the target point cloud to obtain a global map; if no loop is formed, the source point cloud and the target point cloud are stitched together to obtain a global map.

9. A vehicle, characterized in that, The vehicle includes a vehicle body, data acquisition equipment, pose sensors, and a processor, wherein: The acquisition device is used to acquire image data of the environment in which the vehicle body is located and transmit it to the processor; The pose sensor is used to acquire the initial pose data of the vehicle body and transmit it to the processor; The processor is configured to process the image data and the initial pose data according to the semantic information-based map generation method as described in any one of claims 1-7 to obtain a global map of the environment in which the vehicle body is located.

10. A computer storage medium, characterized in that, The computer storage medium stores a map generation program based on semantic information, which, when executed by a processor, implements the steps of the map generation method based on semantic information as described in any one of claims 1-7.

Images