Road element vectorization method, device and equipment

Abstract

The embodiment of the invention provides a road element vectorization method, device and equipment. The method comprises the steps of obtaining a tracking sequence of tracked target road elements, a track pose corresponding to each frame of image in the tracking sequence and an optimization initial value of the target road elements; aiming at the tracking sequence, according to the trajectory pose corresponding to each frame of image, the optimization initial value of the target road element and the pixel coordinate of each frame of image, performing optimization through a constructed residual error model, and determining an optimization result corresponding to a target object of which a residual error model result meets a preset condition in an iteration process; and constructing a vectorization result of the target road element according to an optimization result of the target object. According to the method provided by the embodiment of the invention, the road element vectorization precision can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of map, and particularly relate to a road element vectorization method, device and equipment. BACKGROUND

[0002] In the field of high-precision map, road elements (such as traffic signs, rod-shaped elements, etc.) are very important map elements, and traffic signs in road traffic elements are one of the most important map elements. In mapping, vectorization of traffic signs is an important part of mapping.

[0003] Currently, three-dimensional measurement technology is generally used to measure multiple corner points or multiple feature points of traffic signs, and finally determine the vectorization plane of traffic signs. However, this vectorization method has a great impact on the final result due to different position accuracy of each point vectorization, and it is difficult to obtain a high-precision vectorization result.

[0004] Therefore, the road element vectorization result of the prior art has low accuracy. SUMMARY

[0005] Embodiments of the present application provide a road element vectorization method, device and equipment to overcome the problem of low accuracy of road element vectorization result of the prior art.

[0006] In a first aspect, embodiments of the present application provide a road element vectorization method, comprising:

[0007] obtaining a tracking sequence of a target road element tracked, a trajectory pose corresponding to each frame of image in the tracking sequence, and an optimization initial value of the target road element;

[0008] For the tracking sequence, according to the trajectory pose corresponding to each frame of image, the optimization initial value of the target road element and the pixel coordinates of each frame of image, an optimization result corresponding to a target object in which a residual model result in an iteration process meets a preset condition is determined by optimization through a constructed residual model.

[0009] According to the optimization result of the target object, a vectorization result of the target road element is constructed.

[0010] In a second aspect, embodiments of the present application provide a map updating method, comprising:

[0011] Based on the method of any one of the first aspect, a vectorization result of the target road element is determined;

[0012] According to the vectorization result of the target road element, the map is updated.

[0013] In a third aspect, embodiments of the present application provide a road element vectorization device, comprising:

[0014] an acquisition module configured to acquire a tracking sequence of a tracked target road element, a track pose corresponding to each frame of image in the tracking sequence, and an optimization initial value of the target road element;

[0015] an optimization module configured to optimize, for the tracking sequence, according to the track pose corresponding to each frame of image, the optimization initial value of the target road element, and pixel coordinates of each frame of image, through a constructed residual model, to determine an optimization result of a target object corresponding to a result of the residual model meeting a preset condition in an iteration process;

[0016] a vectorization module configured to construct a vectorization result of the target road element according to the optimization result of the target object.

[0017] In a fourth aspect, an embodiment of the present application provides a map updating apparatus, comprising:

[0018] a processing module configured to determine a vectorization result of the target road element based on the method in any one of the first aspect;

[0019] a map updating module configured to update the map according to the vectorization result of the target road element.

[0020] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising at least one processor and a memory.

[0021] The memory stores computer-executable instructions.

[0022] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method in any one of the above aspects and various possible designs.

[0023] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, when a processor executes the computer-executable instructions, the method in any one of the above aspects and various possible designs is implemented.

[0024] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising a computer program, when a processor executes the computer program, the method in any one of the above aspects and various possible designs is implemented.

[0025] The road element vectorization method, apparatus, and device provided in this embodiment first acquire the tracking sequence of the tracked target road elements, the trajectory pose corresponding to each frame of the tracking sequence, and the initial optimization value of the target road elements. Further, for the tracking sequence, based on the trajectory pose corresponding to each frame of the image, the initial optimization value of the target road elements, and the pixel coordinates of each frame, optimization is performed using a constructed residual model. The optimization result corresponding to the target object whose residual model result meets preset conditions during the iteration process is determined. Further, based on the optimization result of the target object, the vectorization result of the target road elements is constructed. This application, based on target tracking of target road elements, obtains a complete tracking sequence frame corresponding to each target road element. Based on the tracking result, multi-frame image optimization is performed on the tracking sequence, continuously updating the initial optimization value to obtain the final optimization result. Through multi-frame image optimization, the impact of individual frame recognition deviations and pose anomalies on the vectorization position accuracy is reduced, and high-precision vectorization results are obtained through multi-frame adjustment. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A schematic flowchart illustrating the road element vectorization method provided in this application embodiment;

[0028] Figure 2 A flowchart illustrating a road element vectorization method provided in another embodiment of this application;

[0029] Figure 3 A flowchart illustrating a road element vectorization method provided in another embodiment of this application;

[0030] Figure 4 This is a schematic diagram of the mapping results provided in the embodiments of this application;

[0031] Figure 5 This is a schematic diagram of the road element vectorization device provided in the embodiments of this application;

[0032] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0034] The terms "first", "second", "third", "fourth" and the like (if any) in the description, claims, and drawings of the present application, and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices. In view of the problems of the prior art, the technical concept of the present application is to adopt graph optimization, which only needs to input a three-dimensional position of a traffic sign corner point. This point can be selected as a point with minimum error (herein referred to as a target object). The point itself can reduce the vectorization error. In addition, at least one parameter such as position is optimized based on the three-dimensional position point, which can effectively correct the influence of errors and pose errors, so that the vectorization result reaches a high precision level, effectively solving the problem of low precision of the vectorization result.

[0035] The road elements include traffic signs, rod-shaped elements, etc. The following will take traffic signs as an example to describe the road element vectorization method in detail. In actual application, referring to FIG. 1, Figure 1 Figure 1 The flowchart of the road element vectorization method provided by the embodiments of the present application is shown. The road element vectorization method can be applied to monocular camera vectorization. According to the input camera internal and external parameters, trajectory pose and sign corner point, target tracking of the traffic sign can be completed, and the complete tracking sequence frame corresponding to each traffic sign can be obtained. According to the tracking result, the three-dimensional measurement of the traffic sign can be realized by selecting the front and back two frames of results with a certain baseline. After obtaining the three-dimensional measurement result, multi-frame graph optimization is performed on the tracking sequence to obtain the final single-point three-dimensional position, sign heading and width and height (i.e. scale). This method reduces or eliminates the influence of individual frame recognition deviation and pose anomaly on the position precision of vectorization, and high-precision vectorization result is obtained through multi-frame correction.

[0036] Specifically, the input includes:​

[0037] The input conditions required by the embodiments of the application include camera intrinsic parameters, camera extrinsic parameters, recognized corner points (here, referring to objects, such as traffic sign corner points), track poses, tracking sequences, and optimization initial values.

[0038] The camera intrinsic parameters determine the mapping relationship inside the sensor, and include focal length, image principal point, and distortion coefficients, which are denoted by K.

[0039] The camera extrinsic parameters reflect the conversion relationship between the sensor and an external coordinate system, and can be represented by camera installation parameters, including the height h of the camera from the ground and the installation angles Pitch (pitch angle), Roll (roll angle), and Yaw (yaw angle or side angle) relative to the carrier coordinate system, and the camera system coordinate can be converted to the carrier coordinate system, which is denoted by Rc2b.

[0040] The traffic sign corner points, i.e., the image corner points of the traffic sign, are denoted by {C k}, and the number of corner points is different for different shapes of traffic signs.

[0041] The track poses include the shooting position T (i.e., the position information when actually shooting) and the attitude R information of each image, which are denoted by {P i}, where i represents a pose of a certain frame, and the value range is marked as 0-n.

[0042] The tracking sequences, i.e., the complete sequences of each traffic sign corresponding to the traffic sign target tracking, are denoted by {Q i}, where i represents a frame, and the value range is marked as 0-n, and corresponds to the sequence in the track. Here, the tracking refers to the process of identifying and tracking the target, collecting target information, and recording target related information, etc.

[0043] The optimization initial values, i.e., the initial information of the optimization variables of the graph optimization, include the three-dimensional position information (denoted by p) of a single point of the traffic sign, the heading (denoted by θ), the width (denoted by w), and the height (denoted by h), and the complete three-dimensional vector information of the traffic sign can be restored by the four quantities.

[0044] The single point three-dimensional position information p is obtained according to the tracking sequence {Q i},select certain baseline fixed two frame Qs, Qj as a measuring frame, one of the pair of the smaller theoretical error corner as the same name point, the corner is generally selected away from the track point, the selected fixed position corner (here refers to the selected theoretical error smaller corner) is marked as C0, the three-dimensional measurement results corresponding to the pair of the same name corner (i.e. the same name point) can be calculated by triangulation method or two-dimensional plane intersection to restore height, the result is the carrier system (such as vehicle body, etc.) position information relative to one of the frame results, which is called reference frame, represented by Q0.

[0045] Traffic sign heading: the included angle between the normal of the front surface of the sign and the opposite direction of the track running direction, the value range is [-90°, 90°], in order to avoid the optimizer falling into local optimum, resulting in incorrect heading calculation, the embodiment of the application adopts the way of setting initial value in the value range, and sets different initial values of heading.

[0046] Traffic sign width w and height h: a traffic sign plane can be determined according to the single point three-dimensional measurement result and the heading information, and the width and height initial values can be roughly calculated by projecting the plane through other corners.

[0047] Figure optimization (combined with Figure 2 as shown, Figure 2 The flowchart of the road element vectorization method provided by another embodiment of the application, Figure 2 shows the residual calculation method of any frame participating in optimization):

[0048] Based on the tracking sequence, the optimization initial value and the track pose, the single point three-dimensional result (i.e. single point three-dimensional position information) relative to the frame to be optimized (here refers to each frame in the tracking sequence) is calculated, based on the calculated single point three-dimensional result relative to the frame to be optimized, based on the optimization initial value, the three-dimensional position of the traffic card (i.e. traffic sign or sign) is restored, and through the camera external parameter and the camera internal parameter, the three-dimensional corner mapping image is carried out, and based on the sign corner, the residual is calculated.

[0049] Output:

[0050] After the figure optimization, the better optimization result is obtained, according to the optimization result: the single point three-dimensional position of the traffic sign, the heading, the width and the height, the vectorization result of the traffic sign with different shapes can be constructed. The output result is currently the carrier system coordinates relative to the reference frame, which can be directly converted into the world coordinate system such as ECEF (i.e. geocentric and fixed rectangular coordinate system) through the track pose P0.

[0051] Taking the rod-shaped element as an example, the track pose is the same as the sign element, that is, the track pose: contains the shooting position T (i.e. the position information when actually shooting) and the attitude R information corresponding to each image, etc., represented by {P i} indicates that i represents the pose of a certain frame, the value range is marked as 0~n.

[0052] Tracking sequence: that is, the rod-shaped element target tracking has been completed, and the complete sequence corresponding to each rod-shaped element is obtained, and the tracking sequence is represented by {Q i}, wherein i represents a certain frame, and the value range is marked as 0-n. It should be noted that there may be a situation that, for example, the rod part of the rod-shaped element appears in 10 frames, and the part such as the sign or traffic light above the rod part appears in 20 frames. The tracking sequence here marks the complete sequence of the rod-shaped element.

[0053] Optimization initial value: that is, the initial information of the optimization variable of the graph optimization. The optimization variable related to the rod-shaped element includes the three-dimensional position information of the corner point.

[0054] Therefore, the embodiment of the present application breaks the traditional three-dimensional measurement method of adopting two or more frames of image information for vectorization of traffic signs, and increases graph optimization on the basis of measurement, thereby forming a brand-new way of traffic sign vectorization based on graph optimization, effectively adjusting the influence of recognition error and pose error on the vectorization accuracy, and increasing the robustness of the vectorization result.

[0055] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0056] Figure 3 The flowchart of the road element vectorization method provided by another embodiment of the present application is shown in the figure. The method can include:

[0057] S301, obtaining the tracking sequence of the tracked target road element, the trajectory pose corresponding to each frame of image in the tracking sequence, and the optimization initial value of the target road element.

[0058] In the embodiment, the execution subject can be a server or an electronic device, etc. It can be applied to monocular camera vectorization, which is not limited here. Each frame of image in the tracking sequence is used as a frame image to be optimized, and the target road element can be a traffic sign, a rod-shaped element, etc.

[0059] The target road element is tracked by a camera, and multiple frames of images are collected and a tracking sequence is constructed. Each frame of image in the tracking sequence is optimized, and then the influence of individual frame recognition deviation and pose anomaly on the vectorization position accuracy is reduced or removed through multi-frame optimization, and a high-precision vectorization result is obtained through multi-frame adjustment.

[0060] The method of obtaining the tracking sequence of the tracked target road element, the trajectory pose corresponding to each frame of image in the tracking sequence, and the optimization initial value of the target road element can include any of the following:

[0061] receiving a tracking sequence of a target road element sent by a vehicle terminal deployed on a vehicle, a track pose corresponding to each frame of image in the tracking sequence, and an optimization initial value of the target road element;

[0062] extracting a tracking sequence of a target road element, a track pose corresponding to each frame of image in the tracking sequence, and an optimization initial value of the target road element from a database;

[0063] receiving a tracking sequence of a target road element imported by an external device (such as a storage medium, etc.), a track pose corresponding to each frame of image in the tracking sequence, and an optimization initial value of the target road element.

[0064] S302, for the tracking sequence, according to the track pose corresponding to each frame of image, the optimization initial value of the target road element, and the pixel coordinates of each frame of image, the optimization result corresponding to the target object in which the residual model result meets the preset condition in the iteration process is determined by constructing a residual model.

[0065] Wherein, the target road element is any road element of any type of road element, and the optimization initial value includes three-dimensional position information of the target object of the target road element in the reference frame image. If the target road element is a traffic sign, the heading and scale of multiple frames of image can be optimized simultaneously on the basis of the three-dimensional position, thereby improving the accuracy of the optimization result and making the vectorization result more accurate. If the target road element is a rod-shaped element, only the three-dimensional position information needs to be optimized. Other road elements can be determined based on the properties of the road elements themselves to determine the items that can be optimized, which are not limited here.

[0066] S303, constructing a vectorization result of the target road element according to the optimization result of the target object.

[0067] In this embodiment, a better optimization result is obtained after graph optimization, and according to the optimization result, the vectorization result of the traffic sign of different shapes can be constructed.

[0068] The road element vectorization method provided by the embodiment firstly acquires a tracking sequence of a tracked target road element, a track pose corresponding to each frame of image in the tracking sequence, and an optimization initial value of the target road element; further, for the tracking sequence, according to the track pose corresponding to each frame of image, the optimization initial value of the target road element, and pixel coordinates of each frame of image, optimization is performed through a constructed residual error model to determine an optimization result of a target object corresponding to a residual error model result meeting a preset condition in an iterative process; further, according to the optimization result of the target object, a vectorization result of the target road element is constructed. Based on target tracking of a target road element, the application obtains a complete tracking sequence frame corresponding to each target road element, performs multi-frame image optimization in the tracking sequence according to the tracking result, constantly updates the optimization initial value, and obtains a final optimization result. Through the multi-frame image optimization mode, the influence of individual frame recognition deviation and pose anomaly on the vectorization position accuracy is reduced, and a high-precision vectorization result is obtained through multi-frame adjustment.

[0069] In a possible design, the target road element is any road element of any type of road element, and the optimization initial value includes three-dimensional position information of the target object of the target road element in a reference frame image; and the optimization of the tracking sequence according to the track pose corresponding to each frame of image, the optimization initial value of the target road element, and the pixel coordinates of each frame of image through the constructed residual error model to determine the optimization result of the target object corresponding to the residual error model result meeting the preset condition in the iterative process includes:

[0070] Each frame of image in the tracking sequence is traversed to perform the following operations: according to the track pose corresponding to each frame of image and the optimization initial value of the target road element, three-dimensional position information of the target object of the target road element in the current frame of image is determined; according to the three-dimensional position information of the target object of the target road element in the current frame of image, the three-dimensional position information of each object of the target road element in the current frame of image is mapped into image pixel coordinates through the camera extrinsic parameter and the camera intrinsic parameter; and according to the pixel coordinates of the current frame of image and the image pixel coordinates of each object after the mapping, a residual error corresponding to the current frame of image is calculated.

[0071] According to the residual error corresponding to each frame of image, the optimization result of the target object corresponding to the residual error model result meeting the preset condition in the iterative process is determined through the constructed residual error model.

[0072] In the embodiment, the image optimization mainly lies in constructing a residual error model to cyclically calculate a residual error of each frame participating in optimization in the tracking sequence {Q i} according to input quantities, and the residual error model is represented as:

[0073]

[0074] Wherein, ζ represents the total residual term, dx represents the position residual in the x direction, dy represents the position residual in the y direction, dθ represents the heading residual, and dw represents the scale residual.

[0075] Solving the single-point three-dimensional result (herein referred to as the single-point three-dimensional position p i ) of the relative to-be-optimized frame Qi, according to the single-point three-dimensional position p i and other terms corresponding to the optimization initial value, the three-dimensional position information of each corner point of the traffic sign can be restored. According to the three-dimensional position information of each corner point of the traffic sign, the three-dimensional position information is mapped to the image pixel coordinates through the coordinate system, and then based on the formula corresponding to each residual term, the residual of the response term is calculated, and the residual corresponding to each frame of image is calculated. Through iterative optimization of the residual model, the final optimization result is determined. The way of analyzing the data reliability, periodicity or other interference through the residual model is that, since the residual refers to, for example, the position residual in the x direction, the position residual in the y direction, the heading residual and the scale residual, the smaller the value of the residual model is, the smaller the deviation is, and thus the better the input value of the residual model, that is, the term corresponding to the optimization initial value (such as the three-dimensional position information, the heading, the scale, etc.) is.

[0076] Since multiple-frame optimization is easy to fall into local optimum, in order to avoid the optimizer falling into local optimum, an iterative initial value setting method is designed to avoid the optimizer falling into local optimum.

[0077] The calculation and origin of each residual term are specifically introduced as follows. The following introduction is the residual solving process of a certain frame Qi:

[0078] In a possible design, the determining, according to the trajectory pose corresponding to each frame of image and the optimization initial value of the target road element, of the three-dimensional position information of the target object of the target road element in the current frame of image includes:

[0079] According to the trajectory pose corresponding to each frame of image, the trajectory pose corresponding to the target adjacent frame of image and the trajectory pose corresponding to the reference frame of image of the target road element are obtained, the target adjacent frame of image being an adjacent frame of image relative to the current frame of image.

[0080] According to the three-dimensional position information of the target object of the target adjacent frame of image and the reference frame of image and the trajectory pose corresponding to the reference frame of image, the three-dimensional position information of the target object of the target adjacent frame of image is determined.

[0081] According to the principle of intersection of rays, the three-dimensional position information of the target object of the target adjacent frame of image is updated.

[0082] According to the pose relationship, the three-dimensional position information of the target object of the updated target adjacent frame image is converted to the three-dimensional position information of the target object relative to the current frame image.

[0083] The trajectory pose includes a pose and position information at the actual shooting time, and the three-dimensional position information is used to represent a relative three-dimensional position in a carrier coordinate system.

[0084] In this embodiment, the first method is used.

[0085] Solving the three-dimensional position information of the target object of the target frame Q i The three-dimensional position information of the target object of the target frame Q i The three-dimensional position information of the target object of the target frame Q i+1 The three-dimensional position information of the target object of the target frame Q i+1 The solving formula is shown in formula (2).

[0086] The three-dimensional position information of the target object of the target frame Q i+1 The three-dimensional position information of the target object of the target frame Q i+1 -1 The three-dimensional position information of the target object of the target frame Q i+1 -1 The three-dimensional position information of the target object of the target frame Q i+1 (2)

[0087] R i+1 and T i+1 are the pose information (i.e., the trajectory pose) P i+1 corresponding to the pose and the position information at the actual shooting time, wherein R0 and T0 are the pose information (i.e., the trajectory pose) P0 corresponding to the pose and the position information at the actual shooting time.

[0088] After obtaining the three-dimensional position information of the target object of the target frame Q i+1 , the three-dimensional position information of the target object of the target frame Q i+1 is updated according to the ray intersection principle. The solving process is as follows: the carrier system ray r calculated by the position corner C0 corresponding to the target frame Q i+1 intersects with the plane determined by the position information p i+1 and the heading angle θ, and the intersection position is the new p i+1 corresponding to the target frame Q i+1 . The significance of updating the position of the target frame Q i+1 is that there may be deviations between the trajectory pose and the sign corner itself, and the position after the back projection may not correspond to the corner corresponding to the target frame Q i+1 , so updating the position according to the ray intersection principle can reduce a part of the intermediate error term. After obtaining the three-dimensional position information of the target object of the target frame Q i+1 , the three-dimensional position information of the target object of the target frame Q i+1 is converted according to the pose relationship. iPosition p of the frame i , and further reduce the error of position p i .

[0089] Method two: after obtaining p i+1 based on method one, in order to facilitate calculation and further save calculation resources, based on the obtained p i+1 , the position p i of the frame relative to Q i is converted by using the pose relationship.

[0090] In a possible design, the three-dimensional position information is used to represent a relative three-dimensional position in a carrier coordinate system; if the target road element is a traffic sign, the target object is a target corner point in the traffic sign with an error within a preset range; and the three-dimensional position information of each object of the target road element in the current frame image is mapped into image pixel coordinates by using camera extrinsic parameters and camera intrinsic parameters according to the three-dimensional position information of the target object of the target road element in the current frame image, including:

[0091] According to the three-dimensional position information of the target corner point in the current frame image, the three-dimensional position information of each corner point of the traffic sign in the current frame image is determined, wherein the three-dimensional position information of each corner point of the traffic sign in the current frame image is relative to the carrier coordinate system of the current frame image.

[0092] For each of the corner points, the following operations are performed:

[0093] According to the camera extrinsic parameters, the carrier coordinate of the corner point is converted into a camera coordinate;

[0094] According to the camera intrinsic parameters, the camera coordinate of the corner point is converted into a pixel coordinate; wherein the converted pixel coordinate is the image pixel coordinate of the mapped corner point.

[0095] In this embodiment, the traffic sign vector position is restored: according to the single-point three-dimensional position p i , and the input heading, width w, and height h information of the traffic sign, the three-dimensional position information of each corner point of the traffic sign can be restored, and these three-dimensional position information are relative to the carrier coordinate system of the frame Q i to be optimized.

[0096] Corner point mapping image: according to the three-dimensional position information of each corner point of the traffic sign, the carrier coordinate is converted into a camera coordinate according to the camera extrinsic parameters Rc2b, and then the camera coordinate is converted into a pixel coordinate according to the camera intrinsic parameters K, and the mapped corner point coordinate is represented by {C pk}. Thus, the process of mapping the three-dimensional corner point into an image is completed. The mapping result is shown in Figure 4 .Figure 4 The image corner points {C k} (wherein, Figure 4 The mapping corner points {C pk} (wherein, Figure 4 The mapping corner points {C

[0097] In a possible design, the initial value of the optimization further includes an initial heading of the traffic sign and an initial scale of the traffic sign, the residual error includes: a position residual error in the x direction, a position residual error in the y direction, a heading residual error, and a scale residual error; and the calculation of the residual error corresponding to the current frame image according to the pixel coordinates of the current frame image and the image pixel coordinates of each object after mapping includes:

[0098] For each corner point of the traffic sign, the square of the x direction deviation is calculated according to the pixel coordinates of the current frame image corresponding to the corner point and the image pixel coordinates of the corner point after mapping, and the sum of the squares of the x direction deviations corresponding to each corner point is taken as the x direction residual error corresponding to the current frame image.

[0099] For each corner point of the traffic sign, the square of the x direction deviation is calculated according to the pixel coordinates of the current frame image corresponding to the corner point and the image pixel coordinates of the corner point after mapping, and the sum of the squares of the x direction deviations corresponding to each corner point is taken as the x direction residual error corresponding to the current frame image.

[0100] For each corner point of the traffic sign, the square of the x direction deviation is calculated according to the pixel coordinates of the current frame image corresponding to the corner point and the image pixel coordinates of the corner point after mapping, and the sum of the squares of the x direction deviations corresponding to each corner point is taken as the x direction residual error corresponding to the current frame image.

[0101] The sum of the x direction residual error corresponding to the current frame image, the y direction residual error corresponding to the current frame image, the heading residual error corresponding to the current frame image, and the scale residual error is calculated to obtain the residual error corresponding to the current frame image.

[0102] In this embodiment, the traffic sign is taken as an example to calculate the residual error.

[0103] According to the mapping coordinates {Cp k} of each corner point of the traffic sign obtained in the previous operation, the residual error calculated in this application includes four items, which are the x direction position residual error, the y direction position residual error, the heading residual error, and the scale residual error.

[0104] The x direction position residual error dx is the sum of the squares of the x direction deviations calculated for each corresponding corner point pixel position, which is represented by the following formula:

[0105]

[0106] Wherein, m represents the number of corner points, C kx represents the x-direction coordinate of the left side of the image corner point, and Cp k x represents the x-direction coordinate of the mapped corner point.

[0107] The position residual dy in the Y direction: that is, the sum of the squares of the y-direction deviations calculated for each corresponding corner point pixel position, which is expressed by the following formula:

[0108]

[0109] where m represents the number of corner points, and C k y represents the y-direction coordinate of the left side of the image corner point, and Cp k y represents the y-direction coordinate of the mapped corner point.

[0110] The heading residual dθ: The heading result of the traffic sign is constrained by the parallelism of the edges of the traffic sign, and the square of the slope deviation of the mapped and recognized corresponding edges is selected. The calculation process is as follows:

[0111]

[0112] where C k , C k-1 , and Cp k , Cp k-1 are two adjacent corner points corresponding to an edge parallel to the ground.

[0113] The scale residual dw: The scale information of the traffic sign is constrained by the consistency of the width information of the traffic sign, and the square of the width deviation of the mapped and recognized corresponding edges is selected. The calculation formula is as follows:

[0114] dw = (dx1-dx2) 2

[0115] where dx1 and dx2 are consistent with formula (3).

[0116] In one possible design, the optimization result of the residual model result meeting the preset condition in the iteration process is determined according to the residual corresponding to each frame image of the tracking sequence through the constructed residual model, including:

[0117] The sum of the residuals corresponding to each frame image in the tracking sequence is taken as the residual model result in the current iteration process.

[0118] According to the residual model result in the current iteration process, the next iteration operation of residual calculation is performed by adjusting the parameters until the target residual model result meeting the preset condition is determined from the residual model results corresponding to multiple iterations.

[0119] The optimization value corresponding to the target residual model result is taken as the optimization result; the optimization value includes the optimized three-dimensional position information, the optimized heading, and the optimized scale of any corner point of the traffic sign.

[0120] The preset condition is that the target residual model result is minimum or tends to be stable (for example, there is almost no change from the last iteration result, or the change range is within the preset range, and the target residual model result is considered to tend to be stable), or the number of iterations is reached. Different use scenarios or road element types need to be determined, and specific limitations are not given here.

[0121] It should be noted that the preset conditions corresponding to the target objects of different road element types can be the same or different, and specific limitations are not given here; for example, to facilitate calculation, the same preset condition can be used in the same vectorization scene (for example, a vectorization scene for signs or a vectorization scene for rod-shaped elements).

[0122] In this embodiment, the parameter type here is the parameter type corresponding to the initial optimization value. By adjusting the heading, scale, and three-dimensional position information, the final three-dimensional position information, heading, and scale are determined through continuous iteration and optimization. Specifically, the optimal optimization result is obtained after graph optimization, and according to the optimization result: the single-point three-dimensional position, heading, width, and height of the traffic sign, the vectorization result of the traffic sign of different shapes can be constructed. The output result is currently the body coordinate of the reference frame, which can be directly converted into the world coordinate system such as ECEF through the pose P0 in the track. Therefore, by innovatively adding the traffic sign edge parallel constraint and scale constraint in the optimizer residual model, the heading and width / high optimization of the traffic sign can be effectively constrained, and the overall optimal optimization result of the heading and size is obtained.

[0123] It should be noted that if the target road element is a rod-shaped element, the process of graph optimization is similar to the process of graph optimization based on the sign, except that when calculating the residual of the rod-shaped element, the position residual can be directly calculated, and the calculation method is similar to the above-mentioned x-direction position residual and y-direction position residual, which will not be repeated here.

[0124] In this embodiment, the tracking sequence can also be filtered to identify frames with poor results that do not participate in optimization, which is beneficial to improve the optimization accuracy.

[0125] The application fully analyzes the influence factors of the current traffic sign vectorization accuracy, and the pose and recognition are the main reasons affecting the accuracy. By effectively reducing the influence of the two, the traffic sign accuracy can be improved. Through the road element vectorization method proposed in the application, graph optimization is adopted, and the idea of multi-frame adjustment is used to reduce the influence of recognition and pose, so that real-time traffic sign vectorization has no obvious influence on frame rate and memory under the condition of ensuring low cost, and the vectorization accuracy is obviously improved, especially the position accuracy of the traffic sign above the trajectory is improved obviously. Therefore, based on the road element vectorization method proposed in the application, the accuracy of the road element vectorization result is high.

[0126] The application also provides a map updating method. The execution subject of the map updating method can be a server or an electronic device, which can be the same as or different from the execution subject of the road element vectorization method, and is not specifically limited herein. The map updating method comprises:

[0127] The method according to any one of the first aspect determines the vectorization result of the target road element;

[0128] According to the vectorization result of the target road element, the map is updated.

[0129] In the embodiment of the application, based on the target tracking of the target road element, the complete tracking sequence frame corresponding to each target road element is obtained, and after the three-dimensional measurement result corresponding to the reference frame image is obtained according to the tracking result, multi-frame graph optimization is performed on the tracking sequence to obtain the final optimization result such as single-point three-dimensional position. Based on the optimization result, the road element vectorization is realized, and then the map updating is realized. Through the multi-frame graph optimization mode, the influence of the individual frame recognition deviation and the pose anomaly on the vectorization position accuracy is reduced, the high-precision vectorization result is obtained through multi-frame adjustment, and the accuracy of the map making or updating is improved.

[0130] In order to realize the road element vectorization method, the embodiment provides a road element vectorization device. Referring to Figure 5 , Figure 5 The structure schematic diagram of the road element vectorization device provided in the embodiment of the application is shown in the figure; the road element vectorization device 50 comprises an acquisition module 501, an optimization module 502 and a vectorization module 503.

[0131] The acquisition module 501 is used for acquiring the tracking sequence of the tracked target road element, the trajectory pose corresponding to each frame of image in the tracking sequence, and the optimization initial value of the target road element.

[0132] The optimization module 502 is configured to optimize the tracking sequence according to the track pose corresponding to each frame of image, the optimization initial value of the target road element, and the pixel coordinates of each frame of image by using the constructed residual error model, and determine the optimization result of the target object corresponding to the residual error model result meeting the preset condition in the iteration process.

[0133] The vectorization module 503 is configured to construct the vectorization result of the target road element according to the optimization result of the target object.

[0134] In this embodiment, the acquisition module 501, the optimization module 502, and the vectorization module 503 are configured to acquire the tracking sequence of the tracked target road element, the track pose corresponding to each frame of image in the tracking sequence, and the optimization initial value of the target road element. Further, the tracking sequence is optimized according to the track pose corresponding to each frame of image, the optimization initial value of the target road element, and the pixel coordinates of each frame of image by using the constructed residual error model, and the optimization result of the target object corresponding to the residual error model result meeting the preset condition in the iteration process is determined. Further, the vectorization result of the target road element is constructed according to the optimization result of the target object. Based on the target tracking of the target road element, the complete tracking sequence frame corresponding to each target road element is obtained, and the multi-frame image optimization is performed on the tracking sequence according to the tracking result, the optimization initial value is continuously updated, and the final optimization result is obtained. Through the multi-frame image optimization, the influence of the individual frame recognition deviation and the abnormal pose on the vectorization position accuracy is reduced, and the high-precision vectorization result is obtained through multi-frame adjustment.

[0135] The road element vectorization device provided in this embodiment can be used to execute the technical solutions of the road element vectorization method embodiments, and has similar implementation principles and technical effects, which will not be described here again.

[0136] In a possible design, the target road element is any road element of any type of road element, and the optimization initial value includes the three-dimensional position information of the target object of the target road element in the reference frame image. The optimization module is specifically configured to:

[0137] The following operations are performed by traversing each frame of image in the tracking sequence: the three-dimensional position information of the target object of the target road element in the current frame of image is determined according to the track pose corresponding to each frame of image and the optimization initial value of the target road element; the three-dimensional position information of each object of the target road element in the current frame of image is mapped into image pixel coordinates by using the camera external parameter and the camera internal parameter according to the three-dimensional position information of the target object of the target road element in the current frame of image; and the residual error corresponding to the current frame of image is calculated according to the pixel coordinates of the current frame of image and the image pixel coordinates of each object after the mapping.

[0138] According to the residual error corresponding to each frame image, an optimization result is determined by the constructed residual error model, wherein the optimization result is in accordance with a preset condition in the iterative process.

[0139] In a possible design, the optimization module is specifically configured to:

[0140] According to the trajectory pose corresponding to each frame image, the trajectory pose corresponding to a target adjacent frame image and the trajectory pose corresponding to the reference frame image of the target road element are obtained, wherein the target adjacent frame image is an adjacent frame image relative to the current frame image.

[0141] According to the trajectory pose corresponding to the target adjacent frame image, the three-dimensional position information of the target object of the target adjacent frame image, and the trajectory pose corresponding to the reference frame image, the three-dimensional position information of the target object of the target adjacent frame image is determined.

[0142] According to the ray intersection principle, the three-dimensional position information of the target object of the target adjacent frame image is updated.

[0143] According to the pose relationship, the updated three-dimensional position information of the target object of the target adjacent frame image is converted into the three-dimensional position information of the target object relative to the current frame image.

[0144] In a possible design, the three-dimensional position information is used to represent a relative three-dimensional position in a carrier coordinate system; if the target road element is a traffic sign, the target object is a target corner point in the traffic sign with an error within a preset range; and the optimization module is specifically configured to:

[0145] According to the three-dimensional position information of the target corner point of the current frame image, the three-dimensional position information of each corner point of the traffic sign in the current frame image is determined, wherein the three-dimensional position information of each corner point of the traffic sign in the current frame image is relative to the carrier system coordinate of the current frame image.

[0146] For each of the corner points, the following operations are performed:

[0147] According to the camera external parameter, the carrier system coordinate corresponding to the corner point is converted into a camera system coordinate.

[0148] According to the camera internal parameter, the camera system coordinate corresponding to the corner point is converted into a pixel coordinate; wherein the converted pixel coordinate is the image pixel coordinate of the mapped corner point.

[0149] In a possible design, the optimization initial value further includes an initial heading of the traffic sign and an initial scale of the traffic sign, the residual error includes a position residual error in an x direction, a position residual error in a y direction, a heading residual error, and a scale residual error; and the optimization module is specifically configured to:

[0150] According to the pixel coordinates of the current frame image corresponding to the corner points and the image pixel coordinates of the mapped corner points, the square of the x-direction deviation is calculated, and the sum of the squares of the x-direction deviations corresponding to each corner point is taken as the x-direction residual error corresponding to the current frame image.

[0151] According to the pixel coordinates of the current frame image corresponding to the corner points and the image pixel coordinates of the mapped corner points, the square of the y-direction deviation is calculated, and the sum of the squares of the x-direction deviations corresponding to each corner point is taken as the y-direction residual error corresponding to the current frame image.

[0152] According to the pixel coordinates of the current frame image corresponding to the corner points and the image pixel coordinates of the mapped corner points, the heading residual error and the scale residual error corresponding to the current frame image are calculated.

[0153] The sum of the x-direction residual error corresponding to the current frame image, the y-direction residual error corresponding to the current frame image, the heading residual error corresponding to the current frame image, and the scale residual error is calculated to obtain the residual error corresponding to the current frame image.

[0154] In a possible design, the vectorization module is specifically configured to:

[0155] The sum of the residual errors corresponding to each frame image in the tracking sequence is taken as the residual model result in the current iteration process.

[0156] According to the residual model result in the current iteration process, the next iteration operation of residual calculation is performed by adjusting the parameters, until the target residual model result meeting the preset condition is determined from the residual model results corresponding to multiple iterations.

[0157] The optimization value corresponding to the target residual model result is taken as the optimization result; the optimization value includes the optimized three-dimensional position information, the optimized heading, and the optimized scale of any corner point of the traffic sign.

[0158] In order to realize the map updating method, an embodiment of the present application provides a map updating device. The map updating device comprises:

[0159] The processing module is configured to determine the vectorization result of the target road element based on the method of any one of the first aspect.

[0160] The map updating module is configured to update the map according to the vectorization result of the target road element.

[0161] The map updating apparatus provided by the embodiment can be used to execute the technical solutions of the map updating method embodiments, and has similar implementation principles and technical effects, which will not be repeated here.

[0162] To implement the road element vectorization method, the embodiment provides an electronic device. Figure 6 As shown in the structural schematic diagram of the electronic device provided by the embodiment of the present application, the electronic device 60 of the embodiment comprises at least one processor 601 and a memory 602. Figure 6 The memory 602 is used to store computer execution instructions, and the at least one processor 601 is used to execute the computer execution instructions stored by the memory to implement the steps executed in the above-mentioned embodiments. For details, please refer to the related description in the foregoing method embodiments.

[0163] The embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the method as described above is implemented.

[0164] The embodiment of the present application further provides a computer program product, comprising a computer program, which is executed by the processor to implement the method as described above.

[0165] In the several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the modules is only a logical function division. For example, a plurality of modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed can be indirect coupling or communication connection through some interface, device or module, and can be electrical, mechanical or in other forms. In addition, each functional module in the embodiments of the present application can be integrated in one processing unit, or each module can exist alone physically, or two or more modules can be integrated in one unit. The units formed by the above modules can be realized in the form of hardware, or in the form of hardware plus software function units.

[0166] The integrated modules realized in the form of software function modules can be stored in a computer readable storage medium. The software function modules are stored in a storage medium and include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the method described in various embodiments of the present application. It should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the application can be directly embodied as hardware processor execution or executed by a combination of hardware and software modules in the processor.

[0167] The memory can include a high-speed RAM memory, and can also include a non-volatile storage NVM, such as at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc. The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit only one bus or one type of bus. The storage medium described above can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0168] An example storage medium is coupled to the processor such that the processor can read information from, and can write information to, the storage medium. Of course, the storage medium can be a part of the processor. Consistent with the teachings provided herein, the processor (acting in response to a plurality of instructions executed by the processor) can be capable of implementing any of the features, steps, or functions disclosed herein, including the methods and techniques described above. The storage medium can be realized as a non-transitory storage medium. The storage medium can include one or more of the following: a RAM, a ROM, an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or any other tangible medium that can hold instructions or data.

[0169] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program, when executed, executes steps including the above-mentioned method embodiments; and the foregoing storage medium includes: ROM, RAM, magnetic disk or optical disk and various media that can store program codes.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for vectorizing road elements, characterized in that, include: The tracking sequence of the tracked target road features, the trajectory pose corresponding to each frame of the tracking sequence, and the optimized initial value of the target road features are obtained. For the tracking sequence, based on the trajectory pose corresponding to each frame of image, the initial optimization value of the target road element, and the pixel coordinates of each frame of image, optimization is performed through the constructed residual model to determine the optimization result corresponding to the target object whose residual model result meets the preset conditions during the iteration process; Based on the optimization results of the target object, a vectorized result of the target road element is constructed.

2. The method according to claim 1, characterized in that, The target road element is any road element of any type of road element, and the initial optimization value includes the three-dimensional position information of the target road element in the target object of the reference frame image. For the tracking sequence, based on the trajectory pose corresponding to each frame of image, the initial optimization value of the target road element, and the pixel coordinates of each frame of image, optimization is performed using a constructed residual model. The optimization result for the target object whose residual model result meets preset conditions during the iteration process is determined, including: Traverse each frame of the tracking sequence and perform the following operations: Based on the trajectory pose corresponding to each frame and the optimized initial value of the target road element, determine the 3D position information of the target object in the current frame of the target road element; Based on the 3D position information of the target object in the current frame of the target road element, map the 3D position information of each object in the current frame of the target road element to image pixel coordinates using camera extrinsic and intrinsic parameters; Calculate the residual corresponding to the current frame of the image based on the pixel coordinates of the current frame and the mapped image pixel coordinates of each object. Based on the residuals corresponding to each frame of the image, the optimization results that meet the preset conditions are determined by constructing a residual model.

3. The method according to claim 2, characterized in that, The step of determining the 3D position information of the target object in the current frame image based on the trajectory pose corresponding to each frame image and the optimized initial value of the target road element includes: Based on the trajectory pose corresponding to each frame image, the trajectory pose of the target road element in the target adjacent frame image and the trajectory pose corresponding to the reference frame image are obtained, wherein the target adjacent frame image is the adjacent frame image relative to the current frame image. The three-dimensional position information of the target object in the adjacent frame image is determined based on the trajectory pose corresponding to the target, the three-dimensional position information of the target object in the reference frame image, and the trajectory pose corresponding to the reference frame image. Based on the principle of ray intersection, update the three-dimensional position information of the target object in the adjacent frame images of the target; Based on the pose relationship, the 3D position information of the target object in the updated target neighboring frame images is converted to the 3D position information of the target object relative to the current frame image.

4. The method according to claim 2 or 3, characterized in that, The three-dimensional position information is used to represent the relative three-dimensional position in the carrier coordinate system; if the target road element is a traffic sign, the target object is the target corner point of the traffic sign with an error within a preset range; the step of mapping the three-dimensional position information of each object of the target road element in the current frame image to image pixel coordinates based on the three-dimensional position information of the target object in the current frame image, through camera extrinsic and intrinsic parameters, includes: Based on the three-dimensional position information of the target corner point of the current frame image, the three-dimensional position information of the traffic sign at each other corner point of the current frame image is determined, wherein the three-dimensional position information of the traffic sign at each corner point of the current frame image is relative to the system coordinates of the current frame image. For each of the aforementioned corner points, perform the following operations: Based on the camera extrinsic parameters, the coordinates of the vehicle system corresponding to the corner point are transformed to the camera system coordinates; Based on the camera intrinsic parameters, the camera coordinates corresponding to the corner point are converted to pixel coordinates; wherein the converted pixel coordinates are the image pixel coordinates of the mapped corner point.

5. The method according to claim 4, characterized in that, The initial optimization values ​​also include the initial heading and initial scale of the traffic sign. The residuals include: position residuals in the x-direction, position residuals in the y-direction, heading residuals, and scale residuals. The step of calculating the residuals corresponding to the current frame image based on the pixel coordinates of the current frame image and the mapped image pixel coordinates of each object includes: For each corner point of the traffic sign, the square of the x-direction deviation is calculated based on the pixel coordinates of the corner point in the current frame image and the image pixel coordinates of the mapped corner point, and the sum of the squares of the x-direction deviations corresponding to each corner point is taken as the x-direction residual corresponding to the current frame image. Based on the pixel coordinates of the corner point corresponding to the current frame image and the image pixel coordinates of the mapped corner point, the square of the y-direction deviation is calculated, and the sum of the squares of the x-direction deviations corresponding to each corner point is taken as the y-direction residual corresponding to the current frame image. Calculate the heading residual and scale residual corresponding to the current frame image based on the pixel coordinates of the corner point corresponding to the current frame image and the image pixel coordinates of the mapped corner point; The residual corresponding to the current frame image is obtained by calculating the sum of the x-direction residual, the y-direction residual, the heading residual, and the scale residual corresponding to the current frame image.

6. The method according to claim 4, characterized in that, The step of determining the optimization result that meets preset conditions based on the residuals corresponding to each frame image, through the constructed residual model, includes: The sum of the residuals corresponding to each frame image in the tracking sequence is used as the residual model result in the current iteration. Based on the residual model results in the current iteration, the parameters are adjusted to execute the next iteration operation of residual calculation until the target residual model result that meets the preset conditions is determined from the residual model results corresponding to multiple iterations. The optimized value corresponding to the result of the target residual model is used as the optimization result; the optimized value includes the optimized three-dimensional position information of any corner point of the traffic sign, the optimized heading, and the optimized scale.

7. A method for updating a map, characterized in that, include: Based on the method according to any one of claims 1-6, determine the vectorization result of the target road element; The map is updated based on the vectorization results of the target road features.

8. A road element vectorization device, characterized in that, include: The acquisition module is used to acquire the tracking sequence of the tracked target road features, the trajectory pose corresponding to each frame of the tracking sequence, and the optimized initial value of the target road features. The optimization module is used to optimize the tracking sequence based on the trajectory pose corresponding to each frame of image, the initial optimization value of the target road element, and the pixel coordinates of each frame of image, by constructing a residual model, and to determine the optimization result corresponding to the target object whose residual model result meets the preset conditions during the iteration process. The vectorization module is used to construct the vectorized result of the target road elements based on the optimization result of the target object.

9. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium and / or computer program product, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 7; the computer program product includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images