Image processing method and device and model training method and device

Abstract

There is provided an image processing method, comprising: acquiring a forward view of a vehicle, the forward view at least showing an area in front of the vehicle; acquiring parameters of a camera used for shooting the forward view, wherein the parameters comprise external parameters of the camera; and inputting the forward view and parameters of the camera into a neural network model to obtain a semantically segmented top-down view. Therefore, an end-to-end method is provided, and distortion occurring in the process of converting features in a forward view into features in a top-down view is avoided.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to image processing for intelligent transportation tools. Background Art

[0002] As part of image processing technology, image perception technology can be used in the field of autonomous driving to perceive the surrounding environment of the vehicle, thereby serving one or more vehicles to provide a safe driving environment. Semantic segmentation can be used to identify objects in the road ahead of the vehicle, thereby facilitating the provision of driving assistance to the user or making decisions about the vehicle's driving path in autonomous driving. For example, semantic segmentation can be used to detect free space in the road, that is, space where the vehicle can travel without collision, thereby facilitating the determination of the vehicle's subsequent driving path.

[0003] Because the vehicle is traveling on a road in the real world, it is usually desirable to obtain information related to the target in the real world coordinate system. Therefore, in the process of performing semantic segmentation on the image to identify the target on the road, it is desirable to provide target information in the real world coordinate system rather than the vehicle's own coordinate system.

[0004] Currently, in order to obtain target information in the real-world coordinate system, one method requires first obtaining target information in the vehicle-side image (the image of the vehicle's forward perspective, referred to as the forward view), and then using the inverse perspective transformation (IPM) to transform it to obtain the target information in the top-down view (also known as the bird's eye view); another method uses the camera's external parameters to project image features from the forward view to the top-down view. However, improved methods are still desired. Summary of the invention

[0005] It is desirable to provide an improved image processing method and device that can perform semantic segmentation on a target in an image so as to obtain target information in a real-world coordinate system based on a forward view, while avoiding distortion as much as possible to provide protection for traffic safety.

[0006] According to one aspect, an image processing method is provided, comprising: obtaining a forward view of a vehicle, the forward view showing at least an area in front of the vehicle; obtaining parameters of a camera used to capture the forward view, the parameters including external parameters of the camera; and inputting the forward view and the parameters of the camera into a neural network model to obtain a semantically segmented top-down view.

[0007] According to another aspect, a training method for a neural network model is provided, wherein the neural network model receives a forward view of a vehicle and parameters of a camera for capturing the forward view to output a semantically segmented top-down view of a target, wherein the forward view shows at least an area in front of the vehicle, and the parameters include extrinsic parameters of the camera, and the training method comprises: acquiring a training forward view, training parameters of the camera corresponding to the training forward view, and a semantically segmented top-down training true view of the target of the training forward view; and training the neural network model based on the training forward view, the training parameters of the camera, and the training true view.

[0008] According to another aspect, an image processing device is provided, comprising: an acquisition unit, which acquires a forward view of a vehicle and parameters of a camera used to capture the forward view, wherein the forward view at least shows an area in front of the vehicle, and the parameters include external parameters of the camera; and a processing unit, which inputs the forward view and the parameters of the camera into a neural network model to obtain a semantically segmented top-down view.

[0009] According to another aspect, a training device for a neural network model is provided, wherein the neural network model receives a forward view of a vehicle and parameters of a camera for capturing the forward view to output a semantically segmented top-down view of a target, wherein the forward view at least shows an area in front of the vehicle, and the parameters include extrinsic parameters of the camera, and the training device comprises: an acquisition unit, which acquires a training forward view, training parameters of the camera corresponding to the training forward view, and a semantically segmented top-down training true view of the target of the training forward view; and a training unit, which trains the neural network model based on the training forward view, the training parameters of the camera, and the training true view.

[0010] According to another aspect, a computer-readable medium is provided, storing a computer program unit, which, when executed by a processor or a computer, enables the processor or the computer to perform the image processing method according to the various embodiments of the present invention or the training method of the neural network model according to the various embodiments of the present invention.

[0011] According to various embodiments of various aspects of the present invention, the present invention provides an end-to-end image processing method by training a neural network model to learn the correspondence between each pixel in the forward view and the top-down view of a vehicle. For the trained neural network model, the input of the forward view of a vehicle such as a vehicle can directly output the semantic segmentation result for the target in the real-world coordinate system, specifically output the semantic segmentation result for the target in the top-down view. Thereby, the distortion that occurs in the process of transforming the features in the forward view into the features in the top-down view is avoided, and the distortion is caused by the plane assumption in the inverse perspective transformation. Specifically, the IPM transformation is a nonlinear transformation between two planes, which will cause distortion for objects on the ground. With the help of the neural network model of the present invention, the pixel-by-pixel transformation between the forward view and the top-down view can be learned, avoiding the generation of distortion and being able to directly output the semantically segmented view.

[0012] According to one embodiment of various aspects of the present invention, a neural network model is trained as follows: obtaining variation data of the camera's extrinsic parameters; obtaining a training forward view for training the neural network model, training parameters of the camera corresponding to the training forward view, and a semantically segmented top-down training truth view of the target of the training forward view; adjusting the training forward view, the training parameters of the camera, and the training truth view based on the variation data; and training the neural network model based on the adjusted training forward view, the training parameters, and the training truth view. In a further embodiment, the variation data of the camera's extrinsic parameters is obtained as follows: obtaining vibration distribution data obtained by distribution fitting vibration data representing the camera's vibration; selecting specific vibration data of the camera based on the vibration distribution data; and determining the variation data of the camera's extrinsic parameters based on the specific vibration data of the camera.

[0013] Therefore, the camera's extrinsic parameter change data is introduced in the process of training the neural network model, especially the camera's extrinsic parameter change data caused by the camera's vibration. By introducing the camera's extrinsic parameter change data, the trained neural network model takes the extrinsic parameter change into account, providing a neural network model that is aware of the extrinsic parameter change, thereby being able to obtain a semantic segmentation result that is more in line with the actual situation, improving the transformation error from the forward view to the real-world coordinate system caused by the change of the camera's extrinsic parameters, and improving the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Embodiments are illustrated by way of example only and not by way of limitation in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

[0015] Figure 1An example of free space semantic segmentation results is shown;

[0016] Figure 2 A scenario of performing semantic segmentation of an image using a neural network model according to an embodiment of the present invention is shown;

[0017] Figure 3 A flowchart of an image processing method according to an embodiment of the present invention is shown;

[0018] Figure 4 A scenario of training a neural network model according to an embodiment of the present invention is shown;

[0019] Figure 5 A flowchart of a method for training a neural network model according to an embodiment of the present invention is shown;

[0020] Figure 6 The process of obtaining a training truth view for training a neural network model according to one embodiment of the present invention is shown;

[0021] Figure 7 A block diagram of an image processing device according to an embodiment of the present invention is shown;

[0022] Figure 8 A block diagram of a training device according to one embodiment of the present invention is shown.

[0023] Various aspects and features of various embodiments of the present invention are described with reference to the above drawings. The above drawings are only illustrative and not restrictive. In addition, not all parts of the device of the embodiment of the present invention are marked with reference numerals in the above drawings. Only relevant parts are marked in some drawings, which does not limit the parts to those shown in the drawings of the specification. DETAILED DESCRIPTION

[0024] For a vehicle, on the one hand, a camera installed on the vehicle side can obtain a forward view of the vehicle, that is, a view facing forward of the vehicle in the direction of vehicle travel for the camera on the vehicle side, which shows the road conditions in front of the vehicle; on the other hand, a camera installed on the road side (e.g., on a traffic pole on the road side) can obtain a forward view of the vehicle, that is, a view facing forward of the direction of vehicle travel for the camera on the road side, which can also show the road conditions in front of the vehicle. Regardless of whether the forward view is obtained by the camera on the vehicle side or the camera on the road side, it is expected to be converted into a real-world coordinate system, so as to facilitate the display of the result of semantic segmentation of the target in the view in the real-world coordinate system, which is convenient for users to understand and use the target information therein.

[0025] The following description will be made with reference to vehicles, but those skilled in the art should understand that the various embodiments of the present invention are not limited to the field of vehicles, but can actually be applied to any type of vehicles with similar requirements, such as ships, etc. In addition, the "camera" mentioned in the following embodiments, unless otherwise specified, covers any type of camera set at the vehicle (vehicle) end and the roadside.

[0026] The target to be semantically segmented may be any type of target that the user is interested in, such as a vehicle in the road ahead of the vehicle during its travel, or a free space in the road ahead of the vehicle that can be traveled without collision during its travel. Various embodiments of the present invention will be described below with reference to free space, but this is not restrictive. Figure 1 An example of the free space semantic segmentation result is shown in the figure. The white part in the figure represents the road, the two small gray squares represent the vehicles as traffic participants, and the black part represents the area outside the camera's field of view. In addition, each vehicle will make part of the road invisible to the camera, which is indicated by the dotted line in the figure. For each vehicle, the area between the two dotted lines is the occluded (invisible) area. The space on the road that is not marked with vehicles and occluded areas is the free space where vehicles can travel. The area other than the free space is the non-free space.

[0027] Free space can be used as a reference for vehicle driving and help formulate subsequent driving paths. Free space detection can be performed through image perception and semantic segmentation. Usually, free space information can be first obtained in the forward view obtained by the camera, and then projected into the real-world coordinate system, that is, transformed into a top-down view. On the one hand, this can be achieved through the IPM transformation, but as mentioned earlier, this will introduce distortions and thus cannot provide accurate free space information. On the other hand, camera extrinsics can be used to perform the above projection, but this only roughly projects the image features and does not take into account the vibration of the camera. In fact, during the image capture process, the camera may vibrate with its installation position, which will cause its extrinsics to change. It is difficult to obtain accurate projection using fixed camera extrinsics, which in turn affects the accuracy of the acquired free space information.

[0028] According to various embodiments of the present invention, an end-to-end neural network model is introduced, in which the correspondence between the front view and the top-down view of the vehicle, especially the correspondence between the pixels therein, is learned, and the semantic segmentation result of the free space in the top-down view in the real world coordinate system can be output as long as the front view of the vehicle is input. Thus, the distortion caused by the plane assumption in the inverse perspective transformation in the process of transforming the features in the front view into the features in the top-down view is avoided, and accurate projection can be obtained, thereby obtaining accurate free space information.

[0029] Specifically, the neural network model receives the forward view of the vehicle and the parameters (especially external parameters) of the camera that captures the forward view, and outputs a top-down view (i.e., a view containing free space information in a real-world coordinate system) that performs semantic segmentation on the free space. The neural network model is a neural network model that has been trained to learn the correspondence between the forward view and the top-down view of the vehicle. In one embodiment, the neural network model includes at least a transformer and a semantic segmenter, the transformer being used to transform features in the forward view into features in the top-down view of the vehicle based on the parameters of the camera, and the semantic segmenter being used to perform semantic segmentation on the view based on the features in the top-down view to obtain free space information.

[0030] Figure 2 FIG. 2 shows a scenario of using a neural network model to perform semantic segmentation of an image according to an embodiment of the present invention. Figure 2 As shown, the forward view of the vehicle and the parameters C of the camera that captures the forward view are input into the neural network model M. The forward view of the vehicle can be captured by a camera mounted on a traffic pole. The parameters C of the camera include at least the external parameters p of the camera, and can also be expected to include the internal parameters k of the camera. The output of the neural network model M is a top-down view FS showing the free space segmentation result after semantic segmentation. The neural network model can be a deep neural network model.

[0031] In the neural network model, features in the front view are extracted, and the parameters of the camera are transformed into a camera parameter vector, based on which the features in the front view can be transformed into features in a top-down view of the vehicle. In one example, a function f for performing the transformation from the front view to the top-down view is obtained based on the camera parameter vector. cam (k, p). The neural network model may include a multi-layer perceptron, and the parameters of the camera may be input into the multi-layer perceptron to transform the parameters of the camera into a camera parameter vector.

[0032] In an example, Figure 2As shown, the neural network model M includes a transformer T and a semantic segmenter S. The transformer T can transform the features in the front view into the features in the top-down view based on the above function. It is also expected that the transformer also performs the above-mentioned operations such as feature extraction of the front view and transformation of camera parameters. The semantic segmenter S then performs semantic segmentation based on the features in the top-down view (i.e., the view in the real-world coordinate system) to obtain a semantic segmentation result FS for the free space. Afterwards, the model M outputs the semantic segmentation result FS.

[0033] Through the above neural network model, there is no need to make any plane assumptions. It is only necessary to learn the correspondence between the front view and the top-down view (especially the correspondence between each pixel) through model training. The use of such a neural network model avoids the occurrence of distortion and can achieve end-to-end free space semantic segmentation to obtain accurate free space semantic segmentation results that conform to the real world.

[0034] Figure 3 FIG. 1 is a flowchart of an image processing method 100 according to an embodiment of the present invention. Figure 3 The image processing method 100 shown in FIG. 1 comprises: in step 110, obtaining a forward view of a vehicle, wherein the forward view at least shows an area in front of the vehicle; in step 120, obtaining parameters of a camera used to capture the forward view, wherein the parameters at least include external parameters of the camera, and it is also expected that the parameters include internal parameters of the camera; in step 130, inputting the forward view and the parameters of the camera into the above-mentioned combined method. Figure 2 The described neural network model is used to obtain a semantically segmented top-down view of the free space, that is, a view including the semantic segmentation results for the free space in the real-world coordinate system; in step 140, the semantically segmented top-down view of the vehicle is output.

[0035] As mentioned above, during the image capture process, the camera may vibrate depending on its installation position, which will cause its extrinsic parameters to change. It is difficult to obtain accurate projection using fixed camera extrinsics, which in turn affects the accuracy of the acquired free space information.

[0036] To this end, according to an embodiment of the present invention, the change data of the camera's extrinsic parameters are introduced during the training of the neural network model, especially the change data of the camera's extrinsic parameters caused by the camera's vibration. By introducing the change data of the camera's extrinsic parameters, the trained neural network model takes the change of the extrinsic parameters into account, thereby being able to obtain a more accurate semantic segmentation result that is more in line with the actual situation.

[0037] In one embodiment, the neural network model is trained as follows: obtaining change data of the camera's extrinsic parameters; obtaining a training forward view for training the neural network model, training parameters of the camera corresponding to the training forward view (which includes the camera's extrinsic parameters and optional intrinsic parameters), and a semantically segmented top-down training truth view of the training forward view about free space; adjusting the training forward view, the camera's training parameters, and the semantically segmented top-down training truth view based on the change data of the camera's extrinsic parameters; and using the adjusted training forward view, training parameters, and training truth view to train the neural network model. Figure 2 The neural network model shown is trained. Figure 4-6 A training method for a neural network model according to an embodiment of the present invention is described in detail.

[0038] Figure 4 FIG. 2 shows a scenario of training a neural network model according to an embodiment of the present invention. Figure 4 As shown, the acquired change data EV of the camera's extrinsic parameters is applied to the training forward view TI for training the neural network model M and the parameters of the camera corresponding to the training forward view (in order to distinguish from the parameters of the camera involved in the model application, it is referred to as the camera's training parameters TC in this article). In addition, the acquired change data EV of the camera's extrinsic parameters is also applied to the true value (i.e., the semantically segmented top-down training truth view TGT corresponding to the training forward view) for training the neural network model M. The training forward view, the camera's training parameters, and the semantically segmented top-down training ground truth view are adjusted based on the extrinsic parameter change data EV, so that the adjusted training forward view, the camera's training parameters, and the training ground truth view correspond to the changed camera extrinsic parameters. Then the neural network model M is trained based on the adjusted training forward view, the camera's training parameters, and the semantically segmented top-down training ground truth view.

[0039] In one embodiment, the change data EV of the external parameters of the camera corresponds to specific vibration data selected from the vibration distribution data obtained by fitting the vibration data representing the vibration of the camera, and is determined based on the specific vibration data. It is also possible to directly perform distribution fitting on the change of the external parameters of the camera, and directly select specific change data from the distribution data representing the change of the external parameters of the camera to introduce into the training of the neural network model M.

[0040] In one embodiment, the transformation matrix of the calibration extrinsic parameters relative to the camera after the vibration occurs can be determined based on the obtained extrinsic parameter change data, and then the training forward view, the camera's training parameters and the training true value view are adjusted based on the transformation matrix.

[0041] Although the reference Figure 4A scenario for training a neural network model is described, but this is not restrictive. In some embodiments, it is only necessary to train the neural network model to achieve end-to-end free space semantic segmentation to obtain free space semantic segmentation results that are accurate to the real world.

[0042] Figure 5 A flowchart of a method 200 for training a neural network model according to an embodiment of the present invention is shown. The neural network model may be a reference Figure 2 According to the training method 200, in step 210, a training forward view, training parameters of a camera corresponding to the training forward view, and a semantically segmented true view of the training forward view for free space in a real-world coordinate system, i.e., a semantically segmented top-down training true view, are obtained. In step 220, the neural network model is trained based on the training forward view, the training parameters of the camera, and the semantically segmented true view of the free space in the real-world coordinate system.

[0043] In one embodiment, as mentioned above Figure 4 As described, the training method may also include obtaining change data of the camera's external parameters; adjusting the obtained training forward view, the camera's training parameters, and the top-down training truth view after semantic segmentation based on the change data; and training the neural network model using the adjusted training forward view, training parameters, and training truth view.

[0044] In one embodiment, the change data of the external parameters of the camera can be obtained as follows: obtain vibration distribution data obtained by performing distribution fitting on vibration data representing the vibration of the camera; select specific vibration data of the camera based on the vibration distribution data; and then determine the change data of the external parameters of the camera based on the specific vibration data of the camera.

[0045] The vibration distribution data of the camera can be obtained by acquiring vibration data representing the vibration of the camera from an inertial sensor and performing distribution fitting on the vibration data. The inertial sensor is set at any position that can vibrate together with the camera, for example, directly on the camera.

[0046] Although the training method of the neural network model is described with reference to the above steps 210-220, this is not restrictive and covers various embodiments of the training method of the neural network model of the present invention. For example, in step 210, the training front view, the training parameters of the camera corresponding to the training front view, and the semantically segmented ground truth view for the free space in the real world coordinate system corresponding to the training front view can be obtained in various different ways.

[0047] In one embodiment, Figure 6The process of obtaining the training truth view for training the neural network model is shown. The training of the neural network model (especially the deep neural network model) requires a large amount of labeled data. Figure 6 The process shown makes it relatively easy to obtain large amounts of labeled training data.

[0048] like Figure 6 As shown, at step 310, multiple frames of images including sparse point cloud data from a radar sensor (such as a LiDAR sensor) are stacked (stacking) to obtain a stacked view including dense point cloud data, and the LiDAR sensor is scanned non-repetitively. By stacking multiple frames of images, moving targets (such as moving vehicles) on the road surface can be removed, and thus, next, at step 320, by segmenting the stacked view including dense point cloud data, a road surface view showing the road surface can be obtained. There may be holes in the area representing the road surface in the segmented road surface view, and it is expected that image processing means will be used to fill these holes, thereby improving training reliability.

[0049] After the road surface view is obtained according to the above steps, it is possible to incorporate marks for targets and non-targets therein. When the target is a free space, it is possible to incorporate marks for free space and non-free space in the road surface view. The marks for non-free space include: marks for foreground objects (e.g., vehicles) and / or marks for occlusion areas corresponding to foreground objects. The foreground objects include non-free space objects (e.g., vehicles) superimposed or spliced ​​in the road surface view and non-free space objects originally shown in the road surface view (e.g., roadblocks or flower beds originally shown in the road surface view). Specifically, the marks for non-free space may include not only marks for foreground objects (e.g., vehicles) superimposed or spliced ​​in the road surface view and occlusion areas corresponding to the foreground objects, but may also include marks for foreground objects (e.g., roadblocks or flower beds) originally existing in the road surface view and occlusion areas corresponding to the originally existing foreground objects, for example, steps 330 and 340 may be included, at step 330, the marked target area representing the traffic participant (e.g., vehicle) is superimposed or spliced ​​in the road surface view obtained by segmentation. These marked areas can correspond to various areas in a forward view of a certain frame used for model training. Manual marking or automatic clustering methods can be used to obtain markings, and the marked target areas are displayed in the frame image F and can be added to the view showing the road surface. At step 340, it is also possible to mark areas that cannot be seen by radar measurement in the road surface view, that is, mark blocked areas, such as areas that are invisible to the radar due to the vehicle blocking the radar field of view. Specifically, the area that is invisible due to the obstruction of the vehicle VH is marked in the view at step 340. This is merely indicative, and other vehicles or objects can also be marked. After marking non-targets (such as non-free space), the remaining part is the target (free space). At step 350, the part outside the camera's field of view is removed, leaving only the part within the camera's field of view. In this way, a marked true value view for training a neural network model can be obtained.

[0050] Already referenced Figure 1-6 The methods and scenarios according to various embodiments of the present invention are described. Those skilled in the art should understand that this is not restrictive, and the various method steps therein can be arbitrarily combined / changed / split, etc., as long as the technical effects of the present invention can be achieved.

[0051] Figure 7 FIG. 1 shows a block diagram of an image processing device 10 according to an embodiment of the present invention. Figure 7 As shown, the image processing device 10 includes an acquisition unit 11 , a processing unit 12 and an output unit 13 .

[0052] The acquisition unit 11 acquires a forward view of a vehicle and parameters of a camera used to shoot the forward view, wherein the forward view at least shows an area in front of the vehicle, and the parameters include external parameters of the camera; the processing unit 12 inputs the forward view and the parameters of the camera into a neural network model to obtain a semantically segmented top-down view. The neural network model can be pre-stored in an image processing device. The neural network model includes a transformer and a semantic segmenter, wherein the transformer is used to transform features in the forward view into features in a top-down view for the vehicle based on the parameters of the camera, and the semantic segmenter is used to perform semantic segmentation on a target based on the features in the top-down view. The output unit 13 outputs the semantically segmented top-down view of the vehicle.

[0053] In one embodiment, the neural network model further comprises a multi-layer perceptron for transforming the parameters of the camera into a camera parameter vector. The transformer transforms the features in the front view into the features in the top-down view for the vehicle based on the camera parameter vector.

[0054] Figure 8 A block diagram of a training device 20 for a neural network model according to an embodiment of the present invention is shown. The training device 20 comprises at least an acquisition unit 21 and a training unit 22. The acquisition unit 21 acquires a training forward view, training parameters of the camera corresponding to the training forward view, and a top-down training true value view of the training forward view for the target after semantic segmentation. The training unit 22 trains the neural network model based on the training forward view, the training parameters of the camera, and the training true value view.

[0055] In one embodiment, the acquisition unit 21 also acquires change data of the external parameters of the camera, and adjusts the training forward view, the training parameters of the camera and the training true value view based on the change data; the training unit 22 trains the neural network model based on the adjusted training forward view, the training parameters and the training true value view.

[0056] In one embodiment, the acquisition unit 21 obtains the change data of the external parameters of the camera as follows: obtains vibration distribution data obtained by performing distribution fitting on vibration data representing the vibration of the camera; selects specific vibration data of the camera based on the vibration distribution data; and determines the change data of the external parameters of the camera based on the specific vibration data of the camera.

[0057] References Figure 7 and 8The structures and functions of the image processing device and the training device are described respectively. This is not restrictive and does not exclude that the image processing device and the training device also include other units. For example, the training device may also include an output unit for outputting the trained neural network model. Furthermore, although the reference method does not describe in detail which units of the image processing device and the training device perform the various steps of the method, it can be expected that any unit of the image processing device and the training device will perform the various steps of the corresponding method, which is not limited here, but covers various situations.

[0058] The functions of the image processing method and device and the neural network training method and device of the present invention can be implemented by software or corresponding hardware, or by means of a processor to implement the functions of the above-mentioned various units. For example, the processor can read the computer program in the memory and run these computer programs to implement the functions of the above-mentioned various units. In one embodiment, the above-mentioned device of the present invention can also be implemented by a memory and a processor.

[0059] It can be understood that the methods of various embodiments of the present disclosure can be implemented by computer programs / software. These software can be loaded into the working memory of the processor and used to execute the methods according to various embodiments of the present disclosure when running.

[0060] It is understood that the method and apparatus according to the present invention can be performed by any device with processing capabilities, and can also be performed at a remote location. The exemplary embodiments of the present disclosure cover both: creating / using the computer program / software of the present disclosure from the beginning, and converting an existing program / software to use the computer program / software of the present disclosure by means of an update.

[0061] According to another embodiment of the present disclosure, a computer program product is provided, such as a machine (e.g., computer) readable medium, such as a CD-ROM, including a computer program code, which, when executed, causes a computer or a processor to perform a method according to each embodiment of the present disclosure. The machine readable medium is, for example, an optical storage medium or a solid-state medium supplied with or as part of other hardware.

[0062] According to another embodiment of the present disclosure, a system for boxing is provided, the system comprising a three-dimensional camera disposed above a boxing container and the device according to various embodiments of the present disclosure.

[0063] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0064] The present disclosure is described above with reference to specific embodiments. It should be understood by those skilled in the art that the technical solution of the present disclosure can be implemented in various ways without departing from the spirit and basic features of the present disclosure. The specific embodiments are merely illustrative and not restrictive. In addition, these embodiments can be arbitrarily combined to achieve the purpose of the present disclosure. The scope of protection of the present disclosure is defined by the attached claims.

[0065] The word "comprising" in the specification and claims does not exclude the existence of other elements or steps. The expressions "first", "second" and the order of the steps shown in the figures do not limit their order or quantity. The functions of the various elements described in the specification or recorded in the claims can also be split or combined and implemented by corresponding multiple elements or a single element.

Claims

1. An image processing method, comprising: Acquire a forward view of a vehicle, wherein the forward view at least shows an area in front of the vehicle; Acquire parameters of a camera used to capture the forward view, wherein the parameters include external parameters of the camera; The front view and the parameters of the camera are input into a neural network model to obtain a semantically segmented top-down view.

2. The image processing method according to claim 1, wherein: The neural network model includes a transformer and a semantic segmentor, wherein the transformer is used to transform features in the front view into features in a top-down view of the vehicle based on parameters of the camera, and the semantic segmentor is used to perform semantic segmentation on the target based on the features in the top-down view.

3. The image processing method according to claim 1, further comprising: The semantically segmented top-down view of the vehicle is output.

4. The image processing method according to claim 1, wherein: The parameters of the camera also include the intrinsic parameters of the camera.

5. The image processing method according to claim 1, wherein: The neural network model also includes a multi-layer perceptron, which is used to transform the parameters of the camera into a camera parameter vector. The transformer transforms features in the front view into features in a top-down view of the vehicle based on the camera parameter vector.

6. The image processing method according to any one of claims 1 to 5, wherein: The neural network model is trained as follows: Obtaining change data of the external parameters of the camera; Acquire a training forward view for training the neural network model, training parameters of the camera corresponding to the training forward view, and a top-down training truth view of the target semantically segmented corresponding to the training forward view; adjusting the training forward view, the training parameters of the camera, and the training ground truth view based on the change data; and The neural network model is trained using the adjusted training forward view, the training parameters of the camera, and the training ground truth view.

7. The image processing method according to claim 6, wherein: The change data of the external parameters of the camera is obtained as follows: acquiring vibration distribution data obtained by performing distribution fitting on vibration data representing vibration of the camera; selecting specific vibration data for the camera based on the vibration distribution data; and The change data of the external parameter of the camera is determined based on the specific vibration data of the camera.

8. A method for training a neural network model, the neural network model receiving a forward view of a vehicle and parameters of a camera for capturing the forward view to output a semantically segmented top-down view of a target, the forward view at least showing an area in front of the vehicle, the parameters including extrinsic parameters of the camera, the training method comprising: Acquire a training forward view, training parameters of the camera corresponding to the training forward view, and a top-down training truth view of the object corresponding to the training forward view and subjected to semantic segmentation; The neural network model is trained based on the training forward view, the training parameters of the camera and the training ground truth view.

9. The training method according to claim 8, wherein: The neural network model includes a transformer and a semantic segmentor, wherein the transformer is used to transform features in the front view into features in a top-down view of the vehicle based on parameters of the camera, and the semantic segmentor is used to perform semantic segmentation on the target based on the features in the top-down view.

10. The training method according to claim 8, further comprising: Obtaining change data of the external parameters of the camera; adjusting the training forward view, the training parameters of the camera, and the training ground truth view based on the change data; and The neural network model is trained using the adjusted training forward view, the training parameters of the camera, and the training ground truth view.

11. The training method according to claim 10, wherein: The change data of the external parameters of the camera is obtained as follows: acquiring vibration distribution data obtained by performing distribution fitting on vibration data representing vibration of the camera; selecting specific vibration data for the camera based on the vibration distribution data; and The change data of the external parameter of the camera is determined based on the specific vibration data of the camera.

12. The training method according to claim 8, wherein: The target includes a free space, and the training truth view is obtained as follows: Overlaying multiple frames of images acquired by a radar sensor to obtain an overlay view including dense point cloud data, wherein the multiple frames of images are acquired for a road; segmenting the overlay view to obtain a road surface view showing a surface of the road; and Markers corresponding to non-free space are incorporated into the road surface view, and the marks corresponding to the non-free space include: marks of foreground objects and / or marks of occlusion areas corresponding to foreground objects, and the foreground objects include non-free space objects incorporated into the road surface view and non-free space objects originally shown in the road surface view.

13. The training method according to claim 12, wherein: The superimposed view in which moving targets are removed is obtained by superimposing a plurality of frames of images acquired by the radar sensor.

14. An image processing device, comprising: an acquisition unit, which acquires a forward view of a vehicle and parameters of a camera used to capture the forward view, wherein the forward view at least shows an area in front of the vehicle, and the parameters include external parameters of the camera; and A processing unit that inputs the front view and the camera parameters into a neural network model to obtain a semantically segmented top-down view.

15. The image processing apparatus according to claim 14, wherein: The neural network model includes a transformer and a semantic segmentor, wherein the transformer is used to transform features in the front view into features in a top-down view of the vehicle based on parameters of the camera, and the semantic segmentor is used to perform semantic segmentation on the target based on the features in the top-down view.

16. The image processing device according to claim 14, further comprising: An output unit outputs the semantically segmented top-down view of the vehicle.

17. The image processing apparatus according to claim 14, wherein: The neural network model also includes a multi-layer perceptron, which is used to transform the parameters of the camera into a camera parameter vector. The transformer transforms features in the front view into features in a top-down view of the vehicle based on the camera parameter vector.

18. A training device for a neural network model, the neural network model receiving a forward view of a vehicle and parameters of a camera for capturing the forward view to output a semantically segmented top-down view of an object, the forward view at least showing an area in front of the vehicle, the parameters including extrinsic parameters of the camera, the training device comprising: An acquisition unit, configured to acquire a training forward view, training parameters of the camera corresponding to the training forward view, and a top-down training truth view of the object corresponding to the training forward view and subjected to semantic segmentation; and A training unit is configured to train the neural network model based on the training forward view, the training parameters of the camera and the training ground-truth view.

19. The training device according to claim 18, wherein: The neural network model includes a transformer and a semantic segmentor, wherein the transformer is used to transform features in the front view into features in a top-down view of the vehicle based on parameters of the camera, and the semantic segmentor is used to perform semantic segmentation on the target based on the features in the top-down view.

20. The training device according to claim 18, wherein The acquisition unit further acquires change data of the external parameters of the camera, and adjusts the training forward view, the training parameters of the camera and the training true value view based on the change data; The training unit trains the neural network model using the adjusted training forward view, the training parameters of the camera, and the training ground truth view.

21. The training device according to claim 20, wherein: The acquisition unit obtains the change data of the external parameters of the camera as follows: acquiring vibration distribution data obtained by performing distribution fitting on vibration data representing vibration of the camera; selecting specific vibration data for the camera based on the vibration distribution data; and The change data of the external parameter of the camera is determined based on the specific vibration data of the camera.

22. A computer-readable medium storing a computer program unit, which, when executed by a processor or a computer, causes the processor or the computer to execute the image processing method according to any one of claims 1 to 7 or the training method for a neural network model according to any one of claims 8 to 13.

Images