Self-adaptive high beam visual detection method, device and equipment, medium and vehicle

Abstract

The invention discloses a self-adaptive high beam visual detection method. The method comprises the following steps: acquiring a night forward image acquired by a forward-looking camera of a target automatic driving vehicle in real time; processing the night forward image based on a pre-trained lightweight open source neural network model to obtain the category and position of each target in the night forward image; for each target, the position of the target is converted into a world coordinate system according to an internal reference matrix and an external reference matrix of a foresight camera to obtain a corresponding world coordinate position, and the world coordinate position is converted into an adaptive high beam coordinate system according to a light source projection matrix of an adaptive high beam to obtain a corresponding high beam sub-matrix; and controlling light emitting diodes (LEDs) corresponding to the high beam sub-matrix according to a control mode corresponding to the category of the target. By applying the scheme provided by the invention, the generalization of position detection can be improved, automatic iteration can be realized through data updating, the iteration cost is reduced, and the accuracy of high beam control is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to an adaptive high beam visual detection method, device, equipment, medium and vehicle. Background Art

[0002] When driving at night, the adaptive high beam system in the autonomous driving vehicle calculates in real time based on the position of the vehicle in front and controls the corresponding LED in the high beam to turn off, avoiding dazzling effects on the vehicle in front.

[0003] Among the currently known methods, the position of the vehicle in front is mainly determined by manually pre-setting vehicle light source recognition rules. For example, the light source point in the image is located by a light source detection method based on image brightness recognition, and the position of the vehicle in front is located according to the light source point. Then, according to the position of the vehicle in front, all LEDs in a certain overall area of ​​the high beam that may illuminate the vehicle in front are turned off.

[0004] However, the above-mentioned manual rule-setting method mostly relies on engineers to design based on image principles and mathematical analysis, which is prone to missed detection and false detection, poor generalization, and too high manpower and time costs for updating and iteration. In addition, when controlling the high beam, all LEDs in a certain overall area are turned off, but some LEDs in the overall area will not actually illuminate the vehicle in front, resulting in inaccurate control of the high beam. Summary of the invention

[0005] The present invention provides an adaptive high beam vision detection method, device, equipment, medium and vehicle, which can improve the generalization of position detection, reduce iteration cost and improve the accuracy of high beam control. The specific technical solution is as follows.

[0006] In a first aspect, an embodiment of the present invention provides an adaptive high beam visual detection method, the method comprising:

[0007] Obtaining a nighttime forward image collected in real time by a forward-looking camera of a target autonomous driving vehicle;

[0008] The nighttime forward image is processed based on a pre-trained lightweight open source neural network model to obtain the category and position of each target in the nighttime forward image, wherein the lightweight open source neural network model is a neural network model with a number of convolutional layers not greater than a preset number of layers or a parameter amount not greater than a preset parameter amount, and is used to associate the nighttime forward sample image with the category and position of the corresponding target;

[0009] For each target, the position of the target is converted into the world coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the front-view camera to obtain the corresponding world coordinate position, and the world coordinate position is converted into the adaptive high beam coordinate system according to the light source projection matrix of the adaptive high beam to obtain the corresponding high beam sub-matrix, and the light-emitting diodes LED corresponding to the high beam sub-matrix are controlled according to the control method corresponding to the category of the target, wherein the light source projection matrix represents the projection relationship between the world coordinate system and the adaptive high beam coordinate system.

[0010] Optionally, the training process of the lightweight open source neural network model is:

[0011] Build an initial open source neural network model;

[0012] Pruning the initial open source neural network model multiple times according to a preset model pruning method to obtain multiple pruned models of different sizes;

[0013] Analyze the multiple pruned models respectively according to a preset sensitivity analysis method to obtain corresponding sensitivity values;

[0014] The pruned model with the lowest sensitivity value is used as the reference open source neural network model;

[0015] Acquire nighttime forward sample images of the target autonomous driving vehicle and labeling results of each of the nighttime forward sample images, wherein the labeling results include the category and position of each target contained in each of the nighttime forward sample images;

[0016] Processing each labeled nighttime forward sample image based on the reference open source neural network model to obtain a nighttime target detection result corresponding to each of the nighttime forward sample images, and determining whether the accuracy of the reference open source neural network model is greater than a preset threshold according to the nighttime target detection result and the corresponding labeling result;

[0017] When the accuracy of the reference open source neural network model is not greater than the preset threshold, adjusting the parameters of the reference open source neural network model, and returning to execute the step of processing each labeled nighttime forward sample image based on the reference open source neural network model;

[0018] When the accuracy of the reference open source neural network model is greater than the preset threshold, the training is completed to obtain a lightweight open source neural network model that associates the nighttime forward sample image with the category and position of the corresponding target.

[0019] Optionally, the number of the multiple pruned models is 3, and the size of each pruned model is 0.25 times, 0.5 times and 0.75 times the size of the initial open source neural network model, respectively.

[0020] Optionally, the initial open source neural network model is a deep residual network Resnet-18 model or a mobilenetv2 model.

[0021] Optionally, the step of converting the position of the target into a world coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the forward-looking camera to obtain a corresponding world coordinate position includes:

[0022] Determine a first projection relationship between an image coordinate system and a camera coordinate system of the forward-looking camera according to an intrinsic parameter matrix of the forward-looking camera, and convert the position of the target into the camera coordinate system according to the first projection relationship to obtain a corresponding camera coordinate position;

[0023] A second projection relationship between the camera coordinate system of the forward-looking camera and the world coordinate system is determined according to the external parameter matrix of the forward-looking camera, and the camera coordinate position is converted into the world coordinate system according to the second projection relationship to obtain a corresponding world coordinate position.

[0024] Optionally, the step of controlling the light emitting diodes LED corresponding to the high beam sub-matrix according to the control mode corresponding to the category of the target includes:

[0025] When the category of the target is a car, the light emitting diode LED corresponding to the high beam sub-matrix is ​​controlled to turn off;

[0026] When the category of the target is a motorcycle, a bicycle or a pedestrian, the light emitting diode LED corresponding to the high beam sub-matrix is ​​controlled to flash.

[0027] In a second aspect, an embodiment of the present invention provides an adaptive high beam vision detection device, the device comprising:

[0028] An acquisition module, used to acquire a nighttime forward image collected in real time by a forward-looking camera of a target autonomous driving vehicle;

[0029] A detection module, used to process the nighttime forward image based on a pre-trained lightweight open source neural network model to obtain the category and position of each target in the nighttime forward image, wherein the lightweight open source neural network model is a neural network model with a number of convolutional layers not greater than a preset number of layers or a parameter amount not greater than a preset parameter amount, and is used to associate the nighttime forward sample image with the category and position of the corresponding target;

[0030] A control module is used to convert the position of each target into a world coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the front-view camera to obtain a corresponding world coordinate position, convert the world coordinate position into the adaptive high beam coordinate system according to the light source projection matrix of the adaptive high beam to obtain a corresponding high beam sub-matrix, and control the light-emitting diodes LED corresponding to the high beam sub-matrix according to the control method corresponding to the category of the target, wherein the light source projection matrix represents the projection relationship between the world coordinate system and the adaptive high beam coordinate system.

[0031] Optionally, the adaptive high beam vision detection device further includes a model training module, which is used to train the lightweight open source neural network model. The model training module includes:

[0032] Build submodules for building initial open source neural network models;

[0033] A pruning submodule, used to prune the initial open source neural network model multiple times according to a preset model pruning method to obtain multiple pruned models of different sizes;

[0034] A sensitivity analysis submodule, used to analyze the plurality of pruned models respectively according to a preset sensitivity analysis method to obtain corresponding sensitivity values;

[0035] The submodule is determined by referring to the open source neural network model, and is used to use the pruned model with the lowest sensitivity value as the reference open source neural network model;

[0036] An acquisition submodule, used to acquire nighttime forward sample images of the target autonomous driving vehicle and labeling results of each of the nighttime forward sample images, wherein the labeling results include the category and position of each target contained in each nighttime forward sample image;

[0037] A processing submodule, configured to process each labeled nighttime forward sample image based on the reference open source neural network model, obtain a nighttime target detection result corresponding to each of the nighttime forward sample images, and determine whether the accuracy of the reference open source neural network model is greater than a preset threshold value according to the nighttime target detection result and the corresponding labeling result;

[0038] An adjustment submodule, configured to adjust the parameters of the reference open source neural network model and trigger the processing submodule when the accuracy of the reference open source neural network model is not greater than the preset threshold;

[0039] The training completion submodule is used to complete the training when the accuracy of the reference open source neural network model is greater than the preset threshold, so as to obtain a lightweight open source neural network model that associates the nighttime forward sample image with the category and position of the corresponding target.

[0040] Optionally, the number of the multiple pruned models is 3, and the size of each pruned model is 0.25 times, 0.5 times and 0.75 times the size of the initial open source neural network model, respectively.

[0041] Optionally, the initial open source neural network model is a deep residual network Resnet-18 model or a mobilenetv2 model.

[0042] Optionally, the control module includes:

[0043] A first conversion submodule, used to determine a first projection relationship between an image coordinate system and a camera coordinate system of the forward-looking camera according to an intrinsic parameter matrix of the forward-looking camera, and to convert the position of the target into the camera coordinate system according to the first projection relationship to obtain a corresponding camera coordinate position;

[0044] The second conversion submodule is used to determine the second projection relationship between the camera coordinate system of the forward-looking camera and the world coordinate system according to the external parameter matrix of the forward-looking camera, and convert the camera coordinate position into the world coordinate system according to the second projection relationship to obtain the corresponding world coordinate position.

[0045] Optionally, the control module is specifically used to:

[0046] When the category of the target is a car, the light emitting diode LED corresponding to the high beam sub-matrix is ​​controlled to turn off;

[0047] When the category of the target is a motorcycle, a bicycle or a pedestrian, the light emitting diode LED corresponding to the high beam sub-matrix is ​​controlled to flash.

[0048] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the adaptive high beam visual detection method provided in any embodiment of the present invention is implemented.

[0049] In a fourth aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0050] one or more processors;

[0051] The processor is coupled to a storage device, and the storage device is used to store one or more programs;

[0052] When the one or more programs are executed by the one or more processors, the electronic device implements the adaptive high beam visual detection method provided by any embodiment of the present invention.

[0053] In a fifth aspect, an embodiment of the present invention provides a vehicle, wherein the vehicle comprises an adaptive high beam visual detection method as provided in any embodiment of the present invention, or comprises an electronic device as provided in any embodiment of the present invention.

[0054] It can be seen from the above content that the adaptive high beam visual detection method, device, equipment, medium and vehicle provided by the embodiments of the present invention can obtain the night-time forward image collected in real time by the forward-looking camera of the target autonomous driving vehicle; the night-time forward image is processed based on the pre-trained lightweight open source neural network model to obtain the category and position of each target in the night-time forward image, wherein the lightweight open source neural network model is a neural network model with a convolution layer number not greater than a preset layer number or a parameter amount not greater than a preset parameter amount, which is used to associate the night-time forward sample image with the category and position of the corresponding target; for each target, the position of the target is converted to the world coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the forward-looking camera to obtain the corresponding world coordinate position, and the world coordinate position is converted to the adaptive high beam coordinate system according to the light source projection matrix of the adaptive high beam to obtain the corresponding high beam sub-matrix, and the light-emitting diode LED corresponding to the high beam sub-matrix is ​​controlled according to the control method corresponding to the category of the target, wherein the light source projection matrix represents the projection relationship between the world coordinate system and the adaptive high beam coordinate system. In the present invention, the position of the target is determined based on a pre-trained lightweight open source neural network model. Compared with the method of manually setting rules, the occurrence of missed detection and false detection is reduced, the generalization of position detection is improved, and automatic iteration can be achieved through data update, reducing the iteration cost. When controlling the high beam, only the LED corresponding to the high beam sub-matrix corresponding to the target is controlled, rather than controlling all LEDs in an overall area, which improves the accuracy of high beam control. Of course, any product or method implementing the present invention does not necessarily need to achieve all the advantages described above at the same time.

[0055] The innovative features of the embodiments of the present invention include:

[0056] 1. In the present invention, the position of the target is determined based on a lightweight open source neural network model obtained by pre-training. Compared with the method of manually setting rules, the occurrence of missed detection and false detection is reduced, the generalization of position detection is improved, and automatic iteration can be achieved through data update, which reduces the iteration cost. When controlling the high beam, only the LED corresponding to the high beam sub-matrix corresponding to the target is controlled, rather than controlling all LEDs in an overall area, which improves the accuracy of high beam control.

[0057] 2. Since the lightweight open source neural network model undergoes pruning and sensitivity analysis during training, it can identify the category and location of the target with fewer parameters and calculations without losing accuracy.

[0058] 3. First, prune the initial open source neural network model through a preset model pruning method, and then perform sensitivity analysis to select the reference open source neural network model with the best performance. Then, train the reference open source neural network model to obtain a lightweight open source neural network model that is used to associate the night forward sample image with the corresponding target category and position.

[0059] 4. Controlling the high beam based on the target category can achieve different high beam controls for different types of targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work.

[0061] Figure 1 A schematic diagram of a flow chart of an adaptive high beam vision detection method provided by an embodiment of the present invention;

[0062] Figure 2 A schematic diagram of a structure of an adaptive high beam vision detection device provided by an embodiment of the present invention;

[0063] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0064] Figure 4 A schematic structural diagram of a vehicle provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0067] The embodiments of the present invention disclose an adaptive high beam vision detection method, device, equipment, medium and vehicle, which can improve the generalization of position detection, reduce iteration cost and improve the accuracy of high beam control. The embodiments of the present invention are described in detail below.

[0068] Figure 1 A schematic flow chart of a method for visually detecting an adaptive high beam provided by an embodiment of the present invention. The method is applied to electronic equipment. The method specifically includes the following steps.

[0069] S110: Acquire a nighttime forward image collected in real time by a forward-looking camera of a target autonomous driving vehicle.

[0070] In order to control the adaptive high beam of the target autonomous driving vehicle, it is necessary to obtain the nighttime forward image collected in real time by the forward-looking camera of the target autonomous driving vehicle.

[0071] The night-time forward image is an image taken by the vehicle's forward-looking camera at night, and the target autonomous driving vehicle is any autonomous driving vehicle that requires adaptive high beam control.

[0072] S120: Processing the night forward image based on a pre-trained lightweight open source neural network model to obtain the category and position of each target in the night forward image, wherein the lightweight open source neural network model is a neural network model with a number of convolutional layers not greater than a preset number of layers or a parameter amount not greater than a preset parameter amount, and is used to associate the night forward sample image with the category and position of the corresponding target.

[0073] Since the purpose of controlling the adaptive high beam of the target autonomous driving vehicle is to avoid direct illumination of various targets on the road, such as vehicles or pedestrians, which may cause dizziness, target detection is required after obtaining the night-time forward image collected in real time by the forward-looking camera of the target autonomous driving vehicle.

[0074] Specifically, the target detection method is to process the night forward image based on the pre-trained lightweight open source neural network model to obtain the category and position of each target in the night forward image, wherein the lightweight open source neural network model is a neural network model with a convolution layer number not greater than a preset layer number or a parameter amount not greater than a preset parameter amount, which is used to associate the night forward sample image with the category and position of the corresponding target. Exemplarily, the preset number of layers can be 15 layers, and the preset parameter amount can be 450,000.

[0075] Among them, the training process of the lightweight open source neural network model is:

[0076] Build an initial open source neural network model;

[0077] The initial open source neural network model is pruned multiple times according to the preset model pruning method to obtain multiple pruned models of different sizes;

[0078] According to the preset sensitivity analysis method, multiple pruned models are analyzed respectively to obtain corresponding sensitivity values;

[0079] The pruned model with the lowest sensitivity value is used as the reference open source neural network model;

[0080] Obtaining nighttime forward sample images of the target autonomous driving vehicle and labeling results of each nighttime forward sample image, wherein the labeling results include the category and position of each target contained in each nighttime forward sample image;

[0081] Based on the reference open source neural network model, each labeled nighttime forward sample image is processed to obtain a nighttime target detection result corresponding to each nighttime forward sample image, and whether the accuracy of the reference open source neural network model is greater than a preset threshold is determined based on the nighttime target detection result and the corresponding labeling result;

[0082] When the accuracy of the reference open source neural network model is not greater than a preset threshold, the parameters of the reference open source neural network model are adjusted according to a preset normalized exponential function, and the step of processing each labeled nighttime forward sample image based on the reference open source neural network model is returned to be executed;

[0083] When the accuracy of the reference open source neural network model is greater than a preset threshold, the training is completed to obtain a lightweight open source neural network model that associates the nighttime forward sample image with the category and position of the corresponding target.

[0084] It is understandable that first an initial open source neural network model needs to be built, and then trained to obtain a lightweight open source neural network model.

[0085] If the size of the neural network model is large, the running speed is slow. Therefore, in order to improve the running speed, the neural network model can be pruned. Specifically, the initial open source neural network model is pruned multiple times according to a preset model pruning method to obtain multiple pruned models of different sizes. The preset model pruning method can be any model pruning method in the prior art, and the embodiments of the present invention do not impose any limitation on this.

[0086] The number of pruning times can be specifically set based on the accuracy requirements of the model, and the embodiment of the present invention does not limit its specific value. Exemplarily, the number of multiple pruned models can be 3, and the size of each pruned model is 0.25 times, 0.5 times, and 0.75 times the size of the initial open source neural network model.

[0087] After obtaining multiple pruned models, it is necessary to select an optimal model with the best performance and the fastest running speed. Specifically, the multiple pruned models are analyzed respectively according to a preset sensitivity analysis method to obtain corresponding sensitivity values, wherein the preset sensitivity analysis method can be any sensitivity analysis method in the prior art, and the embodiments of the present invention do not impose any limitation on this.

[0088] Since the lower the sensitivity value, the more stable the performance, that is, the better the performance and the faster the running speed, that is to say, the pruned model with the lowest sensitivity value is the optimal model. Therefore, after obtaining the sensitivity value, the pruned model with the lowest sensitivity value is used as the reference open source neural network model, and then it is trained to obtain a lightweight open source neural network model.

[0089] Specifically, a nighttime forward sample image of a target autonomous driving vehicle and a labeling result of each nighttime forward sample image are obtained, wherein the labeling result includes the category and position of each target contained in each nighttime forward sample image, and each labeled nighttime forward sample image is processed based on a reference open source neural network model to obtain a nighttime target detection result corresponding to each nighttime forward sample image, and whether the accuracy of the reference open source neural network model is greater than a preset threshold value is determined based on the nighttime target detection result and the corresponding labeling result; when the accuracy of the reference open source neural network model is not greater than the preset threshold value, the parameters of the reference open source neural network model are adjusted, and the step of processing each labeled nighttime forward sample image based on the reference open source neural network model is returned to execute; when the accuracy of the reference open source neural network model is greater than the preset threshold value, the training is completed to obtain a lightweight open source neural network model that associates the nighttime forward sample image with the category and position of the corresponding target.

[0090] In other words, the parameters of the reference open source neural network model can be adjusted multiple times based on the night-time forward sample images to ultimately obtain a lightweight open source neural network model that meets the accuracy requirements.

[0091] The initial open source neural network model may be a deep residual network Resnet-18 model or a mobilenetv2 model. The labeling process of the nighttime forward sample images may be performed manually. The preset threshold may be specifically set according to the number of nighttime forward sample images or the accuracy requirements of the model, and the embodiment of the present invention does not limit its specific value.

[0092] In one implementation, the model parameters can be adjusted based on the following loss function:

[0093]

[0094] Among them, q is the loss value, z i is the probability value of the first category, z j is the probability value of the second category, i is the first category, j is the second category, and T is the temperature coefficient.

[0095] Where T is the temperature coefficient. When T=1, the above formula is the standard Softmax function formula. The higher T is, the smoother the output probability distribution of the above formula is, the greater the entropy of the loss value distribution is, the information carried by the negative label will be relatively amplified, and the model training will pay more attention to the negative label, that is, pay more attention to the image data that is not the target category. For example, T=0.5.

[0096] Therefore, the initial open source neural network model is first pruned using a preset model pruning method, and then a sensitivity analysis is performed to select the reference open source neural network model with the best performance. Then, the reference open source neural network model is trained to obtain a lightweight open source neural network model that is used to associate night-time forward sample images with the corresponding target categories and positions.

[0097] S130: For each target, the position of the target is converted into the world coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the front-view camera to obtain the corresponding world coordinate position, and the world coordinate position is converted into the adaptive high beam coordinate system according to the light source projection matrix of the adaptive high beam to obtain the corresponding high beam sub-matrix, and the light-emitting diodes LED corresponding to the high beam sub-matrix are controlled according to the control method corresponding to the category of the target, wherein the light source projection matrix represents the projection relationship between the world coordinate system and the adaptive high beam coordinate system.

[0098] After obtaining the category and position of each target, it is necessary to control the adaptive high beam based on the category and position of each target. Specifically, the position of the target is converted to the world coordinate system according to the intrinsic parameter matrix and extrinsic parameter matrix of the front-view camera to obtain the corresponding world coordinate position, and the world coordinate position is converted to the adaptive high beam coordinate system according to the light source projection matrix of the adaptive high beam to obtain the corresponding high beam sub-matrix, and the light-emitting diodes LED corresponding to the high beam sub-matrix are controlled according to the control method corresponding to the category of the target, wherein the light source projection matrix represents the projection relationship between the world coordinate system and the adaptive high beam coordinate system.

[0099] The step of converting the position of the target into the world coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the front-view camera to obtain the corresponding world coordinate position may include:

[0100] Determine a first projection relationship between the image coordinate system and the camera coordinate system of the forward-looking camera according to the intrinsic parameter matrix of the forward-looking camera, and convert the position of the target into the camera coordinate system according to the first projection relationship to obtain a corresponding camera coordinate position;

[0101] The second projection relationship between the camera coordinate system of the front-view camera and the world coordinate system is determined according to the external parameter matrix of the front-view camera, and the camera coordinate position is converted into the world coordinate system according to the second projection relationship to obtain the corresponding world coordinate position.

[0102] Since the image coordinate system, camera coordinate system and world coordinate system can be converted into each other, the basis is the camera's intrinsic parameter matrix and extrinsic parameter matrix. Therefore, if the intrinsic parameter matrix and extrinsic parameter matrix of the front-view camera are known, the position of the target can be converted into the world coordinate system to obtain the corresponding world coordinate position.

[0103] Specifically, the position of the target is first converted from the image coordinate system to the camera coordinate system, that is, the first projection relationship between the image coordinate system and the camera coordinate system of the front camera is determined according to the intrinsic parameter matrix of the front camera, and the position of the target is converted to the camera coordinate system according to the first projection relationship to obtain the corresponding camera coordinate position.

[0104] Then the camera coordinate position is converted from the camera coordinate system to the world coordinate system, that is, the second projection relationship between the camera coordinate system of the front-view camera and the world coordinate system is determined according to the external parameter matrix of the front-view camera, and the camera coordinate position is converted to the world coordinate system according to the second projection relationship to obtain the corresponding world coordinate position.

[0105] Among them, the camera coordinate system is used to represent the position of the object in three-dimensional space, and the image coordinate system is used to represent the pixel position of the object in the two-dimensional image. The function of the camera's intrinsic parameter matrix is ​​to make linear changes between these two coordinate systems.

[0106] The camera's extrinsic parameter matrix is ​​used to describe the camera's motion in a static scene. The camera's extrinsic parameters may include one or more of the following parameters: the camera's optical center coordinates, the optical axis direction, and the camera's posture information.

[0107] Since the light source projection matrix of the adaptive high beam represents the projection relationship between the world coordinate system and the adaptive high beam coordinate system, after obtaining the world coordinate position, the world coordinate position can be converted into the adaptive high beam coordinate system according to the light source projection matrix of the adaptive high beam to obtain the corresponding high beam sub-matrix.

[0108] The high beam sub-matrix is ​​a part of the high beam matrix. The high beam matrix is ​​a matrix arranged in a rectangular array, and each position of the matrix corresponds to an LED (Light Emitting Diode).

[0109] Since different types of targets may correspond to different control methods, for each target, it is necessary to control the light emitting diode LED corresponding to the high beam sub-matrix according to the control method corresponding to the type of the target.

[0110] The step of controlling the light emitting diodes LED corresponding to the high beam sub-matrix according to the control method corresponding to the category of the target may include:

[0111] When the category of the target is a car, the light emitting diode LED corresponding to the high beam sub-matrix is ​​controlled to turn off;

[0112] When the category of the target is a motorcycle, a bicycle or a pedestrian, the light emitting diode LED corresponding to the high beam sub-matrix is ​​controlled to flash.

[0113] For cars, in order to avoid dizziness of the driver caused by high beam, the light-emitting diodes (LED) shining on the car position need to be turned off, that is, when the category of the target is a car, the light-emitting diodes (LED) corresponding to the high beam sub-matrix are turned off; for motorcycles, bicycles or pedestrians, the light-emitting diodes (LED) corresponding to the high beam sub-matrix can be controlled to flash, so as to remind motorcycles, bicycles or pedestrians to dodge.

[0114] Therefore, by controlling the high beam based on the category of the target, different high beam controls can be performed on different types of targets.

[0115] It can be seen from the above content that this embodiment can obtain the nighttime forward image collected in real time by the forward-looking camera of the target autonomous driving vehicle; the nighttime forward image is processed based on the pre-trained lightweight open source neural network model to obtain the category and position of each target in the nighttime forward image, wherein the lightweight open source neural network model is a neural network model with a convolution layer number not greater than a preset layer number or a parameter amount not greater than a preset parameter amount, which is used to associate the nighttime forward sample image with the category and position of the corresponding target; for each target, the position of the target is converted to the world coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the forward-looking camera to obtain the corresponding world coordinate position, and the world coordinate position is converted to the adaptive high beam coordinate system according to the light source projection matrix of the adaptive high beam to obtain the corresponding high beam sub-matrix, and the light-emitting diode LED corresponding to the high beam sub-matrix is ​​controlled according to the control method corresponding to the category of the target, wherein the light source projection matrix represents the projection relationship between the world coordinate system and the adaptive high beam coordinate system. In the present invention, the position of the target is judged based on a lightweight open source neural network model obtained by pre-training. Compared with the method of manually setting rules, the occurrence of missed detection and false detection is reduced, the generalization of position detection is improved, and automatic iteration can be achieved through data update, which reduces the iteration cost. When the high beam is controlled, only the LED corresponding to the high beam sub-matrix corresponding to the target is controlled, rather than controlling all LEDs in an overall area, which improves the accuracy of high beam control.

[0116] In addition, because the lightweight open source neural network model undergoes pruning and sensitivity analysis during training, it can identify the category and location of the target with fewer parameters and computation without losing accuracy.

[0117] Figure 2 An adaptive high beam vision detection device provided by an embodiment of the present invention, see Figure 2 , the device comprises:

[0118] An acquisition module 210 is used to acquire a nighttime forward image collected in real time by a forward-looking camera of a target autonomous driving vehicle;

[0119] A detection module 220 is used to process the nighttime forward image based on a pre-trained lightweight open source neural network model to obtain the category and position of each target in the nighttime forward image, wherein the lightweight open source neural network model is a neural network model with a convolution layer number not greater than a preset layer number or a parameter amount not greater than a preset parameter amount, and is used to associate the nighttime forward sample image with the category and position of the corresponding target;

[0120] The control module 230 is used to convert the position of each target into a world coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the front-view camera to obtain a corresponding world coordinate position, convert the world coordinate position into the adaptive high beam coordinate system according to the light source projection matrix of the adaptive high beam to obtain a corresponding high beam sub-matrix, and control the light-emitting diodes LED corresponding to the high beam sub-matrix according to the control method corresponding to the category of the target, wherein the light source projection matrix represents the projection relationship between the world coordinate system and the adaptive high beam coordinate system.

[0121] The device provided by the embodiment of the present invention can obtain a nighttime forward image collected in real time by a forward-looking camera of a target autonomous driving vehicle; the nighttime forward image is processed based on a pre-trained lightweight open source neural network model to obtain the category and position of each target in the nighttime forward image, wherein the lightweight open source neural network model is a neural network model whose number of convolution layers is not greater than a preset number of layers or whose parameter amount is not greater than a preset parameter amount, and is used to associate the nighttime forward sample image with the category and position of the corresponding target; for each target, the position of the target is converted into a world coordinate system according to an intrinsic parameter matrix and an extrinsic parameter matrix of the forward-looking camera to obtain a corresponding world coordinate position, the world coordinate position is converted into an adaptive high beam coordinate system according to a light source projection matrix of the adaptive high beam to obtain a corresponding high beam sub-matrix, and the light-emitting diode LED corresponding to the high beam sub-matrix is ​​controlled according to a control method corresponding to the category of the target, wherein the light source projection matrix represents the projection relationship between the world coordinate system and the adaptive high beam coordinate system. In the present invention, the position of the target is judged based on a lightweight open source neural network model obtained by pre-training. Compared with the method of manually setting rules, the occurrence of missed detection and false detection is reduced, the generalization of position detection is improved, and automatic iteration can be achieved through data update, which reduces the iteration cost. When the high beam is controlled, only the LED corresponding to the high beam sub-matrix corresponding to the target is controlled, rather than controlling all LEDs in an overall area, which improves the accuracy of high beam control.

[0122] In one implementation, the adaptive high beam vision detection device further includes a model training module, which is used to train the lightweight open source neural network model. The model training module includes:

[0123] Build submodules for building initial open source neural network models;

[0124] A pruning submodule, used to prune the initial open source neural network model multiple times according to a preset model pruning method to obtain multiple pruned models of different sizes;

[0125] A sensitivity analysis submodule, used to analyze the plurality of pruned models respectively according to a preset sensitivity analysis method to obtain corresponding sensitivity values;

[0126] The submodule is determined by referring to the open source neural network model, and is used to use the pruned model with the lowest sensitivity value as the reference open source neural network model;

[0127] An acquisition submodule, used to acquire nighttime forward sample images of the target autonomous driving vehicle and labeling results of each of the nighttime forward sample images, wherein the labeling results include the category and position of each target contained in each nighttime forward sample image;

[0128] A processing submodule, configured to process each labeled nighttime forward sample image based on the reference open source neural network model, obtain a nighttime target detection result corresponding to each of the nighttime forward sample images, and determine whether the accuracy of the reference open source neural network model is greater than a preset threshold value according to the nighttime target detection result and the corresponding labeling result;

[0129] An adjustment submodule, configured to adjust the parameters of the reference open source neural network model and trigger the processing submodule when the accuracy of the reference open source neural network model is not greater than the preset threshold;

[0130] The training completion submodule is used to complete the training when the accuracy of the reference open source neural network model is greater than the preset threshold, so as to obtain a lightweight open source neural network model that associates the nighttime forward sample image with the category and position of the corresponding target.

[0131] In one implementation, the number of the multiple pruned models is 3, and the size of each pruned model is 0.25 times, 0.5 times, and 0.75 times the size of the initial open source neural network model, respectively.

[0132] In one implementation, the initial open source neural network model is a deep residual network Resnet-18 model or a mobilenetv2 model.

[0133] In one implementation, the control module 230 may include:

[0134] A first conversion submodule, used to determine a first projection relationship between an image coordinate system and a camera coordinate system of the forward-looking camera according to an intrinsic parameter matrix of the forward-looking camera, and to convert the position of the target into the camera coordinate system according to the first projection relationship to obtain a corresponding camera coordinate position;

[0135] The second conversion submodule is used to determine the second projection relationship between the camera coordinate system of the forward-looking camera and the world coordinate system according to the external parameter matrix of the forward-looking camera, and convert the camera coordinate position into the world coordinate system according to the second projection relationship to obtain the corresponding world coordinate position.

[0136] In one implementation, the control module 230 may be specifically configured to:

[0137] When the category of the target is a car, the light emitting diode LED corresponding to the high beam sub-matrix is ​​controlled to turn off;

[0138] When the category of the target is a motorcycle, a bicycle or a pedestrian, the light emitting diode LED corresponding to the high beam sub-matrix is ​​controlled to flash.

[0139] The above device embodiment corresponds to the method embodiment and has the same technical effect as the method embodiment. For specific description, please refer to the method embodiment. The device embodiment is obtained based on the method embodiment. For specific description, please refer to the method embodiment part, which will not be repeated here.

[0140] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the electronic device includes:

[0141] One or more processors 310;

[0142] The processor 310 is coupled to a storage device 320, and the storage device 320 is used to store one or more programs;

[0143] When the one or more programs are executed by the one or more processors 310, the electronic device implements the adaptive high beam vision detection method provided by any embodiment of the present invention.

[0144] Based on the above embodiments, another embodiment of the present invention provides a vehicle, wherein the vehicle includes the adaptive high beam vision detection device provided by any embodiment of the present invention, or includes the electronic device provided by any embodiment of the present invention.

[0145] Figure 4 A schematic diagram of a vehicle provided by an embodiment of the present invention. Figure 4As shown, the vehicle includes a speed sensor 41, an ECU (Electronic Control Unit) 42, a GPS (Global Positioning System) positioning device 43, and a T-Box (Telematics Box) 44. The speed sensor 41 is used to measure the vehicle speed and use the vehicle speed as an empirical speed for model training; the GPS positioning device 43 is used to obtain the current geographical location of the vehicle; the T-Box 64 can communicate with the server as a gateway; and the ECU 42 can execute the above-mentioned adaptive high beam visual detection method.

[0146] In addition, the vehicle may also include: a V2X (Vehicle-to-Everything) module 45, a radar 46 and a camera 47. The V2X module 45 is used to communicate with other vehicles, roadside equipment, etc.; the radar 46 or the camera 47 is used to sense the road environment information in the front and / or other directions to obtain original point cloud data; the radar 46 and / or the camera 47 can be configured at the front and / or rear of the vehicle body.

[0147] Based on the above method embodiments, another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the adaptive high beam vision detection method provided by any embodiment of the present invention is implemented.

[0148] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0149] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for visual detection of an adaptive high beam, characterized in that: The method comprises: Obtaining a nighttime forward image collected in real time by a forward-looking camera of a target autonomous driving vehicle; The nighttime forward image is processed based on a pre-trained lightweight open source neural network model to obtain the category and position of each target in the nighttime forward image, wherein the lightweight open source neural network model is a neural network model with a number of convolutional layers not greater than a preset number of layers or a parameter amount not greater than a preset parameter amount, and is used to associate the nighttime forward sample image with the category and position of the corresponding target; For each target, the position of the target is converted into the world coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the front-view camera to obtain the corresponding world coordinate position, and the world coordinate position is converted into the adaptive high beam coordinate system according to the light source projection matrix of the adaptive high beam to obtain the corresponding high beam sub-matrix, and the light-emitting diodes LED corresponding to the high beam sub-matrix are controlled according to the control method corresponding to the category of the target, wherein the light source projection matrix represents the projection relationship between the world coordinate system and the adaptive high beam coordinate system.

2. The method according to claim 1, characterized in that The training process of the lightweight open source neural network model is as follows: Build an initial open source neural network model; Pruning the initial open source neural network model multiple times according to a preset model pruning method to obtain multiple pruned models of different sizes; Analyze the multiple pruned models respectively according to a preset sensitivity analysis method to obtain corresponding sensitivity values; The pruned model with the lowest sensitivity value is used as the reference open source neural network model; Acquire nighttime forward sample images of the target autonomous driving vehicle and labeling results of each of the nighttime forward sample images, wherein the labeling results include the category and position of each target contained in each of the nighttime forward sample images; Processing each labeled nighttime forward sample image based on the reference open source neural network model to obtain a nighttime target detection result corresponding to each of the nighttime forward sample images, and determining whether the accuracy of the reference open source neural network model is greater than a preset threshold according to the nighttime target detection result and the corresponding labeling result; When the accuracy of the reference open source neural network model is not greater than the preset threshold, adjusting the parameters of the reference open source neural network model, and returning to execute the step of processing each labeled nighttime forward sample image based on the reference open source neural network model; When the accuracy of the reference open source neural network model is greater than the preset threshold, the training is completed to obtain a lightweight open source neural network model that associates the nighttime forward sample image with the category and position of the corresponding target.

3. The method according to claim 2, characterized in that The number of the multiple pruned models is 3, and the size of each pruned model is 0.25 times, 0.5 times and 0.75 times the size of the initial open source neural network model, respectively.

4. The method according to claim 2, characterized in that The initial open source neural network model is a deep residual network Resnet-18 model or a mobilenetv2 model.

5. The method according to claim 1, characterized in that The step of converting the position of the target into a world coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the front-view camera to obtain a corresponding world coordinate position comprises: Determine a first projection relationship between an image coordinate system and a camera coordinate system of the forward-looking camera according to an intrinsic parameter matrix of the forward-looking camera, and convert the position of the target into the camera coordinate system according to the first projection relationship to obtain a corresponding camera coordinate position; A second projection relationship between the camera coordinate system of the forward-looking camera and the world coordinate system is determined according to the external parameter matrix of the forward-looking camera, and the camera coordinate position is converted into the world coordinate system according to the second projection relationship to obtain a corresponding world coordinate position.

6. The method according to claim 1, characterized in that The step of controlling the light emitting diodes LED corresponding to the high beam sub-matrix according to the control mode corresponding to the category of the target includes: When the category of the target is a car, the light emitting diode LED corresponding to the high beam sub-matrix is ​​controlled to turn off; When the category of the target is a motorcycle, a bicycle or a pedestrian, the light emitting diode LED corresponding to the high beam sub-matrix is ​​controlled to flash.

7. An adaptive high beam visual detection device, characterized in that: The device comprises: An acquisition module, used to acquire a nighttime forward image collected in real time by a forward-looking camera of a target autonomous driving vehicle; A detection module, used to process the nighttime forward image based on a pre-trained lightweight open source neural network model to obtain the category and position of each target in the nighttime forward image, wherein the lightweight open source neural network model is a neural network model with a number of convolutional layers not greater than a preset number of layers or a parameter amount not greater than a preset parameter amount, and is used to associate the nighttime forward sample image with the category and position of the corresponding target; A control module is used to convert the position of each target into a world coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix of the front-view camera to obtain a corresponding world coordinate position, convert the world coordinate position into the adaptive high beam coordinate system according to the light source projection matrix of the adaptive high beam to obtain a corresponding high beam sub-matrix, and control the light-emitting diodes LED corresponding to the high beam sub-matrix according to the control method corresponding to the category of the target, wherein the light source projection matrix represents the projection relationship between the world coordinate system and the adaptive high beam coordinate system.

8. The device according to claim 1, characterized in that The device also includes a model training module, which is used to train the lightweight open source neural network model. The model training module includes: Build submodules for building initial open source neural network models; A pruning submodule, used to prune the initial open source neural network model multiple times according to a preset model pruning method to obtain multiple pruned models of different sizes; A sensitivity analysis submodule, used to analyze the plurality of pruned models respectively according to a preset sensitivity analysis method to obtain corresponding sensitivity values; The submodule is determined by referring to the open source neural network model, and is used to use the pruned model with the lowest sensitivity value as the reference open source neural network model; An acquisition submodule, used to acquire nighttime forward sample images of the target autonomous driving vehicle and labeling results of each of the nighttime forward sample images, wherein the labeling results include the category and position of each target contained in each nighttime forward sample image; A processing submodule, configured to process each labeled nighttime forward sample image based on the reference open source neural network model, obtain a nighttime target detection result corresponding to each of the nighttime forward sample images, and determine whether the accuracy of the reference open source neural network model is greater than a preset threshold value according to the nighttime target detection result and the corresponding labeling result; An adjustment submodule, configured to adjust the parameters of the reference open source neural network model and trigger the processing submodule when the accuracy of the reference open source neural network model is not greater than the preset threshold; The training completion submodule is used to complete the training when the accuracy of the reference open source neural network model is greater than the preset threshold, so as to obtain a lightweight open source neural network model that associates the nighttime forward sample image with the category and position of the corresponding target.

9. The device according to claim 8, characterized in that The number of the multiple pruned models is 3, and the size of each pruned model is 0.25 times, 0.5 times and 0.75 times the size of the initial open source neural network model, respectively.

10. The device according to claim 8, characterized in that The initial open source neural network model is a deep residual network Resnet-18 model or a mobilenetv2 model.

11. The device according to claim 7, characterized in that The control module comprises: A first conversion submodule, used to determine a first projection relationship between an image coordinate system and a camera coordinate system of the forward-looking camera according to an intrinsic parameter matrix of the forward-looking camera, and to convert the position of the target into the camera coordinate system according to the first projection relationship to obtain a corresponding camera coordinate position; The second conversion submodule is used to determine the second projection relationship between the camera coordinate system of the forward-looking camera and the world coordinate system according to the external parameter matrix of the forward-looking camera, and convert the camera coordinate position into the world coordinate system according to the second projection relationship to obtain the corresponding world coordinate position.

12. The device according to claim 7, characterized in that The control module is specifically used for: When the category of the target is a car, the light emitting diode LED corresponding to the high beam sub-matrix is ​​controlled to turn off; When the category of the target is a motorcycle, a bicycle or a pedestrian, the light emitting diode LED corresponding to the high beam sub-matrix is ​​controlled to flash.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

14. An electronic device, characterized in that: The electronic device comprises: one or more processors; The processor is coupled to a storage device, and the storage device is used to store one or more programs; When the one or more programs are executed by the one or more processors, the electronic device implements the method as described in any one of claims 1 to 6.

15. A vehicle, characterized in that: The vehicle comprises the apparatus as claimed in any one of claims 7 to 12, or comprises the electronic device as claimed in claim 14.

Images