Vehicle-mounted camera control method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a vehicle-mounted camera control method and device, equipment, a storage medium and a product. The method is applied to the vehicle, a vehicle-mounted camera is installed on the top of the vehicle, and the method comprises the steps that after the automatic driving function of the vehicle is started, the vehicle-mounted camera is lifted to the target height according to vehicle state information and surrounding environment information of the vehicle; determining a key observation area of the vehicle-mounted camera according to a current image collected by the vehicle-mounted camera and a to-be-driven path planned by the automatic driving function; determining an optimized image central region according to the plurality of points of interest in the current image; and according to the optimized image center area and the key observation area, adjusting the pose of the vehicle-mounted camera so as to adjust the optimized image center area collected by the vehicle-mounted camera to cover the key observation area. According to the technical scheme, the effect of improving the complex road condition detection accuracy is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile accessories, and in particular to a vehicle-mounted camera control method, device, equipment, storage medium and product. Background Art

[0002] With the development of automobile autonomous driving technology, in order to improve the flexibility and intelligence of autonomous driving, it is necessary to perform intelligent analysis based on the environmental information collected when the vehicle is currently driving, so as to effectively control the vehicle's autonomous driving. Among them, using the vehicle's perception system to collect environmental information is an important part of vehicle autonomous driving.

[0003] The existing technology mainly uses a combination of visual images and multiple sensors to achieve multi-directional road condition perception for autonomous driving. When a vehicle is equipped with an autonomous driving perception system, a combination of multiple cameras is often used, together with millimeter-wave radar and lidar, to build a vehicle autonomous driving perception system.

[0004] However, there is a technical problem in the prior art of inaccurate detection of complex road conditions. Summary of the invention

[0005] One of the objects of the present invention is to provide a vehicle-mounted camera control method to solve the problem of inaccurate detection of complex road conditions by the perception system in the prior art; the second object is to provide a vehicle-mounted camera control device; the third object is to provide an electronic device; the fourth object is to provide a computer-readable storage medium; and the fifth object is to provide a computer program product.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A vehicle-mounted camera control method is applied to a vehicle, wherein a vehicle-mounted camera is installed on the top of the vehicle, and the method comprises:

[0008] After the vehicle turns on the automatic driving function, the vehicle camera is raised to a target height according to the vehicle state information and surrounding environment information of the vehicle, and the target height is the maximum height to which the vehicle camera can be raised;

[0009] Determine the key observation area of ​​the on-board camera based on the current image collected by the on-board camera and the path to be driven planned by the autonomous driving function. The key observation area includes the target position point of the path to be driven.

[0010] Determine an optimized image center area based on multiple points of interest in the current image, where the points of interest are elements on the road that are related to the autonomous driving function, and the area of ​​the optimized image center area is larger than the area of ​​the key observation area;

[0011] According to the central area of ​​the optimized image and the key observation area, the position and posture of the vehicle camera is adjusted to adjust the central area of ​​the optimized image collected by the vehicle camera to cover the key observation area.

[0012] According to the above technical means, in the prior art, due to the limitations of the camera data collection of the perception system, the perception system is not accurate in detecting complex road conditions. Therefore, when the camera collects data, the camera can be adjusted based on the information in the current image to ensure that the camera can collect as many elements related to the autonomous driving function as possible on the road, so as to help the autonomous driving function focus on these elements. First, according to the vehicle's own vehicle state information and surrounding environment information, the on-board camera is adjusted to the target height to ensure that the height of the on-board camera is the maximum height that can be reached in the current environment. According to the current image collected by the on-board camera and the to-be-traveled path planned by the autonomous driving, the key observation area of ​​the on-board camera is determined; and the optimized image center area is determined according to multiple points of interest in the current image; thereby adjusting the posture of the on-board camera based on the optimized image center area and the key observation area. Compared with the prior art, the technical solution first determines the highest height that the camera can reach to ensure that the field of view of the camera is not restricted by the height, and at the same time uses the current image obtained by the camera to adjust the image center area for the key observation area that needs to be collected, thereby controlling the multi-directional rotation of the camera and realizing multi-range information collection, thereby achieving the technical effect of improving the accuracy of complex road condition detection.

[0013] Further, according to a plurality of interest points in the current image, the optimized image center area is determined, including:

[0014] determining a plurality of initial interest points in the current image;

[0015] Determine the priority of each initial point of interest according to preset conditions, where the priority of each initial point of interest is proportional to the importance of the initial point of interest to the autonomous driving function;

[0016] According to the priority of each initial point of interest, a target point of interest is determined from all the initial points of interest, and the priority of the target point of interest is greater than the priorities of other initial points of interest;

[0017] Calculate the sum of the distances from each pixel in the current image to all target points of interest;

[0018] The distance and the smallest pixel point are taken as the center to determine the center area of ​​the optimized image.

[0019] According to the above technical means, multiple points of interest in the current image are used to optimize the determination of the central area of ​​the image, thereby ensuring that there is sufficient information that can identify the road condition in the camera's field of view, further improving the accuracy of road condition detection.

[0020] Furthermore, the sum of the distances from each pixel in the current image to all target points of interest is calculated, including:

[0021] According to the priority of each target point of interest, the weight of each target point of interest is determined, and the weight of each target point of interest is proportional to the priority;

[0022] According to the weight of each target point of interest, the weighted sum of distances from each pixel in the current image to all target points of interest is calculated;

[0023] Accordingly, the distance and the smallest pixel point are taken as the center to determine the optimized image center area, including:

[0024] The weighted distance and the smallest pixel point are taken as the center to determine the center area of ​​the optimized image.

[0025] According to the above technical means, weights are determined for the priorities of multiple points of interest, and weighted calculation is performed using the weights of the points of interest to obtain a pixel point closest to each point of interest as the center of the optimized image center area, thereby determining the optimized image center area.

[0026] Furthermore, according to the current image collected by the vehicle camera and the path to be driven planned by the autonomous driving function, the key observation area of ​​the vehicle camera is determined, including:

[0027] According to the route to be driven and the navigation map information, the camera coordinates of the target location point in the camera coordinate system are determined;

[0028] Project the camera coordinates into the image coordinate system to determine the pixel coordinates of the target location point;

[0029] Determine the key observation area based on the pixel points corresponding to the pixel coordinates.

[0030] According to the above technical means, the key observation area of ​​the camera is determined through the vehicle's driving path and navigation map information, thereby determining the main basis for camera adjustment.

[0031] Furthermore, the vehicle-mounted camera is arranged on a three-axis gimbal, the three-axis gimbal includes a rotating platform and a rotating arm arranged on the rotating platform, and the vehicle-mounted camera is arranged on the rotating arm;

[0032] The three-axis gimbal includes a first axis, a second axis and a third axis;

[0033] The first axis is perpendicular to the rotating table and is used to adjust the rotation angle of the rotating table;

[0034] The second axis is perpendicular to the rotating arm and is used to adjust the angle between the rotating arm and the rotating table;

[0035] The third axis is perpendicular to the optical axis of the vehicle camera and is used to adjust the pitch angle of the vehicle camera.

[0036] According to the above technical means, the height and angle of the vehicle-mounted camera are adjusted through the design of a three-axis gimbal, the detection range of the camera is expanded, and the accuracy of road condition detection is improved.

[0037] Adjust the position and posture of the vehicle camera based on the optimized image center area and key observation area, including:

[0038] If the key observation area is not completely within the center area of ​​the optimized image, the center area of ​​the optimized image is adjusted according to the key observation area to obtain a new center area of ​​the optimized image, and the key observation area is within the new center area of ​​the optimized image;

[0039] Calculate the depth of the central area of ​​the new optimized image;

[0040] Determine the target angle of the first axis and the target angle of the third axis according to the depth information and the center point of the center area of ​​the optimized image, the external parameters, the internal reference focal length and the distortion parameters of the vehicle-mounted camera;

[0041] The three-axis gimbal is controlled according to the target angle of the first axis and the target angle of the third axis to adjust the position and posture of the vehicle-mounted camera.

[0042] According to the above technical means, the optimized image center area of ​​the camera is adjusted based on the key observation area, so as to control the vehicle-mounted camera to adjust in the direction with more road condition information.

[0043] Further, the depth of the new optimized image center area is calculated, including:

[0044] The depth of the central area of ​​the new optimized image is determined according to the depth of each point of interest in the central area of ​​the new optimized image and the depth of the key observation area.

[0045] According to the above technical means, the depth of the point of interest and the depth of the key observation area are used to calculate and optimize the depth of the central area of ​​the image, so as to determine the adjustments that the camera needs to make to reach the central area of ​​the target image.

[0046] Further, according to the vehicle state information and the surrounding environment information of the vehicle, the vehicle camera is raised to a target height, including:

[0047] Calculate the target air resistance of the three-axis gimbal based on the vehicle's own vehicle status information and drag coefficient;

[0048] Determine a first angle and a first height corresponding to the target air resistance according to a preset resistance mapping table, wherein the resistance mapping table is used to indicate the maximum angle of the second axis corresponding to different air resistances and the highest height of the vehicle-mounted camera;

[0049] Determine a second height according to the surrounding environment information, where the second height is the highest height of the vehicle-mounted camera limited by environmental obstacles;

[0050] According to the first height, the second height and the first angle, the three-axis gimbal is operated to raise the vehicle-mounted camera to a target height, and the target height is the first height or the second height.

[0051] According to the above technical means, the height of the camera is determined by calculating the air resistance to avoid excessive air resistance to damage the vehicle-mounted camera.

[0052] Further, according to the first height, the second height and the first angle, the three-axis gimbal is operated to raise the vehicle-mounted camera to a target height, including:

[0053] If the first height is less than or equal to the second height, the three-axis gimbal is controlled according to the first angle to raise the vehicle-mounted camera to the first height, where the first height is the target height;

[0054] If the first height is greater than the second height, determining a second angle of the three-axis gimbal based on the second height;

[0055] The three-axis gimbal is controlled according to the second angle to raise the vehicle-mounted camera to a second height, where the second height is the target height.

[0056] According to the above technical means, the height limit is determined based on obstacles to avoid damage to the vehicle-mounted camera by obstacles.

[0057] A vehicle-mounted camera control device, characterized in that it is applied to a vehicle, a vehicle-mounted camera is installed on the top of the vehicle, and the device comprises:

[0058] A first processing module is used to raise the vehicle-mounted camera to a target height according to the vehicle state information and surrounding environment information of the vehicle after the vehicle turns on the automatic driving function, where the target height is the maximum height to which the vehicle-mounted camera can be raised;

[0059] The second processing module is used to determine the key observation area of ​​the vehicle camera according to the current image collected by the vehicle camera and the path to be driven planned by the automatic driving function, and the key observation area includes the target position point of the path to be driven;

[0060] A third processing module is used to determine an optimized image center area according to a plurality of interest points in the current image, where the interest points are elements in the road related to the autonomous driving function, and the area of ​​the optimized image center area is larger than the area of ​​the key observation area;

[0061] The fourth processing module is used to adjust the posture of the vehicle-mounted camera according to the central area of ​​the optimized image and the key observation area, so as to adjust the central area of ​​the optimized image collected by the vehicle-mounted camera to cover the key observation area.

[0062] Furthermore, the third processing module is also used for:

[0063] determining a plurality of initial interest points in the current image;

[0064] Determine the priority of each initial point of interest according to preset conditions, where the priority of each initial point of interest is proportional to the importance of the initial point of interest to the autonomous driving function;

[0065] According to the priority of each initial point of interest, a target point of interest is determined from all the initial points of interest, and the priority of the target point of interest is greater than the priorities of other initial points of interest;

[0066] Calculate the sum of the distances from each pixel in the current image to all target points of interest;

[0067] The distance and the smallest pixel point are taken as the center to determine the center area of ​​the optimized image.

[0068] Furthermore, the third processing module is also used for:

[0069] According to the priority of each target point of interest, the weight of each target point of interest is determined, and the weight of each target point of interest is proportional to the priority;

[0070] According to the weight of each target point of interest, the weighted sum of distances from each pixel in the current image to all target points of interest is calculated;

[0071] Accordingly, the distance and the smallest pixel point are taken as the center to determine the optimized image center area, including:

[0072] The weighted distance and the smallest pixel point are taken as the center to determine the center area of ​​the optimized image.

[0073] Furthermore, the second processing module is also used for:

[0074] According to the route to be driven and the navigation map information, the camera coordinates of the target location point in the camera coordinate system are determined;

[0075] Project the camera coordinates into the image coordinate system to determine the pixel coordinates of the target location point;

[0076] Determine the key observation area based on the pixel points corresponding to the pixel coordinates.

[0077] Furthermore, the vehicle-mounted camera is arranged on a three-axis gimbal, the three-axis gimbal includes a rotating platform and a rotating arm arranged on the rotating platform, and the vehicle-mounted camera is arranged on the rotating arm;

[0078] The three-axis gimbal includes a first axis, a second axis and a third axis;

[0079] The first axis is perpendicular to the rotating table and is used to adjust the rotation angle of the rotating table;

[0080] The second axis is perpendicular to the rotating arm and is used to adjust the angle between the rotating arm and the rotating table;

[0081] The third axis is perpendicular to the optical axis of the vehicle camera and is used to adjust the pitch angle of the vehicle camera.

[0082] The fourth processing module is also used for:

[0083] If the key observation area is not completely within the center area of ​​the optimized image, the center area of ​​the optimized image is adjusted according to the key observation area to obtain a new center area of ​​the optimized image, and the key observation area is within the new center area of ​​the optimized image;

[0084] Calculate the depth of the central area of ​​the new optimized image;

[0085] Determine the target angle of the first axis and the target angle of the third axis according to the depth information and the center point of the center area of ​​the optimized image, the external parameters, the internal reference focal length and the distortion parameters of the vehicle-mounted camera;

[0086] The three-axis gimbal is controlled according to the target angle of the first axis and the target angle of the third axis to adjust the position and posture of the vehicle-mounted camera.

[0087] According to the above technical means, the fourth processing module adjusts the optimized image center area of ​​the camera based on the key observation area, thereby controlling the vehicle-mounted camera to adjust in the direction with more road condition information.

[0088] Further, the fourth processing module is also used for:

[0089] The depth of the central area of ​​the new optimized image is determined according to the depth of each point of interest in the central area of ​​the new optimized image and the depth of the key observation area.

[0090] Furthermore, the first processing module is also used for:

[0091] Calculate the target air resistance of the three-axis gimbal based on the vehicle's own vehicle status information and drag coefficient;

[0092] Determine a first angle and a first height corresponding to the target air resistance according to a preset resistance mapping table, wherein the resistance mapping table is used to indicate the maximum angle of the second axis corresponding to different air resistances and the highest height of the vehicle-mounted camera;

[0093] Determine a second height according to the surrounding environment information, where the second height is the highest height of the vehicle-mounted camera limited by environmental obstacles;

[0094] According to the first height, the second height and the first angle, the three-axis gimbal is operated to raise the vehicle-mounted camera to a target height, and the target height is the first height or the second height.

[0095] Furthermore, the first processing module is also used for:

[0096] If the first height is less than or equal to the second height, the three-axis gimbal is controlled according to the first angle to raise the vehicle-mounted camera to the first height, where the first height is the target height;

[0097] If the first height is greater than the second height, determining a second angle of the three-axis gimbal based on the second height;

[0098] The three-axis gimbal is controlled according to the second angle to raise the vehicle-mounted camera to a second height, where the second height is the target height.

[0099] An electronic device, comprising: a memory, a processor;

[0100] Memory stores computer-executable instructions;

[0101] The processor executes the computer-executable instructions stored in the memory, so that the processor executes any method further in the above-mentioned vehicle-mounted camera control method.

[0102] A computer-readable storage medium stores computer-executable instructions, which are used to implement the above vehicle-mounted camera control method and any method in the further vehicle-mounted camera control method when executed by a processor.

[0103] A computer program product includes a computer program, which, when executed by a processor, implements any method in the above vehicle-mounted camera control method and the further vehicle-mounted camera control method.

[0104] Beneficial effects of the present invention:

[0105] (1) The present invention adopts a hierarchical control method in the process of controlling the vehicle-mounted camera. First, based on the vehicle state information and surrounding environment information of the autonomous driving vehicle, the vehicle-mounted camera is controlled to reach the highest height that can be reached at present, so as to achieve the effect of improving the range of the field of view of the vehicle-mounted camera; then, the angle of the vehicle-mounted camera is adjusted based on the determination of the center area of ​​the optimized image. By using a hierarchical calculation method, the height and angle adjustment of the vehicle-mounted camera is controlled in stages, so as to avoid the computing resources loaded on the vehicle from performing multiple types of calculation reasoning at the same time, thereby reducing the demand for real-time computing resources.

[0106] (2) In the process of controlling the vehicle-mounted camera, the present invention adopts a multi-target final method to control the camera, obtain multiple points of interest in the current image and the key observation area of ​​the current image, realize multi-target tracking based on the multiple points of interest, and realize the determination of the center area of ​​the optimized image based on the multiple points of interest and the key observation area. By controlling the height and angle of the camera, the field of view of the camera is expanded; at the same time, the camera is adjusted based on the multiple points of interest and the key observation area to ensure that the center area of ​​the image observed by the camera contains multiple points of interest and the key observation area that the vehicle needs to pay attention to, thereby improving the content of key road condition information in the image detected by the camera, and achieving the effect of improving the detection accuracy of the vehicle-mounted camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] Figure 1 The side view and top view of the three-axis gimbal when the vehicle-mounted camera provided by the present disclosure is in the retracted state;

[0108] Figure 2 The side view and top view of the three-axis gimbal when the vehicle-mounted camera provided by the present disclosure is in the extended state;

[0109] Figure 3 A schematic diagram of three-axis motion in a three-axis gimbal provided by the present invention;

[0110] Figure 4 A schematic diagram of the structure of a vehicle-mounted camera control system for autonomous driving provided by the present disclosure;

[0111] Figure 5 The process diagram of the vehicle camera control method provided by the present disclosure is as follows Figure 1 ;

[0112] Figure 6 The process diagram of the vehicle camera control method provided by the present disclosure is as follows Figure 2 ;

[0113] Figure 7 A schematic diagram of a process for determining a central area of ​​an optimized image of a vehicle-mounted camera provided in the present disclosure;

[0114] Figure 8 The process diagram of the vehicle camera control method provided by the present disclosure is as follows Figure 3 ;

[0115] Fig. 9 A schematic diagram of adjusting angle of a three-axis gimbal provided in the present invention;

[0116] Fig.10 Another schematic diagram of calculating the adjustment angle of a three-axis gimbal provided by the present invention;

[0117] Fig.11 A schematic diagram of the installation method of a three-axis gimbal with a camera provided in the present invention Figure 1 ;

[0118] Fig.12 A schematic diagram of the installation method of a three-axis gimbal with a camera provided in the present invention Figure 2 ;

[0119] Fig.13 Schematic diagram of key observation areas, points of interest, and optimized image center areas provided for the present disclosure;

[0120] Fig.14 A schematic diagram of a camera provided by the present disclosure moving from an area in an initial image area to a central area of ​​an optimized image based on an optimal solution of central coordinates;

[0121] Fig.15 A schematic diagram of the structure of the vehicle-mounted camera control device provided by the present disclosure;

[0122] Fig.16 A schematic diagram of the structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0123] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0124] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0125] In the prior art, the perception system installed on the autonomous driving vehicle mainly adopts the combination of visual images and multiple sensors to realize the multi-directional road condition perception of autonomous driving. When the vehicle is equipped with the autonomous driving perception system, a combination of multiple cameras is often used, together with millimeter wave radar and laser radar, to build the perception system of the vehicle's autonomous driving.

[0126] However, this type of perception system cannot solve some problems in complex urban areas or garage scenes. For example, in mountainous city scenes, there are traffic lights or signboards installed relatively high, and the left turn waiting area of ​​the road is too close to the traffic lights or signboards; at this time, the traffic lights or signboards are outside the vertical field of view (FOV) of the vehicle perception system camera, resulting in the vehicle being unable to rely on automatic driving to pass such sections of road. As a result, there is a technical problem of inaccurate detection of complex road conditions in the existing technology.

[0127] In view of the above technical problems, the present method proposes the following technical concept: Due to the limitations of the camera data collection of the perception system in the prior art, the perception system is not accurate in detecting complex road conditions. Therefore, when the camera collects data, the camera can be adjusted based on the information in the current image to ensure the collection efficiency of the camera. First, according to the vehicle's own vehicle state information and surrounding environment information, the vehicle-mounted camera is adjusted to the target height to ensure that the height of the vehicle-mounted camera is the maximum height that can be reached in the current environment. According to the current image collected by the vehicle-mounted camera and the path to be driven planned by the automatic driving, the key observation area of ​​the vehicle-mounted camera is determined; and the optimized image center area is determined according to multiple points of interest in the current image; thereby adjusting the posture of the vehicle-mounted camera based on the optimized image center area and the key observation area. Compared with the prior art, the present method first determines the highest height that the camera can reach to ensure that the field of view of the camera is not restricted by the height, and at the same time uses the current image obtained by the camera to adjust the image center area for the key observation area that needs to be collected, thereby controlling the multi-directional rotation of the camera to achieve multi-range information collection; thereby achieving the technical effect of improving the accuracy of complex road condition detection.

[0128] The technical solution of the present method and how the technical solution of the present method solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present method will be described below in conjunction with the accompanying drawings.

[0129] Figure 1-Figure 3 The present invention provides a schematic diagram of a three-axis gimbal provided with a vehicle-mounted camera. Figure 1-3As shown, the three-axis gimbal includes: a rotating platform 101, a rotating arm 102, a vehicle-mounted camera 103, a first axis 104, a second axis 105, and a third axis 106. The first axis 104 is perpendicular to the rotating platform 101 and is used to adjust the rotation angle of the rotating platform 101; the second axis 105 is parallel to the rotating arm 102 and is used to adjust the angle between the rotating arm 102 and the rotating platform 101. The third axis 106 is perpendicular to the optical axis of the vehicle-mounted camera 103 and is used to adjust the pitch angle of the vehicle-mounted camera 103. Figure 1 The side view and top view of the three-axis gimbal when the vehicle-mounted camera provided by the present invention is in the retracted state, Figure 2 The side view and top view of the three-axis gimbal when the vehicle-mounted camera provided by the present invention is in the extended state, Figure 3 A schematic diagram of three-axis motion in a three-axis gimbal provided in the present invention.

[0130] In such Figure 1-3 Based on the three-axis gimbal provided with the vehicle-mounted camera, the method proposes a control system for the vehicle-mounted camera for autonomous driving. Figure 4 This is a schematic diagram of the structure of the vehicle-mounted camera control system for autonomous driving provided by the present disclosure. Figure 4 As shown, the system includes: a first axis component 401, a second axis component 402, a third axis component 403, a visual perception control component 404, and an automatic driving planning control component 405. Among them, the first axis component 401 includes: a motor and its drive, a rotating table, and an angle measuring sensor corresponding to the first axis and the rotating table; the second axis component 402 includes: a motor and its drive, a rotating arm, and an angle measuring sensor corresponding to the second axis and the rotating arm; the third axis component 403 includes: a motor and its drive, a camera unit, and an angle measuring sensor corresponding to the third axis and the camera unit. Among them, the automatic driving planning control component 405 is used to obtain the perception information or cognitive information sent by the visual perception control component 404, and generate automatic driving planning information, and transmit the automatic driving planning information to the visual perception control component 404; the visual perception control component 404 is used to generate perception or cognitive information based on the image information collected by the third axis component; the visual perception control component 404 is used to obtain the angle measuring information measured by the first to third axis components, and generate control instructions in combination with the angle measuring information and the automatic driving planning information to control the operation of the first to third axis components.

[0131] Based on the above embodiments, this method proposes a vehicle-mounted camera control method, which is applied to a vehicle, and a vehicle-mounted camera is installed on the top of the vehicle. Figure 5 The process diagram of the vehicle camera control method provided by the present disclosure is as follows Figure 1 ,like Figure 5 As shown, the method includes:

[0132] S501. After the vehicle turns on the automatic driving function, the vehicle camera is raised to a target height according to the vehicle's own state information and surrounding environment information.

[0133] In this step, the target height is the maximum height to which the vehicle-mounted camera can be raised.

[0134] It should be noted that the vehicle status information mentioned in this step may be: the current speed of the vehicle, the yaw rate of the vehicle, the body shape of the vehicle, the frontal area of ​​the vehicle camera, and the engine power of the vehicle.

[0135] The surrounding environment information may include: current wind speed, current air pressure, current terrain, and current traffic conditions.

[0136] The purpose of raising the vehicle-mounted camera to the target height according to the vehicle's own vehicle state information and the surrounding environment information in this step is to increase the field of view variation range of the vehicle-mounted camera while ensuring that the vehicle-mounted camera is not affected by the vehicle's driving resistance and the environment.

[0137] S502: Determine a key observation area of ​​the vehicle camera based on a current image captured by the vehicle camera and a path to be traveled planned by the automatic driving function.

[0138] In this step, the key observation area includes the target location point of the path to be driven.

[0139] Optionally, a possible implementation method of determining the key observation area of ​​the vehicle-mounted camera is:

[0140] S5021. Determine the camera coordinates of the target location point in the camera coordinate system according to the path to be driven and the navigation map information.

[0141] In this step, the target position point refers to a point taken at M times the time headway ahead of the path to be driven, where M is greater than 0. Exemplarily, when the vehicle is driving normally, the current speed of the vehicle is determined based on the navigation map information of the vehicle during automatic driving to calculate the unit headway of the vehicle, and the position of the vehicle on the path to be driven at M times the time headway is calculated, and the position is used as the target position point. When the vehicle is driving at a low speed, a minimum distance value is determined according to the blind spot of the vehicle, and M times the time headway is calculated based on the distance value, and the corresponding target position point on the path to be driven is determined when the M times the time headway does not exceed the distance value.

[0142] After determining the target position point, the camera coordinates of the target position point in the camera coordinate system can be determined by first determining the position of the target position point in the world coordinate system, which can be obtained by means of a sensor. Based on the coordinate system conversion method, the target position point in the world coordinate system is mapped to the camera coordinate system, and the camera coordinates corresponding to the target position point are determined, and the camera coordinates are three-dimensional coordinates.

[0143] S5022. Project the camera coordinates into the image coordinate system to determine the pixel coordinates of the target location point.

[0144] In this step, the camera coordinates may be projected to the image coordinates by establishing a projection of the camera coordinate system to the image plane, and determining the camera coordinates as pixel coordinates of the image plane based on projection changes and distortion conversion.

[0145] S5023. Determine the key observation area according to the pixel points corresponding to the pixel coordinates.

[0146] In this step, the key observation area can be determined by: establishing the key observation area based on the preset area size and taking the pixel coordinates as the center, wherein the key observation area can be a rectangle or other shapes.

[0147] It should be noted that the preset area size cannot exceed the maximum image acquisition size of the vehicle camera.

[0148] S503: Determine the optimized image center area according to multiple points of interest in the current image.

[0149] In this step, the points of interest are elements on the road that are related to the autonomous driving function, and the area of ​​the center area of ​​the optimized image is larger than the area of ​​the key observation area.

[0150] Optionally, a possible implementation method for determining the center area of ​​the optimized image based on multiple points of interest is: using a minimum value calculation method to calculate the point with the smallest sum of distances to multiple points of interest, and determine it as the center point of the center area of ​​the optimized image, and based on a preset size of the center area of ​​the optimized image, establish the center area of ​​the optimized image. How to determine the center area of ​​the optimized image is described in the following section of this method Figure 7 Further explanation is provided in the illustrated embodiment and redundant elaboration is not given here.

[0151] S504: Adjust the posture of the vehicle-mounted camera according to the central area of ​​the optimized image and the key observation area, so as to adjust the central area of ​​the optimized image collected by the vehicle-mounted camera to cover the key observation area.

[0152] In this step, the method of adjusting the posture of the vehicle-mounted camera based on the center area of ​​the optimized image and the key observation area can be: first determine whether the center area of ​​the optimized image completely covers the key observation area. When it is completely covered, the center point of the center area of ​​the optimized image is determined as the position point where the main optical axis of the vehicle-mounted camera needs to be adjusted; when it is not completely covered, the center area of ​​the optimized image is adjusted to completely cover the key observation area, and the center point of the new center area of ​​the optimized image after adjustment is determined as the position point where the main optical axis of the vehicle-mounted camera needs to be adjusted.

[0153] It should be noted that the posture adjustment of the vehicle-mounted camera in this step is as follows Figure 8 Further detailed explanation is given in the illustrated embodiment, and no unnecessary elaboration is given.

[0154] This embodiment proposes a vehicle camera control method, which adjusts the vehicle camera to the target height according to the vehicle state information and the surrounding environment information of the vehicle, ensuring that the height of the vehicle camera is the maximum height that can be reached in the current environment. According to the current image collected by the vehicle camera and the path to be driven planned by the automatic driving, the key observation area of ​​the vehicle camera is determined; and the optimized image center area is determined according to multiple points of interest in the current image; thereby adjusting the posture of the vehicle camera based on the optimized image center area and the key observation area. Compared with the prior art, this method first determines the highest height that the camera can reach to ensure that the field of view of the camera is not restricted by the height. At the same time, the current image obtained by the camera is used to adjust the image center area for the key observation area that needs to be collected; multiple points of interest in the current image are used as observation points, and multi-target tracking adjustment is realized in combination with the key observation area, ensuring that the image center area of ​​the camera covers the key observation area and the points of interest with high importance; based on this, the multi-directional rotation of the camera is controlled to ensure that the field of view of the camera covers many aspects of important information of the current road; thereby achieving the technical effect of improving the accuracy of complex road condition detection.

[0155] Based on the above embodiments, the present method also provides a method for controlling a vehicle-mounted camera to reach a target height. Figure 6 The process diagram of the vehicle camera control method provided by the present disclosure is as follows Figure 2 .like Figure 6 As shown, in Figure 5 Based on the illustrated embodiment, the method of raising the camera to the target height in S501 is described in detail. The method includes:

[0156] S601, calculating the target air resistance of the three-axis gimbal according to the vehicle state information and the drag coefficient.

[0157] In this step, a possible implementation method for calculating the target air resistance of the three-axis gimbal is:

[0158] S6011. Determine vehicle status information, including vehicle speed, vehicle yaw rate, and windward area of ​​the three-axis gimbal.

[0159] S6012: Calculate the target air resistance of the three-axis gimbal based on the vehicle speed, the yaw rate of the vehicle, the frontal area of ​​the three-axis gimbal, and the drag coefficient. The yaw rate refers to the rotation speed of the vehicle around the vertical axis.

[0160] Exemplarily, the target air resistance is calculated using calculation formula 1:

[0161]

[0162] Where ρ refers to the air density, F d refers to the target air resistance, V refers to the vehicle speed, C d It refers to the drag coefficient, and A refers to the windward area of ​​the three-axis gimbal.

[0163] S602: Determine a first angle and a first height corresponding to the target air resistance according to a preset resistance mapping table.

[0164] In this step, the resistance mapping table is used to indicate the maximum angle of the second axis and the maximum height of the vehicle-mounted camera corresponding to different air resistances.

[0165] The maximum angle refers to the maximum angle between the rotating arm and the rotating table when the second axis perpendicular to the rotating arm drives the rotating arm to rotate. The highest height of the vehicle-mounted camera refers to the highest distance of the highest point of the vehicle-mounted camera.

[0166] S603: Determine a second height according to the surrounding environment information, where the second height is the highest height of the vehicle-mounted camera limited by environmental obstacles.

[0167] In this step, the second height may be determined according to the surrounding environment information by:

[0168] S6031. Determine the current image acquired by the vehicle-mounted camera, and based on an image detection algorithm, detect whether there is a height restriction sign and / or a height restriction building in the current image.

[0169] S6032: When there are height restriction signs and / or height restricted buildings, obtain height restriction information based on the sensor device of the vehicle-mounted camera and determine the second height.

[0170] S604: Control the three-axis gimbal according to the first height, the second height, and the first angle to raise the vehicle-mounted camera to a target height.

[0171] In this step, the target height is the first height or the second height.

[0172] Optionally, a possible implementation manner of controlling the three-axis gimbal according to the first height, the second height and the first angle to raise the vehicle-mounted camera to a target height is:

[0173] S6041. If the first height is less than or equal to the second height, control the three-axis gimbal according to the first angle to raise the vehicle-mounted camera to the first height, where the first height is the target height.

[0174] S6042: If the first height is greater than the second height, determine a second angle of the three-axis gimbal based on the second height.

[0175] S6043. Control the three-axis gimbal according to the second angle to raise the vehicle-mounted camera to a second height, where the second height is the target height.

[0176] In this step, the method of controlling the vehicle-mounted camera to reach the target height can be: based on the current height and the target height of the vehicle-mounted camera, planning the motion curve of the vehicle-mounted camera smoothed by the interpolation method, controlling the second axis and the third axis so that the angle of the camera relative to the rotating platform of the three-axis gimbal remains unchanged, while controlling the camera to move to the target height.

[0177] In this embodiment, the vehicle status information and environmental information are used to determine the maximum height that the vehicle camera can reach; the movement of the second and third axes of the three-axis gimbal is controlled to adjust the second angle so that the vehicle camera moves from the current height to the target height. The maximum height adjustment method is used to increase the rotation range of the vehicle camera, thereby increasing the field of view of the vehicle camera, which is conducive to the collection of multi-directional road condition information.

[0178] Based on the above embodiments, the present method also provides a method for determining the center area of ​​an optimized image of a vehicle-mounted camera. Figure 7 The following is a flow chart of the method for determining the center area of ​​an optimized image of a vehicle-mounted camera provided by the present disclosure. Figure 7 As shown, in Figure 5 Based on the illustrated embodiment, the determination of the optimized image center area in S503 is described in detail. The method includes:

[0179] S701: Determine a plurality of initial interest points in the current image.

[0180] In this step, the method of determining multiple initial interest points in the current image may be:

[0181] Based on the image recognition algorithm, traffic signs and obstacles in the current image are detected, and these traffic signs and obstacles are determined as a plurality of initial interest points in the current image.

[0182] When the road is relatively empty, there are fewer objects of interest, and the point of interest is set to a point N times the time distance in front of the center axis of the vehicle, so that the three-axis gimbal can be controlled to return to the center and face the front of the vehicle. N is greater than 0.

[0183] Exemplarily, a point on twice the time headway is taken. When the vehicle is traveling at a low speed, a general passenger car can take the minimum value of 30 meters. The mathematical formula is expressed as a point on the distance Max(2×vehicle speed, 30), that is, a distance corresponding to twice the time headway of not less than 30 meters. When the calculated twice the time headway is greater than 30 meters, a point on twice the time headway is selected. When the calculated twice the time headway is less than 30 meters, a point on 30 meters is selected.

[0184] The multiple initial points of interest may be: traffic lights, dynamic targets, and obstacles in the direction of the vehicle's to-be-traveled path.

[0185] S702: Determine the priority of each initial point of interest according to preset conditions.

[0186] In this step, the priority of each initial point of interest is proportional to the importance of the initial point of interest to the autonomous driving function.

[0187] The preset conditions may be: the closer the distance to the autonomous driving vehicle, the higher the priority of the point of interest; the priority of prohibition and warning traffic signs is higher than that of ordinary instruction traffic signs; the priority of dynamic targets is higher than that of static targets.

[0188] For example, if there is a sharp turn sign and a dangerous animal sign ahead, the sharp turn sign should have a higher priority than the dangerous animal sign.

[0189] S703: Determine a target point of interest from all the initial points of interest according to the priority of each initial point of interest.

[0190] In this step, the priority of the target POI is higher than the priority of other initial POIs.

[0191] Optionally, the target point of interest may be determined from the multiple initial points of interest by sorting the multiple initial points of interest from high to low according to priority, and selecting the initial points of interest ranked before a preset number as the target point of interest.

[0192] Exemplarily, when there are more than three initial points of interest, the top three points of interest ranked by priority are selected as target points of interest.

[0193] S704, calculating the sum of distances from each pixel in the current image to all target interest points.

[0194] Optionally, a possible implementation of calculating the distance sum is:

[0195] S7041. Determine the weight of each target point of interest according to the priority of each target point of interest, where the weight of each target point of interest is proportional to the priority.

[0196] For example, assume that there are multiple target interest points I, and the pixel coordinates of each target interest point are I i =(x i ,y i ), the weight of each target point of interest is k i , the higher the priority of each target interest point, the greater the weight of each target interest point.

[0197] S7042. Calculate the weighted sum of distances from each pixel in the current image to all target interest points according to the weight of each target interest point.

[0198] Exemplarily, referring to the weights and pixel coordinates of the target interest points in S7041, the weighted distance sum from each pixel point in the current image to all target interest points may be calculated as follows:

[0199] Assume that there is a center point C1(x,y), calculate the distance from the center point to each target point of interest I i (x i ,y i ), and find the minimum value. As shown in Formula 2:

[0200]

[0201] Where f(x) represents the sum of weighted distances, n is the number of target points of interest, k is i refers to the weight of each target interest point, and n is a positive integer greater than 1; i refers to the i-th target interest point, (x i ,y i ) is the pixel coordinate of the i-th target interest point, and (x, y) is the pixel coordinate of any center point.

[0202] S705: Taking the pixel point with the smallest distance as the center, determine the center area of ​​the optimized image.

[0203] Optionally, based on S7041-S7042, a possible implementation method for determining the center area of ​​the optimized image is: taking the pixel point with the smallest weighted distance as the center to determine the center area of ​​the optimized image.

[0204] Exemplarily, a gradient calculation is performed on Formula 2 to calculate the weighted distance and the change trend in the vector direction, as shown in Formula 3:

[0205]

[0206] in, as well as It refers to the gradient operation on the sum of weighted distances f(x). Other symbols refer to the explanation of formula 2 above.

[0207] The gradient in the gradient operation of the weighted distance sum is set to 0, as shown in Formula 4:

[0208]

[0209] Based on formula 4, we can get formula 5:

[0210]

[0211] The final distance and the minimum pixel point are calculated based on Formula 5 and determined as the center point of the center area of ​​the optimized image, as shown in Formula Combination 6:

[0212]

[0213] The interpretation of each unexplained symbol in Formula 3-6 is consistent with the corresponding interpretation of the symbol in Formula 2.

[0214] In this embodiment, based on the calculation of the sum of the minimum weighted distances, the center point ultimately used to optimize the center area of ​​the image is determined, thereby obtaining the center point closest to the sum of all points of interest. The ultimately determined optimized center area of ​​the image can be regarded as an area with high image information richness.

[0215] Based on the above embodiments, the present method also provides a method for adjusting the posture of a vehicle-mounted camera. Figure 8 The process diagram of the vehicle camera control method provided by the present disclosure is as follows Figure 3 .like Figure 8 As shown, in Figure 5 Based on the illustrated embodiment, the adjustment of the vehicle-mounted camera posture in S504 is described in detail. The method includes:

[0216] S801: If the key observation area is not completely within the center area of ​​the optimized image, the center area of ​​the optimized image is adjusted according to the key observation area to obtain a new center area of ​​the optimized image.

[0217] In this step, the key observation area is located in the central area of ​​the new optimized image.

[0218] Optionally, the method of adjusting the center area of ​​the optimized image according to the key observation area may be:

[0219] The relative positions of the key observation area and the central area of ​​the optimized image are calculated, and the central area of ​​the optimized image is moved toward the direction of the key observation area according to the relative positions until the central area of ​​the optimized image completely covers the key observation area.

[0220] For example, assume that the key observation area is a rectangle S k , optimize the image center area to a rectangle S c0 '; When the key observation area is completely covered by the center area of ​​the optimized image, the coordinates calculated in Formula 6 are the optimal solution of the center coordinates of the center area of ​​the optimized image. If the key observation area and the center area of ​​the optimized image do not completely overlap, it means that Formula 6 calculates an unconstrained optimal solution. In order to meet the constraints of the key observation area, the unconstrained optimal solution needs to be adjusted. See Fig.13 , specifically:

[0221] For the key observation area S k The four vertices RSTU are determined in turn to determine whether they are within the rectangle Sc0' in the center area of ​​the optimized image:

[0222] according to Fig.13 The positions of the four vertices of the key observation area Sk can be determined. The two vertices R and vertex U on the left side of the key observation area Sk exceed the left boundary of the central area SC0' of the unconstrained optimization image. Therefore, the unconstrained optimal solution corresponding to the center of the unconstrained optimization image needs to be moved to the left so that the left boundary of the rectangular key observation area coincides with the left boundary of the central area of ​​the optimized image.

[0223] Similarly, when one of the four points in RSTU is above the upper boundary of the center area of ​​the optimized image corresponding to the unconstrained optimal solution, the unconstrained optimal solution needs to be moved upward; when one of the four points in RSTU is below the lower boundary of the center area of ​​the optimized image corresponding to the unconstrained optimal solution, the unconstrained optimal solution needs to be moved downward; when one of the four points in RSTU is to the right of the right boundary of the center area of ​​the optimized image corresponding to the unconstrained optimal solution, the unconstrained optimal solution needs to be moved to the right. The ultimate goal is to make the center area of ​​the optimized image completely cover the key observation area.

[0224] It should be noted that Fig.13 The change process of the central area of ​​the image is drawn in, specifically:

[0225] When the optimized image center area calculation is not performed, the current camera has an initial image center area S C0 When the unconstrained optimal solution is calculated, the unconstrained optimized image center area S is obtained. C0’ After calculating the constrained optimal solution based on the key observation area Sk, the constrained optimized image center area S is obtainedC1 The image center area changes with the change of the image center point corresponding to the follower. The initial image center area S C0 The corresponding image center point is the initial image center point C0, and the unconstrained image center area S C0’ The center point C0' refers to the unconstrained optimal solution, and the constrained image center area S C1 The center point C1 refers to the constrained optimal solution.

[0226] like Fig.13 As shown in , after the adjustment is completed, the four points of RSTU are all inside the rectangle Sc1 in the center area of ​​the optimized image, that is, a new optimal solution within the constraint range is obtained. As shown in Formula 7:

[0227]

[0228] Where C1 represents the constrained optimal solution of the center point of the new optimized image center area, and C0' represents the unconstrained optimal solution of the center point of the optimized image center area before adjustment; Δx C1C0' and Δy C1C0' Indicates the adjustment offset in the image coordinate system; x C1 and C1 is the coordinate of the constrained optimal solution after adjustment; n is the number of target interest points, and n is a positive integer greater than 1; i refers to the i-th target interest point, (x i ,y i ) is the pixel coordinate of the i-th target interest point, k i Refers to the weight of each target point of interest. Fig.13 Three points of interest are marked as examples, point of interest 1: stop line; point of interest 2: traffic light; point of interest 3: important traffic sign.

[0229] Fig.14 The change process of the central area of ​​the actual image is plotted, and the key observation area S can be completely covered by calculation. k The optimized image center area S C1 Afterwards, from the initial image center area S C0 Move to the optimized image center area S C1 .like Fig.14 As shown, the central area S of the rectangular initial image before the camera is adjusted C0 The four vertices are: M0, N0, P0, O0, and the center point is C0; the center area S of the rectangular optimized image after the camera is adjusted C1 The four vertices are: M1, N1, P1, O1, and the center point is C1; the center area S of the optimized image C1 Fully covered rectangular key observation area S kThe four vertices are: S, R, U, and T.

[0230] S802: Calculate the depth of the central area of ​​the new optimized image.

[0231] Optionally, a possible implementation of calculating the depth of the central area of ​​the new optimized image is:

[0232] The depth of the central area of ​​the new optimized image is determined according to the depth of each point of interest in the central area of ​​the new optimized image and the depth of the key observation area.

[0233] Exemplarily, as shown in Formula 8: the depth of the new optimized image center area is calculated according to the depth of each point of interest in the center area of ​​each image and the depth of the key observation area, and the calculation is performed by weighted averaging.

[0234]

[0235] Among them, d C1 represents the depth of the new optimized image center area finally obtained, k i refers to the weight of each point of interest in the central area of ​​the image, k Sk Indicates the key observation area S k The weight of Sk Indicates the key observation area S k The depth, d i represents the depth of each interest point in the central area of ​​the image, and m refers to the number of all interest points in the central area of ​​the image.

[0236] S803 , determining a target angle of the first axis and a target angle of the third axis according to the depth information of the central area of ​​the optimized image and its central point, the external parameters of the vehicle-mounted camera, the internal reference focal length and the distortion parameters.

[0237] For example, Fig. 9 A schematic diagram of calculating the adjustment angle of a three-axis gimbal provided by the present disclosure, which is used to determine the target angle of the third axis; Fig.10 Another schematic diagram of calculating the adjustment angle of a three-axis gimbal provided by the present disclosure is used to determine the target angle of the first axis. Fig. 9 As shown, the center point C1 of the optimized image center area corresponds to the three-dimensional coordinate point A in the vehicle coordinate system. C1 , A C1 The depth of point C1 from the optical center of the vehicle camera And the three-dimensional coordinate point A can be known through the internal reference according to the pinhole imaging principle. C1 The angle between the horizontal and vertical planes is Projecting onto the horizontal plane passing through the optical center and onto the vertical plane passing through the optical axis, we obtain d1 and d3.

[0238] 3D coordinate point A C1 The projection of the optical center of the vehicle camera on the vertical plane passing through the optical axis is d1. From the vertical pixel distance from the three-dimensional coordinate point to the center of the image after dedistortion and the focal length, it can be obtained that the angle between the straight line from the three-dimensional coordinate point to the optical center and the optical axis is γ1, and the distance from the optical center to the rotation axis is d2. Fig. 9 As shown, we can find the way to make the vehicle-mounted camera aim at the three-dimensional coordinate point A C1 , the target pitch angle that needs to be rotated is γ0, as shown in Formula 9:

[0239]

[0240] 3D coordinate point A C1 The projection of the distance to the optical center of the vehicle camera on the horizontal plane passing through the optical center is d3, which is the three-dimensional coordinate point A after dedistortion. C1 The horizontal pixel distance to the center of the image and the focal length can be used to obtain the three-dimensional coordinate point A C1 The angle between the straight line to the optical center and the optical axis is α1, and the distance from the optical center to the rotation axis is d4. Fig.10 As shown, we can find the three-dimensional coordinate point A that makes the camera point to the camera. C1 , the target yaw angle to be rotated is α0, as shown in formula 10:

[0241]

[0242] S804: Control the three-axis gimbal according to the target angle of the first axis and the target angle of the third axis to adjust the position and posture of the vehicle-mounted camera.

[0243] In this step, motion planning is performed for the first axis of the rotating disk of the three-axis gimbal and the third axis controlling the vehicle-mounted camera; wherein the target pitch angle is γ0 and the target yaw angle is α0. There are many methods for motion planning, and commonly used multi-axis robot motion planning methods can be used. Preferably, the planning method based on interpolation has low computational cost, and since the curve shape of polynomial curve interpolation is flexible and changeable, quintic curve interpolation can be used for three-axis gimbal motion planning.

[0244] The control method for controlling the movement of the first and third axes of the three-axis gimbal can adopt an algorithm of an inner loop of angular velocity and an outer loop of each axis angle; the optical axis moves toward the center point of the center area of ​​the optimized image, and finally the main optical axis direction of the vehicle-mounted camera is aligned with the center of the center area of ​​the optimized image. The control method for angular velocity is to achieve acceleration and deceleration by driving the motor torque.

[0245] In this embodiment, the central area of ​​the image is adjusted and optimized so that it can cover the key observation area and cover every point of interest as much as possible, and finally a new optimized central area of ​​the image is obtained; based on the new central area of ​​the image, the adjustment of the direction and angle of the camera is controlled so that the main optical axis of the vehicle-mounted camera coincides with the center point of the new central area of ​​the image.

[0246] Fig.11 The following is a schematic diagram of the installation method of a three-axis gimbal provided by the present invention. Fig.11 As shown, the three-axis gimbal is installed at the front end of the top of the vehicle and can be used to monitor the road conditions in front of the vehicle. Fig.12 A schematic diagram of the installation method of a three-axis gimbal with a camera provided in the present invention Figure 2 The three-axis gimbal is mounted on the front and rear ends of the vehicle's roof and can be used to monitor road conditions in front of and behind the vehicle.

[0247] Fig.15 The schematic diagram of the structure of the vehicle camera control device provided by the present invention is as follows: Fig.15 As shown, the device is applied to a vehicle, and a vehicle-mounted camera is installed on the top of the vehicle. The device includes:

[0248] The first processing module 1501 is used to raise the vehicle-mounted camera to a target height according to the vehicle state information and surrounding environment information of the vehicle after the vehicle turns on the automatic driving function, and the target height is the maximum height to which the vehicle-mounted camera can be raised;

[0249] The second processing module 1502 is used to determine the key observation area of ​​the vehicle camera according to the current image collected by the vehicle camera and the path to be traveled planned by the automatic driving function, where the key observation area includes the target position point of the path to be traveled;

[0250] The third processing module 1503 is used to determine the optimized image center area according to multiple points of interest in the current image, where the points of interest are elements in the road related to the automatic driving function, and the area of ​​the optimized image center area is larger than the area of ​​the key observation area;

[0251] The fourth processing module 1504 is used to adjust the posture of the vehicle-mounted camera according to the central area of ​​the optimized image and the key observation area, so as to adjust the central area of ​​the optimized image collected by the vehicle-mounted camera to cover the key observation area.

[0252] Further, the third processing module 1503 is also used for:

[0253] determining a plurality of initial interest points in the current image;

[0254] Determine the priority of each initial point of interest according to preset conditions, where the priority of each initial point of interest is proportional to the importance of the initial point of interest to the autonomous driving function;

[0255] According to the priority of each initial point of interest, a target point of interest is determined from all the initial points of interest, and the priority of the target point of interest is greater than the priorities of other initial points of interest;

[0256] Calculate the sum of the distances from each pixel in the current image to all target points of interest;

[0257] The distance and the smallest pixel point are taken as the center to determine the center area of ​​the optimized image.

[0258] Further, the third processing module 1503 is also used for:

[0259] According to the priority of each target point of interest, the weight of each target point of interest is determined, and the weight of each target point of interest is proportional to the priority;

[0260] According to the weight of each target point of interest, the weighted sum of distances from each pixel in the current image to all target points of interest is calculated;

[0261] Accordingly, the distance and the smallest pixel point are taken as the center to determine the optimized image center area, including:

[0262] The weighted distance and the smallest pixel point are taken as the center to determine the center area of ​​the optimized image.

[0263] Furthermore, the second processing module 1502 is also used for:

[0264] According to the route to be driven and the navigation map information, the camera coordinates of the target location point in the camera coordinate system are determined;

[0265] Project the camera coordinates into the image coordinate system to determine the pixel coordinates of the target location point;

[0266] Determine the key observation area based on the pixel points corresponding to the pixel coordinates.

[0267] Furthermore, the vehicle-mounted camera is arranged on a three-axis gimbal, the three-axis gimbal includes a rotating platform and a rotating arm arranged on the rotating platform, and the vehicle-mounted camera is arranged on the rotating arm;

[0268] The three-axis gimbal includes a first axis, a second axis and a third axis;

[0269] The first axis is perpendicular to the rotating table and is used to adjust the rotation angle of the rotating table;

[0270] The second axis is perpendicular to the rotating arm and is used to adjust the angle between the rotating arm and the rotating table;

[0271] The third axis is perpendicular to the optical axis of the vehicle camera and is used to adjust the pitch angle of the vehicle camera.

[0272] The fourth processing module 1504 is further configured to:

[0273] If the key observation area is not completely within the center area of ​​the optimized image, the center area of ​​the optimized image is adjusted according to the key observation area to obtain a new center area of ​​the optimized image, and the key observation area is within the new center area of ​​the optimized image;

[0274] Calculate the depth of the central area of ​​the new optimized image;

[0275] Determine the target angle of the first axis and the target angle of the third axis according to the depth information and the center point of the center area of ​​the optimized image, the external parameters, the internal reference focal length and the distortion parameters of the vehicle-mounted camera;

[0276] The three-axis gimbal is controlled according to the target angle of the first axis and the target angle of the third axis to adjust the position and posture of the vehicle-mounted camera.

[0277] Further, the fourth processing module 1504 is further configured to:

[0278] The depth of the central area of ​​the new optimized image is determined according to the depth of each point of interest in the central area of ​​the new optimized image and the depth of the key observation area.

[0279] Furthermore, the first processing module 1501 is also used for:

[0280] Calculate the target air resistance of the three-axis gimbal based on the vehicle's own vehicle status information and drag coefficient;

[0281] Determine a first angle and a first height corresponding to the target air resistance according to a preset resistance mapping table, wherein the resistance mapping table is used to indicate the maximum angle of the second axis corresponding to different air resistances and the highest height of the vehicle-mounted camera;

[0282] Determine a second height according to the surrounding environment information, where the second height is the highest height of the vehicle-mounted camera limited by environmental obstacles;

[0283] According to the first height, the second height and the first angle, the three-axis gimbal is controlled to raise the vehicle-mounted camera to a target height, which is the first height or the second height.

[0284] Furthermore, the first processing module 1501 is also used for:

[0285] If the first height is less than or equal to the second height, the three-axis gimbal is controlled according to the first angle to raise the vehicle-mounted camera to the first height, where the first height is the target height;

[0286] If the first height is greater than the second height, determining a second angle of the three-axis gimbal based on the second height;

[0287] The three-axis gimbal is controlled according to the second angle to raise the vehicle-mounted camera to a second height, where the second height is the target height.

[0288] Fig.16 This is a schematic diagram of the structure of the electronic device provided by the present disclosure. Fig.16 As shown, the electronic device 160 provided in this embodiment includes: at least one processor 1601 and a memory 1602. Optionally, the device 160 also includes a communication component 1603. The processor 1601, the memory 1602 and the communication component 1603 are connected via a bus 1604.

[0289] In a specific implementation process, at least one processor 1601 executes the computer-executable instructions stored in the memory 1602, so that at least one processor 1601 executes the above method.

[0290] The specific implementation process of the processor 1601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0291] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0292] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0293] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present method is not limited to only one bus or one type of bus.

[0294] The method also provides a computer program product, comprising a computer program, which implements the above method when executed by a processor.

[0295] The method also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0296] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0297] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0298] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0299] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0300] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0301] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0302] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0303] Finally, it should be noted that those skilled in the art will easily think of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any modification, use or adaptive change of the present invention, which follows the general principles of the present invention and includes common knowledge or customary technical means in the technical field not disclosed by the present invention, is not limited to the precise structure described above and shown in the accompanying drawings, and can be variously modified and changed without departing from its scope. The scope of the present invention is limited only by the attached claims. The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by technicians in the technical field on the basis of the present invention are all within the protection scope of the present invention.

Claims

1. A vehicle camera control method, characterized in that: Applied to a vehicle, wherein a vehicle-mounted camera is installed on the top of the vehicle, the method comprises: After the vehicle turns on the automatic driving function, the vehicle-mounted camera is raised to a target height according to the vehicle state information and surrounding environment information of the vehicle, and the target height is the maximum height to which the vehicle-mounted camera can be raised; Determine a key observation area of ​​the vehicle camera according to a current image captured by the vehicle camera and a path to be traveled planned by the automatic driving function, wherein the key observation area includes a target position point of the path to be traveled; Determining an optimized image center region according to a plurality of interest points in the current image, wherein the interest points are elements in the road related to the autonomous driving function, and an area of ​​the optimized image center region is larger than an area of ​​the key observation region; According to the central area of ​​the optimized image and the key observation area, the posture of the vehicle-mounted camera is adjusted to adjust the central area of ​​the optimized image captured by the vehicle-mounted camera to cover the key observation area.

2. The method according to claim 1, characterized in that The step of determining the optimized image center area according to the multiple interest points in the current image includes: determining a plurality of initial interest points in the current image; Determining a priority of each initial point of interest according to a preset condition, wherein the priority of each initial point of interest is proportional to the importance of the initial point of interest to the autonomous driving function; According to the priority of each initial point of interest, a target point of interest is determined from all the initial points of interest, wherein the priority of the target point of interest is greater than the priorities of other initial points of interest; Calculate the sum of the distances from each pixel in the current image to all target points of interest; The distance and the minimum pixel point are taken as the center to determine the center area of ​​the optimized image.

3. The method according to claim 2, characterized in that The calculating the sum of the distances from each pixel point in the current image to all target interest points includes: According to the priority of each target point of interest, the weight of each target point of interest is determined, and the weight of each target point of interest is proportional to the priority; According to the weight of each target point of interest, the weighted sum of distances from each pixel point in the current image to all target points of interest is calculated; Accordingly, taking the distance and the smallest pixel point as the center to determine the center area of ​​the optimized image includes: The weighted distance and the smallest pixel point are taken as the center to determine the center area of ​​the optimized image.

4. The method according to any one of claims 1 to 3, characterized in that: The determining of the key observation area of ​​the vehicle-mounted camera according to the current image captured by the vehicle-mounted camera and the to-be-traveled path planned by the automatic driving function includes: Determine the camera coordinates of the target location point in the camera coordinate system according to the path to be traveled and the navigation map information; Projecting the camera coordinates into an image coordinate system to determine the pixel coordinates of the target location point; The key observation area is determined according to the pixel points corresponding to the pixel coordinates.

5. The method according to any one of claims 1 to 3, characterized in that: The vehicle-mounted camera is arranged on a three-axis gimbal, the three-axis gimbal comprises a rotating platform and a rotating arm arranged on the rotating platform, and the vehicle-mounted camera is arranged on the rotating arm; The three-axis gimbal includes a first axis, a second axis and a third axis; The first axis is perpendicular to the rotating table and is used to adjust the rotation angle of the rotating table; The second axis is perpendicular to the rotating arm and is used to adjust the angle between the rotating arm and the rotating table; The third axis is perpendicular to the optical axis of the vehicle-mounted camera and is used to adjust the pitch angle of the vehicle-mounted camera.

6. The method according to claim 5, characterized in that The adjusting the posture of the vehicle-mounted camera according to the optimized image center area and the key observation area includes: If the key observation area is not completely within the center area of ​​the optimized image, adjusting the center area of ​​the optimized image according to the key observation area to obtain a new center area of ​​the optimized image, wherein the key observation area is within the center area of ​​the new optimized image; Calculating the depth of the central area of ​​the new optimized image; Determine the target angle of the first axis and the target angle of the third axis according to the depth information and the center point of the central area of ​​the optimized image, the external parameters, the internal reference focal length and the distortion parameters of the vehicle-mounted camera; The three-axis gimbal is controlled according to the target angle of the first axis and the target angle of the third axis to adjust the position and posture of the vehicle-mounted camera.

7. The method according to claim 6, characterized in that The calculating the depth of the new optimized image central area comprises: The depth of the central area of ​​the new optimized image is determined according to the depth of each point of interest in the central area of ​​the new optimized image and the depth of the key observation area.

8. The method according to claim 5, characterized in that The step of raising the vehicle-mounted camera to a target height according to the vehicle state information and surrounding environment information of the vehicle includes: Calculating the target air resistance of the three-axis gimbal according to the vehicle state information and the drag coefficient of the vehicle; Determine a first angle and a first height corresponding to the target air resistance according to a preset resistance mapping table, wherein the resistance mapping table is used to indicate the maximum angle of the second axis corresponding to different air resistances and the highest height of the vehicle-mounted camera; Determine a second height according to the surrounding environment information, where the second height is the highest height of the vehicle-mounted camera limited by environmental obstacles; The three-axis gimbal is controlled according to the first height, the second height, and the first angle to raise the vehicle-mounted camera to the target height, where the target height is the first height or the second height.

9. The method according to claim 8, characterized in that The controlling the three-axis gimbal according to the first height, the second height and the first angle to raise the vehicle-mounted camera to the target height includes: If the first height is less than or equal to the second height, controlling the three-axis gimbal according to the first angle to raise the vehicle-mounted camera to a first height, where the first height is the target height; If the first height is greater than the second height, determining a second angle of the three-axis gimbal based on the second height; The three-axis gimbal is controlled according to the second angle to raise the vehicle-mounted camera to a second height, where the second height is the target height.

10. A vehicle-mounted camera control device, characterized in that: Applied to a vehicle, a vehicle camera is installed on the top of the vehicle, and the device comprises: A first processing module is used to raise the vehicle-mounted camera to a target height according to the vehicle state information and surrounding environment information of the vehicle after the vehicle turns on the automatic driving function, and the target height is the maximum height to which the vehicle-mounted camera can be raised; A second processing module is used to determine a key observation area of ​​the vehicle camera according to a current image captured by the vehicle camera and a path to be traveled planned by the automatic driving function, wherein the key observation area includes a target position point of the path to be traveled; a third processing module, configured to determine an optimized image central region according to a plurality of interest points in the current image, wherein the interest points are elements in the road related to the autonomous driving function, and an area of ​​the optimized image central region is larger than an area of ​​the key observation region; The fourth processing module is used to adjust the posture of the vehicle-mounted camera according to the central area of ​​the optimized image and the key observation area, so as to adjust the central area of ​​the optimized image collected by the vehicle-mounted camera to cover the key observation area.

11. An electronic device characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.

Images