Vehicle camera control method, device, equipment, storage medium and product

Abstract

The embodiment of the present disclosure provides a kind of vehicle camera control method, device, equipment, storage medium and product. Wherein the method is applied to vehicle, the top of the vehicle is equipped with vehicle camera, method includes: after the automatic driving function of vehicle is opened, according to the ego state information of vehicle and surrounding environment information, vehicle camera is elevated to target height;According to the current image collected by vehicle camera and the path to be driven planned by automatic driving function, the key observation area of vehicle camera is determined;According to multiple points of interest in current image, the optimization image center area is determined;According to optimization image center area and key observation area, the pose of vehicle camera is adjusted, to adjust the optimization image center area collected by vehicle camera to cover key observation area.The technical scheme reaches the effect of improving the accuracy of complex road condition detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automobile accessories, in particular to a vehicle-mounted camera control method, device, equipment, storage medium and product. BACKGROUND

[0002] With the development of automobile automatic driving technology, in order to improve the flexibility and intelligence of automatic driving, it is necessary to intelligently analyze the environment information collected by the vehicle during current driving, so as to effectively control the automatic driving of the vehicle. Among them, the collection of environment information by the perception system of the vehicle is an important part of the automatic driving of the vehicle.

[0003] In the prior art, the visual image and multiple sensors are combined to realize multi-directional road condition perception of automatic driving. When the vehicle is loaded with the perception system of automatic driving, multiple cameras are combined, and millimeter wave radar and laser radar are added to construct the perception system of automatic driving of the vehicle.

[0004] However, the prior art has the technical problem of inaccurate detection of complex road conditions. SUMMARY

[0005] One of the purposes of the present application is to provide a vehicle-mounted camera control method to solve the problem of inaccurate detection of complex road conditions by the perception system in the prior art; the second purpose is to provide a vehicle-mounted camera control device; the third purpose is to provide an electronic device; the fourth purpose is to provide a computer readable storage medium; and the fifth purpose is to provide a computer program product.

[0006] In order to achieve the above purposes, the technical solution adopted by the present application is as follows:

[0007] A vehicle-mounted camera control method applied to a vehicle, the top of the vehicle being provided with a vehicle-mounted camera, the method comprising:

[0008] After the vehicle starts the automatic driving function, the vehicle-mounted camera is raised to a target height according to the ego state information of the vehicle and the surrounding environment information, the target height being the maximum height that the vehicle-mounted camera can be raised to;

[0009] According to the current image collected by the vehicle-mounted camera and the to-be-traveled path planned by the automatic driving function, a key observation area of the vehicle-mounted camera is determined, the key observation area including a target position point of the to-be-traveled path;

[0010] According to a plurality of interest points in the current image, an optimized image center area is determined, the interest points being elements in the road related to the automatic driving function, and the area of the optimized image center area being greater than that of the key observation area;

[0011] According to the optimization image center area and the key observation area, the pose of the vehicle-mounted camera is adjusted, so as to adjust the optimization image center area collected by the vehicle-mounted camera to cover the key observation area.

[0012] According to the above technical means, in the prior art, due to the limitation of camera data collection of the perception system, the detection of the perception system for complex road conditions is inaccurate. Therefore, when the camera collects data, the adjustment of the camera can be realized based on the information in the current image, so as to ensure that the camera can collect as many elements related to the automatic driving function in the road as possible, so as to help the automatic driving function focus on these elements. First, according to the ego state information of the vehicle and the surrounding environment information, the vehicle-mounted camera is adjusted to a target height, so as to ensure that the height of the vehicle-mounted camera is the maximum height that can be reached in the current environment. According to the current image collected by the vehicle-mounted camera and the to-be-traveled path planned by the automatic driving, the key observation area of the vehicle-mounted camera is determined; and according to the plurality of interest points in the current image, the optimization image center area is determined; and the pose of the vehicle-mounted camera is adjusted based on the optimization image center area and the key observation area. Compared with the prior art, the highest height that the camera can reach is determined first, so as to ensure that the field of view of the camera is not limited by the height, and the current image obtained by the camera is used to adjust the image center area of the key observation area that needs to be collected, so as to control the multi-direction rotation of the camera and realize the multi-range information collection, thereby achieving the technical effect of improving the detection accuracy of complex road conditions.

[0013] Further, according to the plurality of interest points in the current image, the optimization image center area is determined, including:

[0014] determining a plurality of initial interest points in the current image;

[0015] determining the priority of each initial interest point according to a preset condition, the priority of each initial interest point being proportional to the importance of the initial interest point to the automatic driving function;

[0016] determining a target interest point from all the initial interest points according to the priority of each initial interest point, the priority of the target interest point being greater than the priority of other initial interest points;

[0017] calculating the sum of distances of each pixel point in the current image to all the target interest points;

[0018] determining the optimization image center area with the pixel point with the smallest sum of distances as the center.

[0019] According to the above technical means, the determination of the center region of the optimized image is performed by using a plurality of target points of interest in the current image, so that there is sufficient information in the field of view of the camera that can identify the road conditions, and the accuracy of the road condition detection is further improved.

[0020] Further, the sum of distances of each pixel point in the current image to all target points of interest is calculated, including:

[0021] According to the priority of each target point of interest, the weight of each target point of interest is determined, and the weight of each target point of interest is proportional to the priority;

[0022] According to the weight of each target point of interest, the weighted distance sum of each pixel point in the current image to all target points of interest is calculated.

[0023] Accordingly, the pixel point with the minimum distance sum is taken as the center to determine the center region of the optimized image, including:

[0024] The pixel point with the minimum weighted distance sum is taken as the center to determine the center region of the optimized image.

[0025] According to the above technical means, the weight of the priority of the plurality of points of interest is determined, and the weighted calculation is performed by using the weight of the point of interest, so that a pixel point closest to each point of interest is obtained as the center of the center region of the optimized image, thereby determining the center region of the optimized image.

[0026] Further, according to the current image collected by the vehicle-mounted camera and the to-be-traveled path planned by the automatic driving function, the key observation region of the vehicle-mounted camera is determined, including:

[0027] According to the to-be-traveled path and the navigation map information, the camera coordinates of the target position point in the camera coordinate system are determined.

[0028] The camera coordinates are projected to the image coordinate system to determine the pixel coordinates of the target position point.

[0029] According to the pixel point corresponding to the pixel coordinates, the key observation region is determined.

[0030] According to the above technical means, the key observation region of the camera is determined through the to-be-traveled path of the vehicle and the navigation map information, so that the main basis for the camera adjustment is determined.

[0031] Further, the vehicle-mounted camera is arranged on a three-axis gimbal, the three-axis gimbal includes a rotating table and a rotating arm arranged on the rotating table, and the vehicle-mounted camera is arranged on the rotating arm.

[0032] The three-axis gimbal includes a first axis, a second axis and a third axis.

[0033] The first shaft is perpendicular to the rotating table and is used to adjust the rotation angle of the rotating table.

[0034] The second shaft is perpendicular to the rotating arm and is used to adjust the included angle between the rotating arm and the rotating table.

[0035] The third shaft is perpendicular to the optical axis of the vehicle-mounted camera and is used to adjust the pitch angle of the vehicle-mounted camera.

[0036] According to the above technical means, the height and angle of the vehicle-mounted camera are adjusted through the design of the three-axis gimbal, the detection range of the camera is expanded, and the accuracy of road condition detection is improved.

[0037] According to the optimization of the image center area and the key observation area, the pose of the vehicle-mounted camera is adjusted, including:

[0038] If the key observation area is not completely in the optimized image center area, the optimized image center area is adjusted according to the key observation area, a new optimized image center area is obtained, and the key observation area is in the new optimized image center area;

[0039] The depth of the new optimized image center area is calculated.

[0040] According to the depth information and center point of the optimized image center area, the external parameters of the vehicle-mounted camera, the internal focal length and distortion parameters, the target angle of the first shaft and the target angle of the third shaft are determined.

[0041] According to the target angle of the first shaft and the target angle of the third shaft, the three-axis gimbal is controlled to adjust the pose of the vehicle-mounted camera.

[0042] According to the above technical means, the optimized image center area of the camera is adjusted based on the key observation area, so as to control the vehicle-mounted camera to adjust in the direction with more road condition information.

[0043] Further, the depth of the new optimized image center area is calculated, including:

[0044] According to the depth of each point of interest in the new optimized image center area and the depth of the key observation area, the depth of the new optimized image center area is determined.

[0045] According to the above technical means, the depth of the optimized image center area is calculated by using the depth of the point of interest and the depth of the key observation area, so as to determine the adjustment required by the camera to reach the target image center area.

[0046] Further, according to the self-vehicle state information and the surrounding environment information of the vehicle, the vehicle-mounted camera is raised to a target height, including:

[0047] According to the self-vehicle state information of the vehicle and the wind resistance coefficient, a target air resistance of the three-axis gimbal is calculated;

[0048] According to a preset resistance mapping table, a first included angle and a first height corresponding to the target air resistance are determined, the resistance mapping table being used to represent a maximum included angle of the second axis corresponding to different air resistances and a maximum height of the vehicle-mounted camera;

[0049] According to the surrounding environment information, a second height is determined, the second height being a maximum height of the vehicle-mounted camera limited by an environmental obstacle;

[0050] According to the first height, the second height and the first included angle, the three-axis gimbal is controlled to raise the vehicle-mounted camera to a target height, the target height being the first height or the second height.

[0051] According to the above technical means, the height of the camera is determined by calculating the air resistance, so as to avoid excessive air resistance from damaging the vehicle-mounted camera.

[0052] Further, according to the first height, the second height and the first included angle, the three-axis gimbal is controlled to raise the vehicle-mounted camera to a target height, including:

[0053] If the first height is less than or equal to the second height, the three-axis gimbal is controlled according to the first included angle to raise the vehicle-mounted camera to the first height, the first height being the target height;

[0054] If the first height is greater than the second height, a second included angle of the three-axis gimbal is determined based on the second height;

[0055] The three-axis gimbal is controlled according to the second included angle to raise the vehicle-mounted camera to the second height, the second height being the target height.

[0056] According to the above technical means, the height limit is determined based on the obstacle, so as to avoid damage of the obstacle to the vehicle-mounted camera.

[0057] A vehicle-mounted camera control device, characterized in that it is applied to a vehicle, a top of the vehicle being provided with a vehicle-mounted camera, and the device comprising:

[0058] A first processing module is configured to, after the vehicle starts an automatic driving function, raise the vehicle-mounted camera to a target height according to self-vehicle state information of the vehicle and surrounding environment information, the target height being a maximum height to which the vehicle-mounted camera can be raised;

[0059] A second processing module is configured to determine a key observation area of the vehicle-mounted camera according to a current image collected by the vehicle-mounted camera and a to-be-traveled path planned by the automatic driving function, the key observation area including a target position point of the to-be-traveled path;

[0060] The third processing module is configured to determine an optimized image center region according to a plurality of interest points in the current image, the interest points being elements in the road related to the automatic driving function, and the area of the optimized image center region being greater than the area of the key observation region;

[0061] The fourth processing module is configured to adjust the pose of the vehicle-mounted camera according to the optimized image center region and the key observation region, so as to adjust the optimized image center region collected by the vehicle-mounted camera to cover the key observation region.

[0062] Further, the third processing module is further configured to:

[0063] determine a plurality of initial interest points in the current image;

[0064] determine the priority of each initial interest point according to a preset condition, the priority of each initial interest point being proportional to the importance of the initial interest point to the automatic driving function;

[0065] determine a target interest point from all the initial interest points according to the priority of each initial interest point, the priority of the target interest point being greater than the priority of other initial interest points;

[0066] calculate the distance sum of each pixel point in the current image to all the target interest points;

[0067] determine the optimized image center region with the pixel point with the minimum distance sum as the center.

[0068] Further, the third processing module is further configured to:

[0069] determine the weight of each target interest point according to the priority of each target interest point, the weight of each target interest point being proportional to the priority;

[0070] calculate the weighted distance sum of each pixel point in the current image to all the target interest points according to the weight of each target interest point;

[0071] Correspondingly, the optimized image center region with the pixel point with the minimum distance sum as the center includes:

[0072] the optimized image center region with the pixel point with the minimum weighted distance sum as the center.

[0073] Further, the second processing module is further configured to:

[0074] determine the camera coordinates of the target position point in the camera coordinate system according to the to-be-traveled path and the navigation map information;

[0075] project the camera coordinates to the image coordinate system to determine the pixel coordinates of the target position point;

[0076] According to the pixel point corresponding to the pixel coordinate, the key observation region is determined.

[0077] Further, the vehicle-mounted camera is arranged on a three-axis holder, and the three-axis holder comprises a rotating table and a rotating arm arranged on the rotating table, and the vehicle-mounted camera is arranged on the rotating arm.

[0078] The three-axis holder comprises a first axis, a second axis and a third axis.

[0079] The first axis is perpendicular to the rotating table and is used for adjusting the rotation angle of the rotating table.

[0080] The second axis is perpendicular to the rotating arm and is used for adjusting the included angle between the rotating arm and the rotating table.

[0081] The third axis is perpendicular to the optical axis of the vehicle-mounted camera and is used for adjusting the pitch angle of the vehicle-mounted camera.

[0082] The fourth processing module is further configured to:

[0083] If the key observation region is not completely in the optimized image center region, the optimized image center region is adjusted according to the key observation region, a new optimized image center region is obtained, and the key observation region is in the new optimized image center region.

[0084] The depth of the new optimized image center region is calculated.

[0085] According to the depth information and the center point of the optimized image center region, the external parameters of the vehicle-mounted camera, the internal focal length and the distortion parameters, the target angle of the first axis and the target angle of the third axis are determined.

[0086] The three-axis holder is controlled according to the target angle of the first axis and the target angle of the third axis, so as to adjust the pose of the vehicle-mounted camera.

[0087] According to the above technical means, the fourth processing module adjusts the optimized image center region of the camera based on the key observation region, so as to control the vehicle-mounted camera to adjust in the direction of the road condition information.

[0088] Further, the fourth processing module is further configured to:

[0089] According to the depth of each point of interest in the new optimized image center region and the depth of the key observation region, the depth of the new optimized image center region is determined.

[0090] Further, the first processing module is further configured to:

[0091] According to the self-vehicle state information of the vehicle and the wind resistance coefficient, the target air resistance of the three-axis holder is calculated.

[0092] According to the preset resistance mapping table, a first angle and a first height corresponding to the target air resistance are determined, and the resistance mapping table is used to represent different air resistances corresponding to the maximum angle of the second shaft and the highest height of the vehicle-mounted camera.

[0093] According to the surrounding environment information, a second height is determined, and the second height is the highest height of the vehicle-mounted camera limited by the environmental obstacles.

[0094] According to the first height, the second height and the first angle, the three-axis holder is controlled to raise the vehicle-mounted camera to a target height, and the target height is the first height or the second height.

[0095] Further, the first processing module is further used for:

[0096] If the first height is less than or equal to the second height, the three-axis holder is controlled according to the first angle to raise the vehicle-mounted camera to the first height, and the first height is the target height.

[0097] If the first height is greater than the second height, a second angle of the three-axis holder is determined based on the second height.

[0098] The three-axis holder is controlled according to the second angle to raise the vehicle-mounted camera to the second height, and the second height is the target height.

[0099] An electronic device comprises a memory and a processor.

[0100] The memory stores computer execution instructions.

[0101] The processor executes the computer execution instructions stored in the memory, so that the processor executes any method in the above vehicle-mounted camera control method further.

[0102] A computer readable storage medium, the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to realize any method in the above vehicle-mounted camera control method and the vehicle-mounted camera control method further.

[0103] A computer program product comprises a computer program, and the computer program is executed by the processor to realize any method in the above vehicle-mounted camera control method and the vehicle-mounted camera control method further.

[0104] The beneficial effects of the present application are:

[0105] (1) In the control process of the vehicle-mounted camera, a hierarchical control method is adopted, first, based on the self-vehicle state information and the surrounding environment information of the autonomous vehicle, the vehicle-mounted camera is controlled to reach the highest height that can be reached at present, so as to improve the field of view of the vehicle-mounted camera; then, based on the determination of the optimized image center area, the angle of the vehicle-mounted camera is adjusted. By using the hierarchical calculation method, the vehicle-mounted camera is controlled in stages in height and angle, avoiding the calculation resources loaded on the vehicle to perform multiple types of calculation reasoning at the same time, thereby reducing the demand for real-time computing resources.

[0106] (2) In the control process of the vehicle-mounted camera, a multi-target final method is adopted to control the camera, a plurality of interest points in the current image and a key observation area of the current image are obtained, a multi-target tracking is realized based on the plurality of interest points, and the determination of the optimized image center area is realized based on the plurality of interest points and the key observation area. By controlling the camera in height and angle, the field of view of the camera is expanded; at the same time, based on the plurality of interest points and the key observation area, the camera is adjusted to ensure that the image center area observed by the camera contains a plurality of interest points and a key observation area that needs to be focused on during vehicle driving, thereby improving the content of the key road information in the image detected by the camera, and improving the detection accuracy of the vehicle-mounted camera. BRIEF DESCRIPTION OF DRAWINGS

[0107] Figure 1 The side view and top view of the three-axis gimbal when the vehicle-mounted camera provided by the present disclosure is in a retracted state;

[0108] Figure 2 The side view and top view of the three-axis gimbal when the vehicle-mounted camera provided by the present disclosure is in an extended state;

[0109] Figure 3 The schematic diagram of three-axis motion in the three-axis gimbal provided by the present disclosure;

[0110] Figure 4 The structural schematic diagram of the vehicle-mounted camera control system for autonomous driving provided by the present disclosure;

[0111] Figure 5 The flowchart of the vehicle-mounted camera control method provided by the present disclosure Figure 1 ;

[0112] Figure 6 The flowchart of the vehicle-mounted camera control method provided by the present disclosure Figure 2 ;

[0113] Figure 7 The flowchart of the vehicle-mounted camera optimization image center area determination method provided by the present disclosure;

[0114] Figure 8 Flowchart of the vehicle-mounted camera control method provided by the present disclosure Figure 3 ;

[0115] Figure 9 Diagram for calculating the adjustment angle of a three-axis gimbal provided by the present disclosure

[0116] Figure 10 Diagram for calculating the adjustment angle of a three-axis gimbal provided by the present disclosure

[0117] Figure 11 Diagram of the mounting mode of a three-axis gimbal provided by the present disclosure Figure 1 ;

[0118] Figure 12 Diagram of the mounting mode of a three-axis gimbal provided by the present disclosure Figure 2 ;

[0119] Figure 13 Diagram of the key observation area, point of interest, and optimized image center area provided by the present disclosure

[0120] Figure 14 Diagram of the movement of the camera from the initial image area to the optimized image center area based on the center coordinate optimal solution provided by the present disclosure

[0121] Figure 15 Diagram of the structure of the vehicle-mounted camera control device provided by the present disclosure

[0122] Figure 16 Diagram of the structure of the electronic device provided by the present disclosure. DETAILED DESCRIPTION

[0123] Other advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure of the present specification. The present application can also be implemented or applied by different specific embodiments, and the details in the present specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, and are not intended to limit the protection scope of the present application.

[0124] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and only show the components related to the present application in the diagrams, not the number, shape, and size of the components when actually implemented. The shape, number, and proportion of each component when actually implemented can be arbitrarily changed, and the layout pattern of the components can be more complex.

[0125] In the prior art, for the perception system arranged on the automatic driving vehicle, a visual image and a variety of sensors are combined to realize multi-directional road condition perception of automatic driving. When the vehicle is loaded with the perception system of automatic driving, a combination of multiple cameras is usually adopted, and millimeter wave radar and laser radar are added to construct the perception system of vehicle automatic driving.

[0126] However, such a perception system cannot solve some problems in complex urban or garage scenarios. For example, in the scenario of a mountain city, there is a situation that the traffic light or signboard is installed relatively high, and the left-turn waiting area of the road is too close to the traffic light or signboard; at this time, the traffic light or signboard is outside the vertical field of view (FOV) of the camera of the vehicle perception system, which causes the vehicle to be unable to pass through such a road section by relying on automatic driving. Thus, there is the technical problem of inaccurate complex road condition detection in the prior art.

[0127] To solve the above technical problems, the present method proposes the following technical concept: in the prior art, the data collection of the camera of the perception system is limited, which causes the perception system to be inaccurate in detecting complex road conditions. Therefore, when the camera collects data, the camera can be adjusted based on the information in the current image to ensure the collection efficiency of the camera. First, the vehicle-mounted camera is adjusted to a target height according to the self-vehicle state information and the surrounding environment information of the vehicle, to ensure that the height of the vehicle-mounted camera is the maximum height that can be reached in the current environment. The key observation area of the vehicle-mounted camera is determined according to the current image collected by the vehicle-mounted camera and the to-be-traveled path planned by automatic driving; and the optimized image center area is determined according to a plurality of interest points in the current image; and the pose of the vehicle-mounted camera is adjusted based on the optimized image center area and the key observation area. Compared with the prior art, the present method first determines the maximum height that the camera can reach to ensure that the field of view of the camera is not limited by the height, and simultaneously adjusts the image center area for the key observation area that needs to be collected by using the current image obtained by the camera, to control the multi-directional rotation of the camera and realize multi-range information collection; thus, the technical effect of improving the accuracy of complex road condition detection is achieved.

[0128] The technical solutions of the present method and how the technical solutions of the present method solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present method will be described below with reference to the accompanying drawings.

[0129] Figures 1-3 A structure diagram of a three-axis gimbal provided with a vehicle-mounted camera is provided for the present disclosure. As shown in FIG. 1, the three-axis gimbal is provided with a vehicle-mounted camera, and the three-axis gimbal is provided with a first axis, a second axis and a third axis. The first axis is arranged to be parallel to the longitudinal direction of the vehicle, the second axis is arranged to be parallel to the lateral direction of the vehicle, and the third axis is arranged to be parallel to the vertical direction of the vehicle. The vehicle-mounted camera is arranged on the first axis, the second axis and the third axis. Figures 1-3As shown, the three-axis gimbal includes: a rotary table 101, a rotating arm 102, a vehicle-mounted camera 103, a first axis 104, a second axis 105, and a third axis 106. The first axis 104 is perpendicular to the rotary table 101 and is used to adjust the rotation angle of the rotary table 101. The second axis 105 is parallel to and perpendicular to the rotating arm 102 and is used to adjust the angle between the rotating arm 102 and the rotary table 101. The third axis 106 is perpendicular to the optical axis of the vehicle-mounted camera 103 and is used to adjust the pitch angle of the vehicle-mounted camera 103. Figure 1 The vehicle-mounted camera provided in this disclosure is a side view and a top view of the three-axis gimbal when it is in the retracted state. Figure 2 The vehicle-mounted camera provided in this disclosure is shown in side and top views of a three-axis gimbal in its extended position. Figure 3 This is a schematic diagram of the three-axis motion in the three-axis gimbal provided in this disclosure.

[0130] In such Figures 1-3 Based on the three-axis gimbal with an in-vehicle camera mentioned above, this method proposes a control system for an in-vehicle camera used in autonomous driving. Figure 4 This is a schematic diagram of the structure of an onboard camera control system for autonomous driving provided in this disclosure. Figure 4 As shown, the system includes: a first axis assembly 401, a second axis assembly 402, a third axis assembly 403, a vision perception control component 404, and an autonomous driving planning control component 405. The first axis assembly 401 includes: a motor and its drive, a rotary table, and angle sensors corresponding to the first axis and the rotary table; the second axis assembly 402 includes: a motor and its drive, a rotating arm, and angle sensors corresponding to the second axis and the rotating arm; the third axis assembly 403 includes: a motor and its drive, a camera unit, and angle sensors corresponding to the third axis and the camera unit. The autonomous driving planning control component 405 acquires perception or cognitive information sent by the vision perception control component 404, generates autonomous driving planning information, and transmits the autonomous driving planning information to the vision perception control component 404; the vision perception control component 404 generates perception or cognitive information based on image information acquired by the third axis assembly; the vision perception control component 404 acquires angle information measured by the first to third axis assemblies, and combines the angle information and the autonomous driving planning information to generate control commands to control the operation of the first to third axis assemblies.

[0131] Based on the above embodiments, this method proposes a vehicle-mounted camera control method. This method is applied to a vehicle, and a vehicle-mounted camera is installed on the roof of the vehicle. Figure 5 A flowchart illustrating the vehicle-mounted camera control method provided in this disclosure. Figure 1 ,like Figure 5 As shown, the method includes:

[0132] S501, after the vehicle starts the automatic driving function, according to the self-vehicle state information and the surrounding environment information of the vehicle, the vehicle-mounted camera is raised to a target height.

[0133] In this step, the target height is the maximum height that the vehicle-mounted camera can be raised to.

[0134] It should be noted that the self-vehicle state information mentioned in this step can be: the current driving speed of the vehicle, the yaw rate of the vehicle, the body shape of the vehicle, the windward area of the vehicle-mounted camera, and the engine power of the vehicle.

[0135] The surrounding environment information can be: the current wind speed, the current air pressure, the current terrain, and the current traffic situation.

[0136] The purpose of raising the vehicle-mounted camera to the target height according to the self-vehicle state information and the surrounding environment information in this step is to improve the field of view of the vehicle-mounted camera, and at the same time to ensure that the vehicle-mounted camera is not affected by the driving resistance of the vehicle and the environment.

[0137] S502, according to the current image collected by the vehicle-mounted camera and the to-be-traveled path planned by the automatic driving function, determine the key observation area of the vehicle-mounted camera.

[0138] In this step, the key observation area includes a target position point of the to-be-traveled path.

[0139] Optionally, one possible implementation of determining the key observation area of the vehicle-mounted camera is:

[0140] S5021, according to the to-be-traveled path and the navigation map information, determine the camera coordinates of the target position point in the camera coordinate system.

[0141] In this step, the target position point refers to a point taken M times of the time interval in front of the to-be-traveled path, where M is greater than 0. For example, when the vehicle is driving normally, based on the navigation map information of the vehicle during automatic driving, the current speed of the vehicle is determined to calculate the unit time interval of the vehicle, and the position of the vehicle on the to-be-traveled path when M times of the time interval is calculated is determined as the target position point. When the vehicle is driving at low speed, a minimum distance value is determined according to the blind area of the vehicle, and M times of the time interval is calculated based on the distance value. When M times of the time interval does not exceed the distance value, the corresponding target position point on the to-be-traveled path is determined.

[0142] After the target position point is determined, the manner of determining the camera coordinates of the target position point in the camera coordinate system can be: first, determining the position of the target position point in the world coordinate system, which can be obtained by a sensor. The target position point in the world coordinate system is mapped to the camera coordinate system based on the coordinate system conversion, and the corresponding camera coordinates of the target position point are determined, and the camera coordinates are a three-dimensional coordinate.

[0143] S5022, projecting the camera coordinates to the image coordinate system to determine the pixel coordinates of the target position point.

[0144] In this step, the manner of projecting the camera coordinates to the image coordinates can be: establishing a projection of the camera coordinate system to the image plane, and determining the camera coordinates as the pixel coordinates of the image plane based on the projection change and the distortion conversion.

[0145] S5023, determining the key observation area according to the pixel point corresponding to the pixel coordinates.

[0146] In this step, the manner of determining the key observation area can be: based on the preset area size, establishing the key observation area with the pixel coordinates as the center. The key observation area can be a rectangle or other shapes.

[0147] It should be noted that the preset area size cannot exceed the maximum image acquisition size of the vehicle-mounted camera.

[0148] S503, determining the optimized image center area according to the plurality of interest points in the current image.

[0149] In this step, the interest points are elements related to the automatic driving function in the road, and the area of the optimized image center area is greater than the area of the key observation area.

[0150] Optionally, one possible implementation method of determining the optimized image center area based on the plurality of interest points is: using the minimum value calculation method to calculate the point with the minimum distance sum to the plurality of interest points, and determining the point as the center point of the optimized image center area. Based on the preset size of the optimized image center area, the optimized image center area is established. How to determine the optimized image center area is further explained in the following embodiments, which will not be described here. Figure 7 Figure 7

[0151] S504, adjusting the pose of the vehicle-mounted camera according to the optimized image center area and the key observation area, so as to adjust the optimized image center area collected by the vehicle-mounted camera to cover the key observation area.

[0152] In this step, the method for adjusting the pose of the vehicle-mounted camera based on the optimized image center region and the key observation region can be as follows: First, determine whether the optimized image center region completely covers the key observation region. If it completely covers the key observation region, determine the center point of the optimized image center region as the position point where the principal optical axis of the vehicle-mounted camera needs to be adjusted; if it does not completely cover the key observation region, adjust the optimized image center region to completely cover the key observation region, and then determine the center point of the new optimized image center region after adjustment as the position point where the principal optical axis of the vehicle-mounted camera needs to be adjusted.

[0153] It should be noted that the pose adjustment of the vehicle-mounted camera in this step is as follows: Figure 8 The embodiments shown will be explained in further detail, without going into unnecessary repetition.

[0154] This embodiment proposes a vehicle-mounted camera control method. Based on the vehicle's status information and surrounding environment information, the vehicle-mounted camera is adjusted to a target height, ensuring that the camera's height is the maximum achievable height under the current environment. Based on the current image captured by the vehicle-mounted camera and the planned driving path for autonomous driving, the key observation area of ​​the vehicle-mounted camera is determined; and an optimized image center region is determined based on multiple points of interest in the current image. The pose of the vehicle-mounted camera is then adjusted based on the optimized image center region and the key observation area. Compared to existing technologies, this method first determines the maximum height the camera can reach, ensuring that the camera's field of view is not limited by height. Simultaneously, it uses the current image acquired by the camera to adjust the image center region for the key observation area that needs to be captured; multiple points of interest in the current image are used as observation points, combined with the key observation area to achieve multi-target tracking adjustment, ensuring that the camera's image center region covers the key observation area and highly important points of interest; based on this, the multi-directional rotation of the camera is controlled, ensuring that the camera's field of view covers multiple important aspects of the current road; thereby achieving the technical effect of improving the accuracy of complex road condition detection.

[0155] Based on the above embodiments, this method also provides a method for controlling a vehicle-mounted camera to reach a target height. Figure 6 A flowchart illustrating the vehicle-mounted camera control method provided in this disclosure. Figure 2 .like Figure 6 As shown, in Figure 5 Based on the illustrated embodiment, the method of raising the camera to the target height in S501 is described in detail, including:

[0156] S601. Calculate the target air resistance of the three-axis gimbal based on the vehicle's own status information and drag coefficient.

[0157] In this step, one possible way to calculate the target air resistance of the three-axis gimbal is as follows:

[0158] S6011, determine the vehicle state information, including vehicle speed, vehicle yaw rate and wind area of the three-axis gimbal.

[0159] S6012, based on vehicle speed, vehicle yaw rate, wind area of the three-axis gimbal and wind resistance coefficient, calculate the target air resistance of the three-axis gimbal. The yaw rate refers to the rotation speed of the vehicle around the vertical axis.

[0160] For example, the target air resistance is calculated using formula 1:

[0161]

[0162] Where ρ refers to air density, F d refers to the target air resistance, V refers to the vehicle speed, C d refers to the wind resistance coefficient, A refers to the wind area of the three-axis gimbal.

[0163] S602, according to the preset resistance mapping table, determine the first angle and the first height corresponding to the target air resistance.

[0164] In this step, the resistance mapping table is used to represent the maximum angle of the second axis corresponding to different air resistance and the highest height of the vehicle-mounted camera.

[0165] Where the maximum angle refers to the maximum angle between the rotating arm and the rotating table when the second axis of the rotating arm is rotated perpendicular to the rotating arm. The highest height of the vehicle-mounted camera refers to the highest distance of the highest point of the vehicle-mounted camera.

[0166] S603, according to the surrounding environment information, determine the second height, which is the highest height of the vehicle-mounted camera limited by the environmental obstacles.

[0167] In this step, the way to determine the second height according to the surrounding environment information can be:

[0168] S6031, determine the current image obtained by the vehicle-mounted camera, and based on the image detection algorithm, detect whether there is a height limit mark and / or a height limit building in the current image.

[0169] S6032, when there is a height limit mark and / or a height limit building, obtain the height limit information based on the sensing device of the vehicle-mounted camera, and determine the second height.

[0170] S604, according to the first height, the second height and the first angle, control the three-axis gimbal to raise the vehicle-mounted camera to the target height.

[0171] In this step, the target height is the first height or the second height.

[0172] Optionally, according to the first height, the second height and the first included angle, the three-axis gimbal is controlled to raise the vehicle-mounted camera to a possible implementation of the target height:

[0173] S6041, if the first height is less than or equal to the second height, the three-axis gimbal is controlled according to the first included angle to raise the vehicle-mounted camera to the first height, and the first height is the target height.

[0174] S6042, if the first height is greater than the second height, a second included angle of the three-axis gimbal is determined based on the second height.

[0175] S6043, the three-axis gimbal is controlled according to the second included angle to raise the vehicle-mounted camera to the second height, and the second height is the target height.

[0176] In this step, the way to control the vehicle-mounted camera to reach the target height can be: based on the current height of the vehicle-mounted camera and the target height, a smooth motion curve of the vehicle-mounted camera is planned by an interpolation method, the second axis and the third axis are controlled to keep the angle of the camera relative to the rotating table of the three-axis gimbal unchanged, and the camera is controlled to move to the target height.

[0177] In this embodiment, the maximum height that the vehicle-mounted camera can reach is determined by using the ego state information and the environment information; the motion of the second axis and the third axis of the three-axis gimbal is controlled to adjust the second included angle, so that the vehicle-mounted camera moves from the current height to the target height. By using the adjustment method with the maximum height, the rotation range of the vehicle-mounted camera is improved, thereby increasing the field of view change range of the vehicle-mounted camera, which is beneficial to multi-directional road condition information collection.

[0178] On the basis of the above-mentioned embodiments, the method further provides a vehicle-mounted camera optimal image center region determination method. Figure 7 A flowchart of the vehicle-mounted camera optimal image center region determination method provided by the present disclosure is shown in FIG. 7. Figure 7 As shown in the embodiment shown in FIG. 7, on the basis of the embodiment shown in FIG. 6, the determination of the optimal image center region in S503 is described in detail, and the method comprises: Figure 5

[0179] S701, a plurality of initial interest points in a current image are determined.

[0180] In this step, the way to determine the plurality of initial interest points in the current image can be:

[0181] Based on an image recognition algorithm, traffic signs and obstacles in the current image are detected, and these traffic signs and obstacles are determined as the plurality of initial interest points in the current image.

[0182] ​When the road is relatively empty, there are fewer objects of interest, and the point of interest is set to the point N times the distance in front of the vehicle center axis, so that the three-axis gimbal can be controlled to return to the correct position and face the front of the vehicle. Where N is greater than 0.

[0183] For example, the point at 2 times the distance is taken, and in the case of low speed of the vehicle, the minimum value of 30 meters can be taken for a general passenger car. The mathematical formula is expressed as the point at Max(2x vehicle speed, 30) distance, that is, the distance corresponding to 2 times the distance of not less than 30 meters; when the calculated 2 times the distance is greater than 30 meters, the point at 2 times the distance is selected, and when the calculated 2 times the distance is less than 30 meters, the point at 30 meters is selected.

[0184] Among them, the plurality of initial points of interest can be: traffic lights, dynamic targets and obstacles in the direction of the vehicle's to-be-traveled path.

[0185] S702, determine the priority of each initial point of interest according to the preset condition.

[0186] In this step, the priority of each initial point of interest is proportional to the importance of the initial point of interest to the automatic driving function.

[0187] The preset condition can be: the closer the distance between the autonomous vehicle and the point of interest, the higher the priority of the point of interest; the priority of the forbidden and warning traffic signs should be higher than that of the ordinary indication traffic signs; the priority of the dynamic target should be higher than that of the static target.

[0188] For example: if there are sharp turn indication traffic signs and dangerous animal habitat traffic signs in front, the priority of the sharp turn indication traffic signs should be higher than that of the dangerous animal habitat traffic signs.

[0189] S703, determine the target point of interest from all initial points of interest according to the priority of each initial point of interest.

[0190] In this step, the priority of the target point of interest is higher than that of the other initial points of interest.

[0191] Optionally, the way to determine the target point of interest from the plurality of initial points of interest can be: according to the priority, the plurality of initial points of interest are sorted from high to low, and the initial points of interest ranked before the preset number are selected as the target points of interest.

[0192] For example, when there are more than three initial points of interest, the three points of interest with the top three priorities are selected as the target points of interest.

[0193] S704, calculate the sum of distances of each pixel point in the current image to all target points of interest.

[0194] Optionally, one possible implementation of calculating the sum of distances is as follows:

[0195] S7041, determine the weight of each target interest point according to the priority of each target interest point, and the weight of each target interest point is proportional to the priority.

[0196] For example, assuming that there are multiple target interest points I, the pixel coordinates of each target interest point are I i = (x i ,y i ), and the weight of each target interest point is k i The higher the priority of each target interest point, the greater the weight of each target interest point.

[0197] S7042, calculate the weighted distance sum of each pixel point in the current image to all target interest points according to the weight of each target interest point.

[0198] For example, referring to the weight and pixel coordinates of the target interest point in S7041, the way to calculate the weighted distance sum of each pixel point in the current image to all target interest points can be as follows:

[0199] Assuming that there is a center point C1(x,y), calculate the weighted distance sum of the center point to each target interest point I i (x i ,y i ), and find the minimum value. As shown in formula 2:

[0200]

[0201] Where f(x) represents the weighted distance sum, n is the number of target interest points, k i is the weight of each target interest point, and n is a positive integer greater than 1; i is the i-th target interest point, (x i ,y i ) is the pixel coordinates of the i-th target interest point, and (x,y) is the pixel coordinates of any center point.

[0202] S705, determine the optimization image center area with the pixel point with the minimum distance sum as the center.

[0203] Optionally, one possible implementation of determining the optimization image center area based on S7041-S7042 is as follows: determine the optimization image center area with the pixel point with the minimum weighted distance sum as the center.

[0204] For example, gradient calculation is performed on formula 2 to calculate the change trend of the weighted distance sum in the vector direction, as shown in formula 3:

[0205]

[0206] wherein, and means gradient operation on the sum of weighted distances f(x), and other symbols refer to the explanation of formula 2.

[0207] Set the gradient in the gradient operation of the sum of weighted distances to 0, as shown in formula 4:

[0208]

[0209] Calculate formula 5 based on formula 4:

[0210]

[0211] Calculate the final distance and the minimum pixel point based on formula 5, and determine it as the center point of the optimized image center region, as shown in formula combination 6:

[0212]

[0213] The unexplained symbols in formulas 3-6 are consistent with the explanation of the corresponding symbols in formula 2.

[0214] In this embodiment, based on the calculation of the minimum sum of weighted distances, the final center point used for determining the optimized image center region is determined, so that the center point closest to the sum of distances of each point of interest is obtained, and the finally determined optimized image center region can be regarded as a region with high image information richness.

[0215] On the basis of the above-mentioned embodiments, the method further provides a vehicle-mounted camera pose adjustment method. Figure 8 The flowchart of the vehicle-mounted camera control method provided by the present disclosure Figure 3 . As Figure 8 shown, on the basis of the embodiment shown in Figure 5 , the adjustment of the vehicle-mounted camera pose in S504 is described in detail, and the method comprises:

[0216] S801, if the key observation region is not completely in the optimized image center region, adjust the optimized image center region according to the key observation region to obtain a new optimized image center region.

[0217] In this step, the key observation region is in the new optimized image center region.

[0218] Optionally, the way of adjusting the optimized image center region according to the key observation region can be:

[0219] Calculate the relative position between the critical observation area and the optimized image center area. Based on the relative position, move the optimized image center area in the direction of the critical observation area until the optimized image center area completely covers the critical observation area.

[0220] For example, suppose the key observation region is a rectangle S k Optimize the image center region into a rectangle S c0 When the critical observation region is completely covered by the optimized image center region, the coordinates calculated in Equation 6 are the optimal solution for the center coordinates of the optimized image center region. If the critical observation region and the optimized image center region do not completely overlap, it indicates that Equation 6 calculates an unconstrained optimal solution. To satisfy the constraints of the critical observation region, the unconstrained optimal solution needs to be adjusted. See [link to Equation 6]. Figure 13 Specifically:

[0221] For the key observation area S k The four vertices RSTU are sequentially determined to be within the rectangle Sc0' in the center region of the optimized image:

[0222] according to Figure 13 The positions of the four vertices of the key observation region Sk can be determined. The two vertices R and U on the left side of the key observation region Sk exceed the left boundary of the center region SC0' of the unconstrained optimization image. Therefore, it is necessary to move the unconstrained optimal solution corresponding to the center of the unconstrained optimization image to the left so that the left boundary of the rectangular key observation region coincides with the left boundary of the center region of the optimization image.

[0223] Similarly, if one of the four RSTU points lies above the upper boundary of the optimized image center region corresponding to the unconstrained optimal solution, the unconstrained optimal solution needs to be moved upwards; if one of the four RSTU points lies below the lower boundary of the optimized image center region corresponding to the unconstrained optimal solution, the unconstrained optimal solution needs to be moved downwards; and if one of the four RSTU points lies to the right of the right boundary of the optimized image center region corresponding to the unconstrained optimal solution, the unconstrained optimal solution needs to be moved to the right. The ultimate goal is to ensure that the optimized image center region completely covers the key observation area.

[0224] It should be noted that, Figure 13 The changes in the central region of the image are illustrated in the diagram, specifically as follows:

[0225] Without optimizing the image center region calculation, the current camera has an initial image center region S. C0 When calculating the unconstrained optimal solution, the unconstrained optimized image center region S is obtained. C0’ After calculating the constrained optimal solution based on the key observation region Sk, the constrained optimized image center region S is obtained.C1 The image center region changes with the image center point corresponding to the follower; the initial image center region S C0 The corresponding image center point is the initial image center point C0, and the unconstrained image center region S. C0’ The center point C0' refers to the unconstrained optimal solution, and the constrained image center region S C1 The center point C1 refers to the constrained optimal solution.

[0226] like Figure 13 As shown, after the adjustment, all four points of RSTU are within the rectangle Sc1 in the center region of the optimized image, thus obtaining a new optimal solution within the constraints. This is illustrated in Equation 7.

[0227]

[0228] Where C1 represents the constrained optimal solution for the center point of the new optimized image center region, and C0' represents the unconstrained optimal solution for the center point of the optimized image center region before adjustment; Δx C1C0' and Δy C1C0' Indicates the adjustment offset in the image coordinate system; x C1 and y C1 Here are the coordinates of the constrained optimal solution after adjustment; n is the number of target points of interest, and n is a positive integer greater than 1; i refers to the i-th target point of interest, (x i ,y i Let be the pixel coordinates of the i-th target point of interest, and k be the pixel coordinates of the i-th target point of interest. i This refers to the weight of each target point of interest. Figure 13 The example shows three points of interest: Point of Interest 1: stop line; Point of Interest 2: traffic light; Point of Interest 3: important traffic sign.

[0229] Figure 14 The changes in the central region of the actual image were plotted, and the calculated value was used to completely cover the key observation area S. k Optimized image center region S C1 Then, from the initial image center region S C0 Move to the optimized image center region S C1 .like Figure 14 As shown, the rectangular initial image center region S before camera adjustment C0 The four vertices are M0, N0, P0, and O0, with C0 as the center point; the rectangular optimized image center region S after camera adjustment C1 The four vertices are M1, N1, P1, and O1, with C1 as the center point; the optimized image center region S C1 The rectangular key observation area S with full coverage kThe four vertices of the quadrangle are S, R, U, and T.

[0230] S802, calculate the depth of the new optimized image center region.

[0231] Optionally, one possible implementation of calculating the depth of the new optimized image center region is as follows:

[0232] According to the depth of each point of interest in the new optimized image center region and the depth of the key observation region, the depth of the new optimized image center region is determined.

[0233] For example, as shown in formula 8, the depth of the new optimized image center region is calculated according to the depth of each point of interest in each image center region and the depth of the key observation region, and the calculation is performed in a weighted mean value manner.

[0234]

[0235] wherein d C1 represents the depth of the new optimized image center region finally obtained, k i represents the weight of each point of interest in the image center region, k Sk represents the weight of the key observation region S k , d Sk represents the depth of the key observation region S k , and d i represents the depth of each point of interest in the image center region, and m represents the number of all points of interest in the image center region.

[0236] S803, according to the depth information of the optimized image center region and the center point thereof, the extrinsic parameters of the vehicle-mounted camera, the intrinsic focal length, and the distortion parameters, the target angle of the first axis and the target angle of the third axis are determined.

[0237] For example, Figure 9 a three-axis gimbal adjustment angle calculation schematic diagram provided by the present disclosure is used to determine the target angle of the third axis; Figure 10 another three-axis gimbal adjustment angle calculation schematic diagram provided by the present disclosure is used to determine the target angle of the first axis. As Figure 9 shown, the center point C1 of the optimized image center region corresponds to a three-dimensional coordinate point A C1 in the ego coordinate system, the distance from A C1 to the optical center of the vehicle-mounted camera is the depth of the C1 point Moreover, according to the pinhole imaging principle, the angle between the three-dimensional coordinate point A C1 and the horizontal plane and the vertical plane can be known through the intrinsic parameters, and A is projected onto the horizontal plane passing through the optical center and the vertical plane passing through the optical axis to obtain d1 and d3.

[0238] 3D coordinate point A C1 The projection of the optical center of the vehicle camera onto the vertical plane passing through the optical axis is d1. From the vertical pixel distance from the distorted 3D coordinate point to the image center and the focal length, the angle between the line from the 3D coordinate point to the optical center and the optical axis can be obtained as γ1, and the distance from the optical center to the rotation axis is d2. For example... Figure 9 As shown, it can be determined how to make the vehicle-mounted camera align with the three-dimensional coordinate point A. C1 The target pitch angle that needs to be rotated is γ0, as shown in Formula 9:

[0239]

[0240] 3D coordinate point A C1 The projection of the distance to the optical center of the vehicle camera onto the horizontal plane passing through the optical center is d3. This is derived from the distortion-free 3D coordinates of point A. C1 The horizontal pixel distance to the image center and the focal length can be used to obtain the three-dimensional coordinates of point A. C1 The angle between the line to the optical center and the optical axis is α1, and the distance from the optical center to the axis of rotation is d4. For example... Figure 10 As shown, we can then determine how to make the camera align with the three-dimensional coordinate point A. C1 The target to be rotated has a yaw angle of α0, as shown in Formula 10:

[0241]

[0242] S804. Control the three-axis gimbal according to the target angle of the first axis and the target angle of the third axis to adjust the pose of the vehicle camera.

[0243] In this step, motion planning is performed on the first axis of the three-axis gimbal's rotating disk and the third axis controlling the vehicle-mounted camera; the target pitch angle is γ0, and the target yaw angle is α0. There are many motion planning methods; commonly used multi-axis robot motion planning methods can be employed. Preferably, interpolation-based planning methods have low computational costs, and because polynomial curve interpolation offers flexible curve shapes, fifth-order curve interpolation can be used for three-axis gimbal motion planning.

[0244] The control method for the first and third axes of the three-axis gimbal can employ an inner loop algorithm for angular velocity and an outer loop algorithm for the angle of each axis. This ensures that the optical axis moves towards the center point of the optimized image center region, ultimately aligning the main optical axis of the vehicle camera with the center of the optimized image center region. The angular velocity is controlled by using the torque of the drive motor to achieve acceleration and deceleration.

[0245] In the embodiment, the center region of the optimized image is adjusted to cover the key observation region and as many points of interest as possible, and a new center region of the optimized image is finally obtained; the direction and angle of the camera are controlled based on the new center region of the optimized image, so that the main optical axis of the vehicle-mounted camera coincides with the center point of the new center region of the optimized image.

[0246] Figure 11 A schematic diagram of an installation mode of a three-axis gimbal provided with a camera is provided in the disclosure. As shown in the figure, Figure 11 the three-axis gimbal is installed at the front end of the top of the vehicle and can be used to monitor the road conditions in front of the vehicle. Figure 12 A schematic diagram of an installation mode of a three-axis gimbal provided with a camera is provided in the disclosure. Figure 2 The three-axis gimbal is installed at the front end and the rear end of the top of the vehicle and can be used to monitor the road conditions in front of and behind the vehicle.

[0247] Figure 15 A schematic diagram of the structure of a vehicle-mounted camera control device is provided in the disclosure, as shown in the figure, Figure 15 the device is applied to a vehicle, and a vehicle-mounted camera is installed on the top of the vehicle. The device comprises:

[0248] A first processing module 1501 is configured to, after the vehicle starts the automatic driving function, raise the vehicle-mounted camera to a target height according to the ego state information of the vehicle and the surrounding environment information, and the target height is the maximum height to which the vehicle-mounted camera can be raised.

[0249] A second processing module 1502 is configured to determine a key observation region of the vehicle-mounted camera according to a current image collected by the vehicle-mounted camera and a to-be-traveled path planned by the automatic driving function, and the key observation region comprises a target position point of the to-be-traveled path.

[0250] A third processing module 1503 is configured to determine an optimized image center region according to a plurality of points of interest in the current image, the points of interest are elements related to the automatic driving function in the road, and the area of the optimized image center region is greater than the area of the key observation region.

[0251] A fourth processing module 1504 is configured to adjust the pose of the vehicle-mounted camera according to the optimized image center region and the key observation region, so as to adjust the optimized image center region collected by the vehicle-mounted camera to cover the key observation region.

[0252] Further, the third processing module 1503 is further configured to:

[0253] determine a plurality of initial points of interest in the current image;

[0254] According to a preset condition, a priority of each initial interest point is determined, and the priority of each initial interest point is proportional to an importance degree of the initial interest point to the automatic driving function;

[0255] According to the priority of each initial interest point, a target interest point is determined from all the initial interest points, and the priority of the target interest point is greater than the priority of the other initial interest points;

[0256] A distance sum of each pixel point in the current image to all the target interest points is calculated;

[0257] A pixel point with a minimum distance sum is taken as a center to determine an optimized image center region.

[0258] Further, the third processing module 1503 is further configured to:

[0259] According to the priority of each target interest point, a weight of each target interest point is determined, and the weight of each target interest point is proportional to the priority;

[0260] According to the weight of each target interest point, a weighted distance sum of each pixel point in the current image to all the target interest points is calculated;

[0261] Correspondingly, a pixel point with a minimum weighted distance sum is taken as a center to determine an optimized image center region, including:

[0262] A pixel point with a minimum weighted distance sum is taken as a center to determine an optimized image center region.

[0263] Further, the second processing module 1502 is further configured to:

[0264] According to the to-be-traveled path and the navigation map information, a camera coordinate of the target position point in a camera coordinate system is determined;

[0265] The camera coordinate is projected to an image coordinate system to determine a pixel coordinate of the target position point;

[0266] According to a pixel point corresponding to the pixel coordinate, a key observation region is determined.

[0267] Further, the vehicle-mounted camera is arranged on a three-axis gimbal, the three-axis gimbal includes a rotating table and a rotating arm arranged on the rotating table, and the vehicle-mounted camera is arranged on the rotating arm;

[0268] The three-axis gimbal includes a first axis, a second axis and a third axis;

[0269] The first axis is perpendicular to the rotating table and is used to adjust a rotating angle of the rotating table;

[0270] The second axis is perpendicular to the rotating arm and is used to adjust an included angle between the rotating arm and the rotating table;

[0271] The third axis is perpendicular to the optical axis of the vehicle-mounted camera, and is used to adjust the pitch angle of the vehicle-mounted camera.

[0272] The fourth processing module 1504 is further configured to:

[0273] If the key observation region is not completely in the optimized image center region, adjusting the optimized image center region according to the key observation region to obtain a new optimized image center region, and the key observation region is in the new optimized image center region;

[0274] Calculating the depth of the new optimized image center region;

[0275] According to the depth information and the center point of the optimized image center region, the external parameters of the vehicle-mounted camera, the internal focal length and the distortion parameters, the target angle of the first axis and the target angle of the third axis are determined;

[0276] According to the target angle of the first axis and the target angle of the third axis, the three-axis gimbal is controlled to adjust the pose of the vehicle-mounted camera.

[0277] Further, the fourth processing module 1504 is further configured to:

[0278] According to the depth of each point of interest in the new optimized image center region and the depth of the key observation region, the depth of the new optimized image center region is determined.

[0279] Further, the first processing module 1501 is further configured to:

[0280] According to the self-vehicle state information of the vehicle and the wind resistance coefficient, the target air resistance of the three-axis gimbal is calculated;

[0281] According to the preset resistance mapping table, the first included angle and the first height corresponding to the target air resistance are determined, and the resistance mapping table is used to represent the maximum included angle of the second axis and the maximum height of the vehicle-mounted camera corresponding to different air resistances;

[0282] According to the surrounding environment information, the second height is determined, and the second height is the maximum height of the vehicle-mounted camera limited by the environmental obstacles;

[0283] According to the first height, the second height and the first included angle, the three-axis gimbal is controlled to raise the vehicle-mounted camera to the target height, and the target height is the first height or the second height.

[0284] Further, the first processing module 1501 is further configured to:

[0285] If the first height is less than or equal to the second height, the three-axis gimbal is controlled according to the first included angle to raise the vehicle-mounted camera to the first height, and the first height is the target height;

[0286] If the first height is greater than the second height, the second included angle of the three-axis gimbal is determined based on the second height;

[0287] The three-axis gimbal is controlled according to the second included angle, so as to raise the vehicle-mounted camera to the second height, and the second height is the target height.

[0288] Figure 16 The structural schematic diagram of the electronic device provided in the present disclosure is shown. As shown in the figure, the electronic device 160 provided in the present embodiment comprises at least one processor 1601 and a memory 1602. Optionally, the device 160 further comprises a communication component 1603. The processor 1601, the memory 1602 and the communication component 1603 are connected through a bus 1604. Figure 16

[0289] In the specific implementation process, the at least one processor 1601 executes the computer execution instructions stored in the memory 1602, so that the at least one processor 1601 executes the above-mentioned method.

[0290] The specific implementation process of the processor 1601 can refer to the above-mentioned method embodiment, which has similar implementation principles and technical effects, and will not be described here in detail.

[0291] In the above-mentioned embodiment, it should be understood that the processor can be a central processing unit (English: Central Processing Unit, for short: CPU), and can also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, for short: DSP), application specific integrated circuits (English: Application Specific Integrated Circuit, for short: ASIC) and the like. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, etc. The steps of the method disclosed in the present application can be directly embodied as the execution of the hardware processor, or executed by the combination of the hardware and software modules in the processor.

[0292] The memory can contain a random access memory (Random Access Memory, for short: RAM), and can also include a non-volatile memory (Non-volatile Memory, for short: NVM), for example, at least one disk memory.

[0293] ​The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the method is not limited to only one bus or one type of bus.

[0294] The method also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described above.

[0295] The method also provides a computer-readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the method described above is implemented.

[0296] The readable storage medium described above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0297] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium, and can write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0298] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0299] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0300] In addition, each functional unit in various embodiments of the present application can be integrated into one processing unit, or each unit can be physically present separately, or two or more units can be integrated into one unit.

[0301] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0302] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. The program executes the steps of the above-mentioned method embodiments when executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various media that can store program codes.

[0303] Finally, it should be noted that those skilled in the art, after considering the specification and practicing the application disclosed herein, will easily think of other embodiments of the present application. The present application is intended to cover any variations, uses or adaptations of the present application that follow the general principles of the present application and include known or customary technical means in the art that are not disclosed in the present application, and is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is only limited by the appended claims. The above embodiments are only preferred embodiments for fully illustrating the present application, and the protection scope of the present application is not limited thereto. Any equivalent replacement or transformation made by those skilled in the art based on the present application is within the protection scope of the present application.

Claims

1. A method for controlling a car camera, characterized in that, The method is applied to a vehicle, a vehicle camera is installed on the top of the vehicle, and the method comprises the following steps: After the vehicle starts the automatic driving function, the vehicle camera is raised to a target height according to the self-vehicle state information and the surrounding environment information of the vehicle, and the target height is the maximum height that the vehicle camera can be raised to; According to the current image collected by the vehicle camera and the to-be-traveled path planned by the automatic driving function, a key observation area of the vehicle camera is determined, and the key observation area includes a target position point of the to-be-traveled path; According to a plurality of interest points in the current image, an optimized image center area is determined, the interest points are elements related to the automatic driving function in the road, and the area of the optimized image center area is greater than the area of the key observation area; According to the optimized image center area and the key observation area, the pose of the vehicle camera is adjusted, so that the optimized image center area collected by the vehicle camera is adjusted to cover the key observation area.

2. The method of claim 1, wherein, The method comprises the following steps: A plurality of initial interest points in the current image are determined; According to a preset condition, the priority of each initial interest point is determined, and the priority of each initial interest point is proportional to the importance of the initial interest point to the automatic driving function; According to the priority of each initial interest point, a target interest point is determined from all the initial interest points, and the priority of the target interest point is greater than the priority of other initial interest points; The sum of distances from each pixel point in the current image to all target interest points is calculated; The pixel point with the minimum distance sum is taken as the center to determine the optimized image center area.

3. The method of claim 2, wherein, The method comprises the following steps: According to the priority of each target interest point, the weight of each target interest point is determined, and the weight of each target interest point is proportional to the priority; According to the weight of each target interest point, the weighted distance sum of each pixel point in the current image to all target interest points is calculated; Correspondingly, the method comprises the following steps: The pixel point with the minimum weighted distance sum is taken as the center to determine the optimized image center area.

4. The method according to any one of claims 1 to 3, characterized in that, The method comprises the following steps: According to the to-be-traveled path and the navigation map information, a camera coordinate of the target position point in a camera coordinate system is determined; The camera coordinate is projected to an image coordinate system to determine a pixel coordinate of the target position point; According to the pixel corresponding to the pixel coordinate, the key observation area is determined.

5. The method according to any one of claims 1 to 3, characterized in that, The vehicle camera is arranged on a three-axis gimbal, the three-axis gimbal comprises a rotating table and a rotating arm arranged on the rotating table, and the vehicle camera is arranged on the rotating arm; The three-axis gimbal comprises a first axis, a second axis and a third axis; The first shaft is perpendicular to the rotating table, and is used for adjusting the rotation angle of the rotating table. The second shaft is perpendicular to the rotating arm, and is used for adjusting the included angle between the rotating arm and the rotating table. The third shaft is perpendicular to the optical axis of the vehicle-mounted camera, and is used for adjusting the pitch angle of the vehicle-mounted camera.

6. The method of claim 5, wherein, The adjusting the pose of the vehicle-mounted camera according to the optimized image center region and the key observation region comprises: if the key observation region is not completely in the optimized image center region, adjusting the optimized image center region according to the key observation region to obtain a new optimized image center region, and the key observation region is in the new optimized image center region; calculating the depth of the new optimized image center region; determining the target angle of the first shaft and the target angle of the third shaft according to the depth information and the center point of the optimized image center region, the external parameter of the vehicle-mounted camera, the internal focal length and the distortion parameter; controlling the three-axis gimbal according to the target angle of the first shaft and the target angle of the third shaft to adjust the pose of the vehicle-mounted camera.

7. The method of claim 6, wherein, The calculating the depth of the new optimized image center region comprises: determining the depth of the new optimized image center region according to the depth of each point of interest in the new optimized image center region and the depth of the key observation region.

8. The method of claim 5, wherein, The lifting the vehicle-mounted camera to the target height according to the self-vehicle state information and the surrounding environment information of the vehicle comprises: calculating the target air resistance of the three-axis gimbal according to the self-vehicle state information and the wind resistance coefficient of the vehicle; determining the first included angle and the first height corresponding to the target air resistance according to a preset resistance mapping table, the resistance mapping table being used for representing the maximum included angle of the second shaft and the highest height of the vehicle-mounted camera corresponding to different air resistances; determining a second height according to the surrounding environment information, the second height being the highest height of the vehicle-mounted camera limited by environmental obstacles; controlling the three-axis gimbal according to the first height, the second height and the first included angle to lift the vehicle-mounted camera to the target height, the target height being the first height or the second height.

9. The method of claim 8, wherein, The controlling the three-axis gimbal according to the first height, the second height and the first included angle to lift the vehicle-mounted camera to the target height comprises: if the first height is less than or equal to the second height, controlling the three-axis gimbal according to the first included angle to lift the vehicle-mounted camera to the first height, the first height being the target height; if the first height is greater than the second height, determining a second included angle of the three-axis gimbal based on the second height; controlling the three-axis gimbal according to the second included angle to lift the vehicle-mounted camera to the second height, the second height being the target height.

10. A car camera control device characterized by comprising: The device is applied to a vehicle, and a vehicle-mounted camera is installed on the top of the vehicle. The first processing module is configured to, after the vehicle starts the automatic driving function, raise the vehicle-mounted camera to a target height according to the self-vehicle state information and the surrounding environment information of the vehicle, the target height being a maximum height at which the vehicle-mounted camera can be raised; The second processing module is configured to determine a key observation area of the vehicle-mounted camera according to a current image collected by the vehicle-mounted camera and a to-be-traveled path planned by the automatic driving function, the key observation area including a target position point of the to-be-traveled path; The third processing module is configured to determine an optimized image center area according to a plurality of interest points in the current image, the interest points being elements related to the automatic driving function in a road, and an area of the optimized image center area being greater than an area of the key observation area; The fourth processing module is configured to adjust a pose of the vehicle-mounted camera according to the optimized image center area and the key observation area, so as to adjust the optimized image center area collected by the vehicle-mounted camera to cover the key observation area.

11. An electronic device, comprising: comprising: a memory, a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor executes the method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by the processor to implement the method according to any one of claims 1-9.

13. A computer program product, characterised in that, comprising a computer program, which is executed by the processor to implement the method according to any one of claims 1-9.

Images