Vehicle-mounted panoramic image frame rate detection method and device, vehicle and storage medium

Abstract

The invention discloses a vehicle-mounted panoramic image frame rate detection method and device, a vehicle and a storage medium, and relates to the technical field of vehicle control. The method includes: extracting a vehicle image frame including a vehicle image from a vehicle-mounted panoramic image video stream, the vehicle image frame including a top view area, and the top view area including a vehicle area having the vehicle image; performing vertex mapping based on the vehicle area to obtain four vertexes; performing region division on the top view region based on the four vertexes to obtain at least one target region; performing region division on each target region based on the vertexes to obtain fan-shaped regions; frame change detection is carried out according to the fan-shaped areas of the two adjacent vehicle image frames, a frame detection mark is obtained, and the frame detection mark comprises an image frame change value; determining the vehicle image frame as a target change frame according to the image frame change value; and performing frame rate calculation according to the target change frame to obtain a target frame rate. The method can improve the frame rate detection accuracy and detection efficiency, and is suitable for a vehicle-mounted scene.

Description

Technical Field

[0001] The present application relates to the field of vehicle control technology, and in particular to a vehicle-mounted panoramic image frame rate detection method and device, a vehicle, and a storage medium. Background Art

[0002] In the field of autonomous driving, panoramic imaging is a function that stitches four surround-view cameras into a bird's-eye view and can assist users in parking. When the panoramic image video stream freezes, it will directly affect the user's subjective experience. In severe cases, it will interfere with the user's parking and cause the user to misjudge the location of obstacles. Therefore, it is necessary to perform on-board panoramic image frame rate detection in order to evaluate the smoothness of the panoramic image video stream. For example, the output frequency of the panoramic image can be obtained by counting the number of consecutive images of the panoramic image video stream output by the vehicle computer per unit time.

[0003] The traditional method is to manually count the number of changes in the video stream with the naked eye, which is labor-intensive, inefficient, and seriously affected by subjective factors and may not be accurate. Related technologies also propose a video frame rate detection method based on visual perception. This method detects whether the video is stuck by the change rate of adjacent frames. However, when the video stream resolution is too large, the real-time detection efficiency is reduced, which is not suitable for vehicle-mounted scenarios. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a vehicle-mounted panoramic image frame rate detection method and device, a vehicle, and a storage medium, which can improve the frame rate detection accuracy and detection efficiency and is suitable for vehicle-mounted scenes.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a vehicle-mounted panoramic image frame rate detection method, the method comprising:

[0006] Extracting a vehicle image frame containing a vehicle image from a vehicle-mounted panoramic image video stream, wherein the vehicle image frame contains a top-view area, and the top-view area contains a vehicle area having the vehicle image;

[0007] Perform vertex mapping based on the vehicle area to obtain four vertices;

[0008] Dividing the top view area based on the four vertices to obtain at least one target area; wherein each target area includes one vertex;

[0009] Divide each target area based on the vertices to obtain a sector-shaped area;

[0010] Performing frame change detection according to the sector-shaped areas of two adjacent vehicle image frames to obtain a frame detection mark; wherein the frame detection mark includes an image frame change value, and the image frame change value indicates that the two adjacent vehicle image frames are different;

[0011] Determining the vehicle image frame as a target change frame according to the image frame change value;

[0012] The frame rate is calculated according to the target change frame to obtain a target frame rate.

[0013] Optionally, the frame detection mark further includes an image frame unchanged value, and the image frame unchanged value indicates that two adjacent vehicle image frames are identical;

[0014] After performing frame change detection according to the sector-shaped areas of two adjacent vehicle image frames to obtain a frame detection mark, the method further includes:

[0015] Taking a preset number of continuous vehicle image frames as image frames to be smoothed;

[0016] Setting increasing weights for a preset number of the image frames to be smoothed in chronological order;

[0017] Performing weighted calculation according to the image frame change values ​​or the image frame unchanged values ​​of a preset number of the image frames to be smoothed and the weight of each of the image frames to be smoothed, to obtain frame change smoothing data;

[0018] Based on the frame change smoothing data, a target operation is determined, where the target operation includes any one of the following: no operation, warning reporting, or fault reporting.

[0019] Optionally, performing frame change detection according to the sector-shaped areas of two adjacent vehicle image frames to obtain a frame detection mark includes:

[0020] Calculating the weight of each sector area to obtain a sector area weight value;

[0021] Performing difference calculation on the sector-shaped area weight values ​​of two adjacent vehicle image frames to obtain an area weight difference;

[0022] The frame detection mark is determined by performing a mark determination according to the area weight difference of the sector-shaped area to obtain the frame detection mark.

[0023] Optionally, the frame detection mark further includes an image frame unchanged value, and the image frame unchanged value indicates that two adjacent vehicle image frames are identical;

[0024] The step of performing labeling and determining according to the area weight difference of the sector-shaped area to obtain the frame detection label comprises:

[0025] If the area weight difference of each of the fan-shaped areas is greater than a preset weight difference threshold, the fan-shaped area is determined to be a change area;

[0026] Counting the number of the changed regions to obtain the number of changed regions;

[0027] If the number of changed regions is greater than or equal to a preset change number threshold, determining that the frame detection mark is the image frame change value;

[0028] If the number of changed areas is less than the change number threshold, the frame detection mark is determined to be the unchanged value of the image frame.

[0029] Optionally, dividing the top view area based on the four vertices to obtain at least one target area includes:

[0030] Taking each of the vertices as an origin, a first cutting line is derived based on a first direction and a second cutting line is derived based on a second direction; wherein the first direction is parallel to at least one side of the top view area, and the first direction and the second direction are perpendicular to each other;

[0031] The top view area is segmented based on the first segmentation line and the second segmentation line to obtain the target area.

[0032] Optionally, dividing each target area based on the vertices to obtain a sector-shaped area includes:

[0033] Taking the vertex as the center of the circle, a diagonal line is drawn in the target area;

[0034] Extending from the diagonal line to both sides, an area with a preset angular arc and a preset radius is selected to obtain the fan-shaped area.

[0035] Optionally, the performing frame rate calculation according to the target change frame to obtain a target frame rate includes:

[0036] Accumulating the number of target change frames within a preset time period to obtain the number of frame changes;

[0037] The target frame rate is obtained by dividing the frame change quantity by the preset time period.

[0038] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a vehicle-mounted panoramic image frame rate detection device, the device comprising:

[0039] A frame acquisition module, used for extracting a vehicle image frame containing a vehicle image from a vehicle-mounted panoramic image video stream, wherein the vehicle image frame contains a bird's-eye view area, and the bird's-eye view area includes a vehicle area having the vehicle image;

[0040] A vertex mapping module, used for performing vertex mapping based on the vehicle area to obtain four vertices;

[0041] A first division module is used to divide the top view area into regions based on the four vertices to obtain at least one target region; wherein each target region includes one vertex;

[0042] A second division module is used to divide each of the target areas based on the vertices to obtain a sector-shaped area;

[0043] A frame detection module, configured to perform frame change detection according to the sector-shaped areas of two adjacent vehicle image frames to obtain a frame detection mark; wherein the frame detection mark includes an image frame change value, and the image frame change value indicates that the two adjacent vehicle image frames are different;

[0044] a frame determination module, configured to determine the vehicle image frame as a target change frame according to the image frame change value;

[0045] The frame rate calculation module is used to calculate the frame rate according to the target change frame to obtain a target frame rate.

[0046] To achieve the above objectives, the third aspect of an embodiment of the present application proposes a vehicle, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0047] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0048] The vehicle-mounted panoramic image frame rate detection method and device, vehicle, and storage medium proposed in the present application first extract vehicle image frames from the vehicle-mounted panoramic image video stream; secondly, the overhead view area contained in the vehicle image frame is divided into regions to obtain at least one target area, and then the target area is divided into fan-shaped regions; then all fan-shaped regions of two adjacent vehicle image frames are subjected to frame change detection to determine whether the vehicle image frame is a target change frame; finally, the target frame rate is calculated based on the target change frame. The difference between the fan-shaped regions of two adjacent vehicle image frames can indicate whether the two adjacent vehicle image frames are the same or different. Therefore, the accuracy of frame change detection based on fan-shaped regions is high, thereby improving the accuracy of frame rate detection. Moreover, there is no need to perform frame change detection on the entire vehicle image frame, which can reduce the amount of detection calculation, thereby improving the efficiency of frame rate detection. It can be seen that the embodiments of the present application can improve the accuracy and efficiency of frame rate detection and are suitable for vehicle-mounted scenarios.

[0049] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a vehicle-mounted panoramic image frame rate detection method provided in an embodiment of the present application;

[0051] Figure 2 is a schematic diagram of a vehicle image frame provided in an embodiment of the present application;

[0052] Figure 3 is a schematic diagram of a top view area provided in an embodiment of the present application;

[0053] Figure 4 is a schematic diagram of four vertices provided in an embodiment of the present application;

[0054] Figure 5 is another schematic diagram of a top view area provided in an embodiment of the present application;

[0055] Figure 6 It is a schematic diagram of dividing the top view area to obtain the target area provided in an embodiment of the present application;

[0056] Figure 7 It is a schematic diagram of dividing a target area into sector-shaped areas provided in an embodiment of the present application;

[0057] Figure 8 is a flow chart of a vehicle-mounted panoramic image frame rate detection method provided by another embodiment of the present application;

[0058] Fig. 9 It is a structural schematic diagram of the vehicle-mounted panoramic image frame rate detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0063] Panoramic imaging: also known as 360° panoramic imaging. A panoramic imaging system can be installed on the vehicle. The panoramic imaging system covers all the fields of view around the vehicle by setting up 4 to 8 wide-angle, high-sensitivity cameras around the vehicle. The images of the front, back, left, and right sides of the vehicle collected at the same time are converted into digital information by the acquisition component and sent to the image processing component. After the image processing component performs distortion restoration → perspective conversion → image stitching → image enhancement, it is converted into an analog signal output to generate a 360° top view of the vehicle body (hereinafter referred to as the top view area), which is finally displayed on the screen of the center console, allowing the driver to clearly check whether there are obstacles around the vehicle and understand the relative position and distance of the obstacles, helping the driver to park the vehicle easily.

[0064] Panoramic video stream: It is composed of panoramic images collected at different times.

[0065] On-vehicle panoramic image video stream: refers to the panoramic image video stream output by the vehicle, or the panoramic image video stream output by the panoramic image system installed in the vehicle.

[0066] Frame rate: refers to the number of image frames displayed within a period of time, usually expressed in "frames per second". Frame rate is an important indicator for measuring the smoothness of videos, animations and games. In the embodiment of the present application, the frame rate can be used to characterize the smoothness of the vehicle-mounted panoramic image video stream.

[0067] The vehicle-mounted panoramic image frame rate detection method provided in the embodiment of the present application can be applied to the terminal and the server side, and can also be software running on the server side. The server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the vehicle-mounted panoramic image frame rate detection method, but is not limited to the above forms.

[0068] The vehicle-mounted panoramic image frame rate detection method, vehicle-mounted panoramic image frame rate detection device, vehicle and computer-readable storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the vehicle-mounted panoramic image frame rate detection method in the embodiments of the present application is described.

[0069] Please refer to Figure 1 , Figure 1 An optional flow chart of a vehicle-mounted panoramic image frame rate detection method is disclosed. Figure 1 The method may include but is not limited to steps 101 to 107.

[0070] Step 101, extracting a vehicle image frame containing a vehicle image from a vehicle-mounted panoramic image video stream, wherein the vehicle image frame contains a bird's-eye view area, and the bird's-eye view area contains a vehicle area having a vehicle image;

[0071] Step 102, performing vertex mapping based on the vehicle area to obtain four vertices;

[0072] Step 103, dividing the top view area based on the four vertices to obtain at least one target area;

[0073] Step 104, dividing each target area based on the vertices to obtain a sector-shaped area;

[0074] Step 105, performing frame change detection according to the fan-shaped area of ​​two adjacent vehicle image frames to obtain a frame detection mark; wherein the frame detection mark includes an image frame change value, and the image frame change value indicates that the two adjacent vehicle image frames are different;

[0075] Step 106, determining the vehicle image frame as a target change frame according to the image frame change value;

[0076] Step 107, calculating the frame rate according to the target change frame to obtain the target frame rate.

[0077] Steps 101 to 107 shown in the embodiment of the present application do not perform frame rate detection on the entire vehicle image frame, but first perform coarse-grained regional division on the overhead view area contained in the vehicle image frame to obtain at least one target area, and then perform fine-grained regional division on the target area to obtain a fan-shaped area. The overhead view area is usually spliced ​​from captured images in four directions (front, back, left, and right), and each fan-shaped area can represent a spliced ​​area formed by two adjacent images captured in four directions. Even if the captured image in one direction changes, the fan-shaped area will change. It can be seen that the use of fan-shaped areas can improve detection accuracy, has strong generalization, and is better than global area detection; it can also reduce the amount of data, improve efficiency, meet real-time requirements, and is suitable for vehicle-mounted scenarios.

[0078] In step 101, the vehicle-mounted panoramic image video stream includes at least two vehicle image frames at different times. The vehicle image frame may also be called a panoramic image. Specifically, each frame of the vehicle-mounted panoramic image video stream is extracted to obtain a vehicle image frame.

[0079] Reference Figure 2 The vehicle image frame includes a dock bar (also called a DOCK bar) area, a status bar area, a menu bar area, a front view area, and a top view area.

[0080] In one example, the vehicle image frame may further include a rear view area, a left view area, and a right view area.

[0081] It should be noted that the vehicle image contained in the vehicle image frame is generally located in the top view area. Figure 3 As shown, the top view area includes a vehicle area having a vehicle image.

[0082] The top view area may be formed by projecting four fisheye images onto the ground and stitching them together. The four fisheye images include a front fisheye image, a rear fisheye image, a left fisheye image, and a right fisheye image.

[0083] In one embodiment, the top view area can be extracted from the vehicle image frame in the following manner: (1) performing region matching between a preset template and the vehicle image frame to determine the region boundary of the top view, using a maximum circumscribed rectangular frame to intercept the region contained in the region boundary from the vehicle image frame to obtain the maximum circumscribed rectangular frame region, and using the maximum circumscribed rectangular frame region as the top view area. (2) Inputting the vehicle image frame into a target detection model (such as a Faster-R-CNN model) to perform target detection to obtain the top view area. The target detection model will automatically divide the top view area according to the region distribution in the vehicle image frame.

[0084] In practice, the top view area may be tilted at an angle and needs to be calibrated to the front view. Therefore, in one embodiment, after obtaining the maximum circumscribed rectangular frame area, the maximum circumscribed rectangular frame area can be converted to a regular rectangular frame area based on a projection transformation, using operations such as angle rotation and scale scaling, and then the regular rectangular frame area is used as the top view area.

[0085] In step 102, referring to Figure 3 , the vehicle area is a rectangle. A rectangle generally contains four points that form the four corners of the rectangle. Therefore, based on the vehicle area, vertex mapping is performed on the top view area to obtain four vertices. Figure 4 As shown, after vertex mapping, vertex P1, vertex P2, vertex P3 and vertex P4 can be obtained.

[0086] In step 103, the target area is a sub-area in the top view area. The target areas do not overlap each other, and the target area does not overlap with the vehicle area. After the area division, one, two, three, or four target areas can be obtained, that is, at least one target area.

[0087] In one example, refer to Figure 5 , based on vertex P1, vertex P2, vertex P3 and vertex P4, the top view area is divided into regions, and the front region, rear region, left region, right region, left front region, right front region, left rear region and right rear region can be obtained, a total of eight regions. The images of the front region, rear region, left region and right region are respectively composed of the front fisheye image, the rear fisheye image, the left fisheye image and the right fisheye image. The left front region, the right front region, the left rear region and the right rear region are respectively composed of the left fisheye image and the front fisheye image, the right fisheye image and the front fisheye image, the left fisheye image and the rear fisheye image, the right fisheye image and the rear fisheye image, and the two views are composed. The left front region is taken as the target region M1, the right front region is taken as the target region M2, the left rear region is taken as the target region M3 and the right rear region is taken as the target region M4. The front region, the rear region, the left region and the right region can be called non-target regions.

[0088] Each target region contains a vertex. For example, Figure 5 , the target area M1 includes vertex P1, the target area M2 includes vertex P2, the target area M3 includes vertex P3, and the target area M4 includes vertex P4. The non-target area includes more than two vertices, for example, the front area includes vertex P1 and vertex P2.

[0089] Reference Figure 6 In one embodiment, step 103 includes:

[0090] Taking each vertex as an origin, a first cutting line is drawn based on a first direction x and a second cutting line is drawn based on a second direction y;

[0091] The top view area is segmented based on the first segmentation line and the second segmentation line to obtain a target area.

[0092] Specifically, the first direction x is parallel to at least one side of the top view area, and the first direction x is perpendicular to the second direction y. With the vertex P1 as the origin, the first cutting line L1 is derived based on the first direction x and the second cutting line L2 is derived based on the second direction y; the top view area is segmented based on the first cutting line L1 and the second cutting line L2 to obtain the target area M1. With the vertex P2 as the origin, the first cutting line L3 is derived based on the first direction x and the second cutting line L4 is derived based on the second direction y; the top view area is segmented based on the first cutting line L3 and the second cutting line L4 to obtain the target area M2. With the vertex P3 as the origin, the first cutting line L5 is derived based on the first direction x and the second cutting line L6 is derived based on the second direction y; the top view area is segmented based on the first cutting line L5 and the second cutting line L6 to obtain the target area M3. Taking the vertex P4 as the origin, a first cutting line L7 is derived based on the first direction x and a second cutting line L8 is derived based on the second direction y; the top view area is segmented based on the first cutting line L7 and the second cutting line L8 to obtain a target area M4.

[0093] The benefit of the above embodiment is that, on the basis of being able to determine the target area, the complexity is reduced and the determination efficiency is improved.

[0094] In step 104, the sector is a part of a circle, which is surrounded by two radii and an arc. The sector area is an area with a vertex as the center, a sector outline, and a median line that coincides with a diagonal line of the target area.

[0095] In one embodiment, step 104 includes:

[0096] With the vertex as the center, draw a diagonal line in the target area;

[0097] Extending from the diagonal line to both sides, select an area with a preset angle and a preset radius to obtain a sector-shaped area.

[0098] For example, refer to Figure 6 and Figure 7, with the vertex P1 as the center, draw a diagonal line in the target area M1; extend from the diagonal line to both sides, select the area with a preset angle of θ and a preset radius of r, and obtain the fan-shaped area S1. With the vertex P2 as the center, draw a diagonal line in the target area M2; extend from the diagonal line to both sides, select the area with a preset angle of θ and a preset radius of r, and obtain the fan-shaped area S2. With the vertex P3 as the center, draw a diagonal line in the target area M3; extend from the diagonal line to both sides, select the area with a preset angle of θ and a preset radius of r, and obtain the fan-shaped area S3. With the vertex P1 as the center, draw a diagonal line in the target area M4; extend from the diagonal line to both sides, select the area with a preset angle of θ and a preset radius of r, and obtain the fan-shaped area S4.

[0099] It should be noted that each sector area covers the fusion point of two fisheye images (also called camera views), for example, sector area S1 covers the fusion point of the front road fisheye image and the left road fisheye image. The preset angle radian and preset radius can be set according to actual needs. The preset angle radian fisheye controls the proportion of the fusion area. The preset radius is used to control the size of the fusion area. For example, θ can be set to 45° and r to 9cm.

[0100] An advantage of the above embodiment is that the sector-shaped area can be determined flexibly.

[0101] In step 105, the frame detection mark is used to indicate whether two adjacent vehicle image frames are the same or different. The frame detection mark includes an image frame change value or an image frame unchanged value. The image frame change value indicates that two adjacent vehicle image frames are different. The image frame unchanged value indicates that two adjacent vehicle image frames are the same. For example, the image frame change value is 1, and the image frame unchanged value is 0.

[0102] In one embodiment, step 105 includes:

[0103] Calculate the weight of each sector area to obtain the sector area weight value;

[0104] Performing difference calculation on the fan-shaped area weight values ​​of two adjacent vehicle image frames to obtain the area weight difference;

[0105] The marking is determined according to the area weight difference of the sector area to obtain the frame detection mark.

[0106] The sector area weight value indicates the pixel information of the sector area. The regional weight difference indicates the difference between the sector areas of two adjacent vehicle image frames. The larger the regional weight difference, the greater the difference. For example, the sector area weight value of the vehicle image frame at the N-1th moment and the sector area weight value of the vehicle image frame at the Nth moment are calculated to obtain the regional weight difference of the i-th vehicle image frame; the regional weight difference of the sector area of ​​the vehicle image frame at the Nth moment is marked and determined to obtain the frame detection mark of the vehicle image frame at the Nth moment.

[0107] The benefit of the above embodiment is that, on the basis of being able to determine the frame detection flag, the determination complexity is reduced.

[0108] The sector-shaped area includes at least two pixel points. In one embodiment, the step of performing weight calculation on each sector-shaped area to obtain the weight value of the sector-shaped area may include:

[0109] Get the pixel value of the pixel point to get the initial pixel value;

[0110] Convert the initial pixel value from the color space to the brightness space to obtain the target pixel value;

[0111] The target pixel values ​​of all pixels are counted to obtain the fan-shaped area weight value.

[0112] Specifically, the color space refers to the RGB color space. The brightness space is the YUV color space. The conversion formula is: Y = 0.298*R + 0.612*G + 0.117*B, U = -0.168*R-0.330*G + 0.498*B + 128, V = 0.449*R-0.435*G-0.083*B + 128. Among them, Y is the brightness value, U, V are the color difference values, R is the red channel value, G is the green channel value, and B is the blue channel value.

[0113] The calculation formula of the sector area weight value is: This formula is an expression of the functional relationship between Y, U, and V as θ and r change. Take a functional relationship as an example, in, Y refers to the fan-shaped area weight value of the i-th fan-shaped area. i Refers to the brightness value of the i-th sector area, U i , V i All refer to the color difference value of the i-th sector area. α represents the weight scaling value, which can be set to 30 according to experience. t Indicates the fixed value difference threshold, which can be set to 30 according to experience. V t It represents the fixed color difference threshold, which can be set to 30 according to experience. θ×R represents the range of the fan-shaped area, and the pixels beyond the fan-shaped area are discarded.

[0114] The benefit of the above embodiment is that it can reduce the amount of calculation, detect frame rate accuracy better than single gray value comparison without adding excessive computing power, and help improve frame rate detection efficiency.

[0115] The formula for calculating the difference is: Wherein, D represents the regional weight difference, pre represents the previous frame of two adjacent vehicle image frames, which can also be called the previous frame, and cur represents the next frame of two adjacent vehicle image frames, which can also be called the current frame.

[0116] In one embodiment, the step of performing marking determination based on the area weight difference of the sector-shaped area to obtain the frame detection mark may include:

[0117] If the regional weight difference of each sector area is greater than a preset weight difference threshold, the sector area is determined to be a change area;

[0118] Count the number of changed areas to obtain the number of changed areas;

[0119] If the number of changed regions is greater than or equal to a preset change number threshold, determining the frame detection mark as an image frame change value;

[0120] If the number of changed areas is less than the change number threshold, the frame detection mark is determined to be the image frame unchanged value.

[0121] Specifically, refer to Figure 7 , assuming that the weight difference threshold is 0.5, the regional weight difference of sector area S1 is 0.6, the regional weight difference of sector area S2 is 0.7, the regional weight difference of sector area S3 is 0.4, and the regional weight difference of sector area S4 is 0.3. Compare the regional weight difference of each sector area with the weight difference threshold. If the regional weight difference is greater than the weight difference threshold, the sector area is determined to be a change area. Since 0.6 and 0.7 are both greater than 0.5, sector area S1 can be determined as the target change area and sector area S2 can be determined as the target change area. Since 0.4 and 0.3 are both less than 0.5, sector areas S3 and S4 cannot be used as target areas.

[0122] By counting the number of all target change regions in the vehicle image frame, the number of change regions can be obtained. For example, based on the above example, the number of change regions can be obtained to be 2.

[0123] After obtaining the number of changed areas, the number of changed areas is compared with a preset change number threshold. If the number of changed areas is greater than or equal to the change number threshold, the frame detection mark is determined to be an image frame change value; if the number of changed areas is less than the change number threshold, the frame detection mark is determined to be an image frame unchanged value. For example, assume that the change number threshold is 2. In the above example, the number of changed areas is 2, which is equal to the change number threshold, and the frame detection mark of the vehicle image frame is an image frame change value (for example, 1). If the number of changed areas is 1, which is less than the change number threshold, the frame detection mark of the vehicle image frame is an image frame unchanged value (for example, 0).

[0124] The benefit of the above embodiment is that the frame detection mark can be determined by using the number of target change regions in the sector area, which has high efficiency and accuracy.

[0125] In step 106, the vehicle image frame is determined as a target change frame according to the image frame change value. In combination with the above, the image frame change value is 1, and the image frame change value is 0. In one example, the frame detection marks of the 1st to 5th vehicle image frames are 1, 1, 0, 0, 1, respectively, and the 1st, 2nd and 5th vehicle image frames are determined as target change frames.

[0126] In step 107, the frame rate is calculated according to the target change frame to obtain the target frame rate. The target frame rate indicates the smoothness of the vehicle-mounted panoramic image video stream.

[0127] In one embodiment, step 107 includes:

[0128] Accumulate the number of target change frames within a preset time period to obtain the number of frame changes;

[0129] The target frame rate is obtained by dividing the number of frame changes by the preset time period.

[0130] For example, the preset time period is 1 second. The number of target change frames within 1 second is accumulated, and the number of frame changes is 20, so the target frame rate is 20 frames / second.

[0131] In one embodiment, the step of accumulating the number of target change frames within a preset time period to obtain the number of frame changes may include:

[0132] Get the time of each vehicle image frame, get the current frame time, and if the vehicle image frame is a target change frame, add 1 to the count;

[0133] The time difference between the current frame time and the preset start frame time is calculated to obtain the cumulative time period;

[0134] If the accumulated time period is greater than the preset time period, the count is used as the number of frame changes, and the start frame time is updated based on the next frame of the vehicle image frame, and the count is updated to 0.

[0135] It should be noted that the frame rate detection can be triggered at a fixed frequency, and the fixed frequency value can be set to five minutes. The fixed frequency can reduce the load of the vehicle detection system and improve the detection stability.

[0136] In one embodiment, after step 105, refer to Figure 8 The vehicle-mounted panoramic image frame rate detection method may further include:

[0137] Step 801, taking a preset number of consecutive vehicle image frames as image frames to be smoothed;

[0138] Step 802, setting increasing weights for a preset number of image frames to be smoothed in chronological order;

[0139] Step 803, performing weighted calculation according to the image frame change values ​​or image frame unchanged values ​​of a preset number of image frames to be smoothed and the weight of each image frame to be smoothed, to obtain frame change smoothing data;

[0140] Step 804: determine a target operation based on the frame change smoothing data, where the target operation includes any one of the following: no operation, warning reporting, or fault reporting.

[0141] The higher the actual frame rate of the on-board panoramic image video stream, the larger the preset number. For example, if the preset number is 5, the N-4th, N-3rd, N-2nd, N-1st and Nth vehicle image frames can all be used as image frames to be smoothed. Increasing weights are set for the 5 image frames to be smoothed in chronological order. The increasing weight specifically means that the weight of the latter of two adjacent image frames to be smoothed is greater than or equal to the weight of the previous one. The weighted calculation formula is: w1*f1+w2*f2+w3*f3+w4*f4+w5*f5. Among them, f1, f2, f3, f4 and f5 represent the frame detection marks of 5 consecutive image frames to be smoothed, and the values ​​are 0 (image frame unchanged value) or 1 (image frame changed value). w1, w2, w3, w4 and w5 represent the weights of the 5 image frames to be smoothed, and the values ​​are set to 0.1, 0.1, 0.2, 0.2, and 0.4 respectively.

[0142] In one example, step 804 includes: if the frame change smooth data is less than or equal to the first smooth threshold, determining the target operation as a fault report; if the frame change smooth data is greater than the first smooth threshold and less than the second smooth threshold, determining the target operation as a warning report; if the frame change smooth data is greater than or equal to the second smooth threshold, determining the target operation as no operation.

[0143] It should be noted that the first balance threshold is smaller than the second smoothing threshold. The first balance threshold can be set to 0.4, and the second smoothing threshold can be set to 0.8.

[0144] The benefit of the above embodiment is that it takes into account the various loads and line complexities of the vehicle-mounted scene, introduces smoothing processing, can accurately perform target operations, can reduce false alarms, and improve reliability.

[0145] In summary, the present application can at least achieve the following effects: (1) Real-time frame rate detection and embedding in the autonomous driving system, without the need for external video recording, and the user does not need to perceive or perform additional operations. (2) It can be applied to in-vehicle panoramic video to expand application scenarios. (3) Detect whether the panoramic image function is normal and improve software security.

[0146] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, any combination of the various technical features in the above embodiments also falls within the scope of this specification.

[0147] The present application also discloses a vehicle-mounted panoramic image frame rate detection device, referring to Fig. 9 The vehicle-mounted panoramic image frame rate detection device includes: a frame acquisition module 901, a vertex mapping module 902, a first division module 903, a second division module 904, a frame detection module 905, a frame determination module 906 and a frame rate calculation module 907. Among them, the frame acquisition module 901 is used to extract a vehicle image frame containing a vehicle image from the on-board panoramic image video stream, the vehicle image frame contains a top view area, and the top view area includes a vehicle area with a vehicle image; the vertex mapping module 902 is used to perform vertex mapping based on the vehicle area to obtain four vertices; the first division module 903 is used to divide the top view area based on the four vertices to obtain at least one target area; wherein each target area contains a vertex; the second division module 904 is used to divide each target area based on the vertex to obtain a fan-shaped area; the frame detection module 905 is used to perform frame change detection based on the fan-shaped areas of two adjacent vehicle image frames to obtain a frame detection mark; wherein the frame detection mark includes an image frame change value, and the image frame change value indicates that the two adjacent vehicle image frames are different; the frame determination module 906 is used to determine the vehicle image frame as a target change frame according to the image frame change value; the frame rate calculation module 907 is used to perform frame rate calculation based on the target change frame to obtain a target frame rate.

[0148] In one embodiment, the vehicle-mounted panoramic image frame rate detection device also includes a smoothing processing module, which is used to: take a preset number of consecutive vehicle image frames as image frames to be smoothed; set increasing weights for the preset number of image frames to be smoothed in chronological order; perform weighted calculation based on the image frame change values ​​or image frame unchanged values ​​of the preset number of image frames to be smoothed, and the weight of each image frame to be smoothed to obtain frame change smoothing data; and determine the target operation based on the frame change smoothing data.

[0149] It should be noted that the specific implementation of the vehicle-mounted panoramic image frame rate detection device is basically the same as the specific implementation example of the above-mentioned vehicle-mounted panoramic image frame rate detection method, and will not be repeated here.

[0150] An embodiment of the present application also discloses a vehicle, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned vehicle-mounted panoramic image frame rate detection method is implemented.

[0151] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned vehicle-mounted panoramic image frame rate detection method.

[0152] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0154] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0155] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0157] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0158] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0159] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0160] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0162] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0163] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A vehicle-mounted panoramic image frame rate detection method, characterized in that: The method comprises: Extracting a vehicle image frame containing a vehicle image from a vehicle-mounted panoramic image video stream, wherein the vehicle image frame contains a top-view area, and the top-view area contains a vehicle area having the vehicle image; Perform vertex mapping based on the vehicle area to obtain four vertices; Dividing the top view area based on the four vertices to obtain at least one target area; wherein each target area includes one vertex; Divide each target area based on the vertices to obtain a sector-shaped area; Performing frame change detection according to the sector-shaped areas of two adjacent vehicle image frames to obtain a frame detection mark; wherein the frame detection mark includes an image frame change value, and the image frame change value indicates that the two adjacent vehicle image frames are different; Determining the vehicle image frame as a target change frame according to the image frame change value; The frame rate is calculated according to the target change frame to obtain a target frame rate.

2. The method according to claim 1, characterized in that The frame detection mark also includes an image frame unchanged value, and the image frame unchanged value indicates that two adjacent vehicle image frames are the same; After performing frame change detection according to the sector-shaped areas of two adjacent vehicle image frames to obtain a frame detection mark, the method further includes: Taking a preset number of continuous vehicle image frames as image frames to be smoothed; Setting increasing weights for a preset number of the image frames to be smoothed in chronological order; Performing weighted calculation according to the image frame change values ​​or the image frame unchanged values ​​of a preset number of the image frames to be smoothed and the weight of each of the image frames to be smoothed, to obtain frame change smoothing data; Based on the frame change smoothing data, a target operation is determined, where the target operation includes any one of the following: no operation, warning reporting, or fault reporting.

3. The method according to claim 1, characterized in that: The performing frame change detection according to the sector-shaped areas of two adjacent vehicle image frames to obtain a frame detection mark includes: Calculating the weight of each sector area to obtain a sector area weight value; Performing difference calculation on the sector-shaped area weight values ​​of two adjacent vehicle image frames to obtain an area weight difference; The frame detection mark is determined by performing a mark determination according to the area weight difference of the sector-shaped area to obtain the frame detection mark.

4. The method according to claim 3, characterized in that The frame detection mark also includes an image frame unchanged value, and the image frame unchanged value indicates that two adjacent vehicle image frames are the same; The step of performing labeling and determining according to the area weight difference of the sector-shaped area to obtain the frame detection label comprises: If the area weight difference of each of the fan-shaped areas is greater than a preset weight difference threshold, the fan-shaped area is determined to be a change area; Counting the number of the changed regions to obtain the number of changed regions; If the number of changed regions is greater than or equal to a preset change number threshold, determining that the frame detection mark is the image frame change value; If the number of changed areas is less than the change number threshold, the frame detection mark is determined to be the unchanged value of the image frame.

5. The method according to any one of claims 1 to 4, characterized in that: The step of dividing the top view area based on the four vertices to obtain at least one target area includes: Taking each of the vertices as an origin, a first cutting line is derived based on a first direction and a second cutting line is derived based on a second direction; wherein the first direction is parallel to at least one side of the top view area, and the first direction and the second direction are perpendicular to each other; The top view area is segmented based on the first segmentation line and the second segmentation line to obtain the target area.

6. The method according to any one of claims 1 to 4, characterized in that: The step of dividing each target area based on the vertices to obtain a sector-shaped area includes: Taking the vertex as the center of the circle, a diagonal line is drawn in the target area; Extending from the diagonal line to both sides, an area with a preset angular arc and a preset radius is selected to obtain the fan-shaped area.

7. The method according to any one of claims 1 to 4, characterized in that: The step of calculating the frame rate according to the target change frame to obtain the target frame rate includes: Accumulating the number of target change frames within a preset time period to obtain the number of frame changes; The target frame rate is obtained by dividing the frame change quantity by the preset time period.

8. A vehicle-mounted panoramic image frame rate detection device, characterized in that: The device comprises: A frame acquisition module, used for extracting a vehicle image frame containing a vehicle image from a vehicle-mounted panoramic image video stream, wherein the vehicle image frame contains a top view area, and the top view area contains a vehicle area having the vehicle image; A vertex mapping module, used for performing vertex mapping based on the vehicle area to obtain four vertices; A first division module is used to divide the top view area into regions based on the four vertices to obtain at least one target region; wherein each target region includes one vertex; A second division module is used to divide each of the target areas based on the vertices to obtain a sector-shaped area; A frame detection module, configured to perform frame change detection according to the sector-shaped areas of two adjacent vehicle image frames to obtain a frame detection mark; wherein the frame detection mark includes an image frame change value, and the image frame change value indicates that the two adjacent vehicle image frames are different; a frame determination module, configured to determine the vehicle image frame as a target change frame according to the image frame change value; The frame rate calculation module is used to calculate the frame rate according to the target change frame to obtain a target frame rate.

9. A vehicle, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images