Smart cockpit integrated system and pedestrian detection method thereof

Abstract

This disclosure provides a smart cockpit integrated system. [Solution] The surround view video capture device is used to capture surround view video of the area around the vehicle. The surround view video detection server is connected to the surround view video capture device to receive surround view video and includes a pedestrian detection module, a video analysis module, and a warning module. The pedestrian detection module is used to detect pedestrians in the surround view video using a pedestrian detection model and then generate a detection frame surrounding the pedestrians in the surround view video. The video analysis module is used to determine detection points on the detection frame based on the relative position of the detection frame to the video center and to calculate the pedestrian distance between the detection points and the video center. The warning module is used to determine whether the pedestrian distance is less than or equal to the warning distance and to generate a warning signal.

Description

Technical Field

[0001] The present disclosure relates to a smart cockpit integration system and a pedestrian detection method thereof, and particularly to a smart cockpit integration system capable of detecting pedestrians and issuing corresponding warnings, and a pedestrian detection method thereof.

Background Art

[0002] In a conventional vehicle surround view system, since lenses with a narrow field of view are used, it is necessary to install a plurality of lenses in a plurality of directions of the vehicle to supplement the blind spots in the driver's field of view.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Furthermore, large vehicles and machinery used in places other than general roads have safety requirements for the working environment that are not inferior to the driving safety requirements on general roads. However, due to the special use locations and the smaller types and quantities of vehicles and machinery compared to general vehicles, there are few examples of developing smart cockpit integration systems corresponding to such vehicles, machinery, and their corresponding working locations, and it is difficult to meet the safety requirements of special locations.

Means for Solving the Problems

[0004] This disclosure provides a smart cockpit integrated system including a surround-view video capture device and a surround-view video detection server. The surround-view video capture device is used to capture surround-view video of the area around the vehicle, and the surround-view video extends upward into the air with the image centered below the surround-view video capture device. The surround-view video detection server is connected to the surround-view video capture device to receive the surround-view video and includes a pedestrian detection module, an image analysis module, and a warning module. The pedestrian detection module is used to detect pedestrians in the surround-view video using a pedestrian detection model and then generate a detection frame surrounding pedestrians in the surround-view video. The image analysis module is connected to the pedestrian detection module and is used to determine detection points on the detection frame based on the relative position of the detection frame to the image center and to calculate the pedestrian distance between the detection points and the image center. The warning module is connected to the image analysis module and is used to determine whether the pedestrian distance is less than or equal to the warning distance and, if so, generate a warning signal.

[0005] In one embodiment, the pedestrian image is the range from the upper body to the entire body of the pedestrian in any rotational direction within the surround view image.

[0006] In one embodiment, the warning module is further used to determine whether the pedestrian video does not correspond to the entire pedestrian, and if so, to generate a warning signal.

[0007] In one embodiment, the video analysis module performs distortion correction processing on the corresponding range of the detection frame before calculating the pedestrian distance.

[0008] In one embodiment, the warning distance includes a first warning distance and a second warning distance, and the warning signal includes a first warning signal and a second warning signal. The warning module generates a first warning signal when it determines that the pedestrian distance is less than or equal to the first warning distance, and generates a second warning signal when it determines that the pedestrian distance is less than or equal to the second warning distance.

[0009] In one embodiment, the acquisition areas for multiple training surround-view images for training a pedestrian detection model include at least one of a mine, forestry field, cargo dock, construction site, farmland, and warehouse.

[0010] In one embodiment, multiple training surround view videos for training a pedestrian detection model include multiple augmented surround view videos that have undergone data augmentation through angle-of-view transformation.

[0011] In one embodiment, the smart cockpit integrated system further includes a display device connected to the surround view video detection server, and the surround view video detection server further includes a video conversion module connected to a warning module. The video conversion module is used to convert the surround view video into at least one planar video and transmit it to the display device for display.

[0012] This disclosure further provides a pedestrian detection method used in a smart cockpit integrated system. The pedestrian detection method includes capturing surround view video of the vehicle; detecting the surround view video and then generating a detection frame surrounding the pedestrian video within the surround view video; determining a detection point on the detection frame based on the relative position of the detection frame to the video center, calculating the pedestrian distance between the detection point and the video center; and determining whether the pedestrian distance is less than a warning distance, and if so, generating a warning signal. The surround view video extends into the air with the bottom of the surround view video capture device as the video center.

[0013] In one embodiment, the pedestrian image is the range from the upper body to the entire body of the pedestrian in any rotational direction within the surround view image. [Brief explanation of the drawing]

[0014] For a more complete understanding of the examples and their advantages, please refer to the following description in conjunction with the attached drawings. [Figure 1]This is a schematic diagram of a smart cockpit integrated system in one embodiment of the present disclosure. [Figure 2] This is a schematic diagram illustrating the installation of a surround-view video capture device in one embodiment of the present disclosure. [Figure 3] A schematic rear view diagram of the mounting of a surround-view video capture device in one embodiment of the present disclosure. [Figure 4] This is a schematic diagram of surround view video in one embodiment of the present disclosure. [Figure 5] This is a flowchart of a pedestrian detection method in one embodiment of the present disclosure. [Figure 6] This is a schematic diagram of pedestrian detection in surround view video in one embodiment of the present disclosure. [Figure 7] This is a schematic diagram of data augmentation of training surround view video in one embodiment of the present disclosure. [Figure 8] This is a schematic diagram illustrating the display of surround view video in one embodiment of the present disclosure. [Modes for carrying out the invention]

[0015] The embodiments of this disclosure are described in detail below. However, to make it clear, the embodiments provide many applicable concepts that can be implemented in various specific content. The embodiments of the disclosure discussed are for illustrative purposes only and do not limit the scope of this disclosure.

[0016] As used in this disclosure, the term “connection” means a direct or indirect electrical or communication connection, and “first” and “second” are used solely to distinguish elements of multiple identical or similar concepts and do not refer to any specific sequential relationship between the multiple elements. Furthermore, when describing a particular person, object, or event in a video in this disclosure, it should be understood that the video refers to the corresponding particular person, object, or event in the video, even if the term is not specifically followed by “video.”

[0017] This disclosure provides a smart cockpit integrated system and a pedestrian detection method thereof that detects the area around a vehicle or equipment in a special region, determines whether a pedestrian is approaching based on that detection, generates a warning, and thereby ensures the safety of pedestrians around the vehicle.

[0018] Figure 1 is a schematic diagram of a smart cockpit integrated system 100 in one embodiment of the present disclosure. As shown in Figure 1, the smart cockpit integrated system 100 includes a surround view video detection server 110, a surround view video capture device 120, a warning device 130, and a display device 140, the surround view video detection server 110 being connected to the surround view video capture device 120, the warning device 130, and the display device 140. The surround view video detection server 110 includes a pedestrian detection module 111, a video analysis module 112, a warning module 113, and a video conversion module 114, the pedestrian detection module 111 being connected to the video analysis module 112, the video analysis module 112 being connected to the warning module 113, and the warning module 113 being connected to the video conversion module 114, and each module can transmit data or signals to each other, and is not limited to transmission between the two interconnected modules as described above. After the surround view video of the vehicle's surroundings is captured by the surround view video detection server 110, it is transmitted to the surround view video detection server 110 for pedestrian detection and analysis, the warning device 130 issues a corresponding warning, and the display device 140 displays the corresponding video.

[0019] In one embodiment, the surround view imaging device 120 may be a fisheye lens. When this fisheye lens is directed downward for imaging, it can capture a surround view image within a hemispherical range of 360 degrees around with the downward direction as the image center and at least 180 degrees from the rear through the downward direction to the front. Therefore, compared with a general imaging device, even in the case of a small number (e.g., one), it can capture surround view images of the surroundings with an extremely wide viewing angle. The surround view imaging device 120 can be attached to a vehicle (e.g., a general vehicle, a large construction vehicle, or an agricultural vehicle, etc.). The surround view image detection server 110, the warning device 130, and the display device 140 may be devices installed in this vehicle and / or devices installed in a monitoring room and remotely connected. Alternatively, the surround view image detection server 110 may be a small device installed in the vehicle together with the surround view imaging device 120, and the present disclosure does not particularly limit these.

[0020] The areas where the smart cockpit integrated system 100 can be used include, for example, mines, forest farms, cargo ports, construction sites, farmlands, warehouses, etc., and can be attached to vehicles and machines in these areas, such as forklifts, tillage tractors, excavator trucks, etc. Thereby, the operator can use the smart cockpit integrated system 100 to detect whether nearby pedestrians (e.g., on-site workers) appear around the vehicle or machine and issue corresponding warnings. Hereinafter, taking the vehicle as an example, the attachment of the surround view imaging device 120 in the smart cockpit integrated system 100 will be described.

[0021] FIG. 2 is a schematic diagram of the installation of the surround view imaging device 120 in an embodiment of the present disclosure. As shown in the side schematic diagram of FIG. 2(a), the surround view imaging device 120 is installed behind the vehicle 200 using a bracket. In one embodiment, the vehicle 200 is a forklift. Referring to the top schematic diagram shown in FIG. 2(b), the front of the vehicle 200 faces the direction D. At this time, the range of the surround view image of the vehicle surroundings captured by the surround view imaging device 120 is the range A of the angle a. When the surround view imaging device 120 is installed via an extension bracket from a high position (such as on the roof or sunroof) of the vehicle and is attached to a position extended by a distance d1 behind the vehicle 200, the range of the surround view image of the vehicle surroundings captured by the surround view imaging device 120 at this time is the range B of the angle b. The ranges A and B are used only to indicate the corresponding ranges of the angles a and b, and the farthest distance that the surround view imaging device 120 can capture is not limited to the ranges A and B, but depends on the farthest distance that the actual device can capture.

[0022] As can be seen from this, the distance between the attachment position of the surround view imaging device 120 to the vehicle 200 and the vehicle body affects its captureable angle and range. Generally, the driver's seat of the vehicle 200 is installed in the front of the vehicle body of the vehicle 200 (i.e., on the direction D side of the vehicle 200), and the surrounding scenery in the front can be directly judged by the driver. However, when the vehicle body is large, it is necessary to install an imaging device to complement the driver's visual range on the vehicle body. For example, based on the type of the vehicle body and the range where detection is required, the attachment position of the surround view imaging device 120 to the vehicle 200 can be determined. Also, in one embodiment, when the window or the vehicle body is transparent or not shielded, even if the surround view imaging device 120 is attached to the rear side of the vehicle 200, through the unshielded part of the window or the vehicle body, a wider viewing angle can be detected compared to the non-transparent vehicle body, and the side or front area of the driver can be included.

[0023] Furthermore, the mounting height of the surround-view video capture device 120 on the vehicle also affects its shooting range. Figure 3 is a schematic rear view diagram of the mounting of the surround-view video capture device 120 in one embodiment of the present disclosure. Referring to Figure 3, if the surround-view video capture device 120 is mounted above the vehicle 200, the shooting range of the vehicle 200 is limited by the vehicle body itself. For example, if the surround-view video capture device 120 is mounted above the vehicle 200 at a distance d2, only objects within the range of angles c1 and c2 will be captured, and ranges C1 and C2 will be blind spots for the surround-view video capture device 120.

[0024] As shown in Figure 3, when pedestrians H1 and H2 are standing on the ground G around the vehicle 200, pedestrian H1 is standing within the range of angle c2, so the entire body of pedestrian H1 is captured by the surround-view video capture device 120. However, since pedestrian H2 is standing on the ground G corresponding to range C2, the surround-view video capture device 120 can only capture the upper body of pedestrian H1 within the range of angle c2. However, in this case, pedestrian H2 is closer to the vehicle 200, i.e., the danger level at the location where pedestrian H2 is standing is higher than that of pedestrian H1. Therefore, if it is necessary to detect and warn of pedestrians based on the surround-view video after the surround-view video capture device 120 has captured the video, then the upper half of pedestrian H2's video must also be detectable. As can be seen from this, the range that can be captured can be determined by adjusting the installation height of the surround-view video capture device 120, and therefore the surround-view video capture device 120 can be mounted, for example, at a high position on the vehicle (such as on the roof or sunroof) via an extension bracket. For example, if the vehicle height is 190 cm, the surround view video capture device 120 can be installed at a height of 190 cm to 300 cm.

[0025] Figure 4 is a schematic diagram of a surround view image 400 in one embodiment of the present disclosure, which is an unprocessed image of a person visible to the human eye, captured by the surround view image capture device 120. In one embodiment, the surround view image capture device 120 is mounted above the vehicle 200 and captures images downwards, and the surround view image 400 extends from the ground G towards the air with the image center 410 below it. When standing in a normal direction, the images of pedestrians H3, H4, and H5 all show their feet facing the image center 410, and when observing and analyzing the surround view image 400 after capture, it is possible that pedestrians H3, H4, and H5 are rotating in any direction. Also, as described above, pedestrian H5 is closer to the vehicle and therefore located in the blind spot of the surround view image capture device 120, and thus only the upper body of pedestrian H5 is captured in the image.

[0026] Figure 5 is a flowchart of the surround view video detection method 500 in one embodiment of the present disclosure. The operation of the smart cockpit integrated system 100 will be described below with reference to Figures 1 and 5.

[0027] First, in step S510, a surround view video is captured using the surround view video capture device 120, and the surround view video extends from the ground towards the air with the center of the video as the focal point. Next, in step S520, the surround view video detection server 110 receives the surround view video, and in step S530, the pedestrian detection module 111 detects the surround view video using a pedestrian detection model and generates a detection frame that surrounds the pedestrian video within the surround view video.

[0028] Figure 6 is a schematic diagram of pedestrian detection in a surround view image 600 in one embodiment of the present disclosure. In Figure 6, since the surround view image capture device 120 is pointed directly downwards, the image center 410 of the captured surround view image 600 is the ground directly below the surround view image capture device 120, and the surrounding scenery is the scenery around the surround view image capture device 120. As shown in Figure 6, the surround view image 600 includes pedestrians H6, H7, H8, and H9, which exist within the surround view image 600 at different rotational directions around the image center 410, and each is surrounded by a detection frame F. Specifically, the pedestrian detection model is used to detect pedestrians within the surround view image 600, and when the pedestrian detection model detects the presence of a pedestrian within the surround view image 600, it generates a corresponding detection frame F surrounding this pedestrian. Generally, this detection frame F is a rectangular frame that surrounds the body range of the detected pedestrian. For example, regardless of the pedestrian's posture, the direction of rotation within the surround view image, or whether it is the whole body or just the upper body (as in pedestrian H9), if the presence of a pedestrian is detected, it will be enclosed in this detection frame F. Here, the upper body refers to the range from the non-whole body including the head to the whole body, and this disclosure does not impose any particular limitations on this, as long as it is detectable by the pedestrian detection model and can be identified as a pedestrian, such as above the knees, above the waist, or above the chest.

[0029] Here, the pedestrian detection model is an object detection model that can identify the presence of pedestrians through supervised or unsupervised learning methods and can distinguish whether the detected pedestrian is half-body or full-body. When the surround view image 600 is captured by a fisheye lens, as shown in Figure 4, the scenery in it is distorted by the wide angle, and pedestrians H3, H4, and H5 exist in different rotational directions, not with their heads up and feet down as in a typical image, and may be in various postures such as walking, standing, running, and crouching. This smart cockpit integrated system 100 is used in special areas other than general roads, such as mines belonging to specific industries, forests, cargo docks, construction sites, farms, and warehouses, and each area has its own places where pedestrian detection is difficult. For example, in forests and farms where tall crops are planted, the crops around the vehicle are tall and cluttered, making it easy for pedestrians to be obscured or difficult to identify in the image. For the reasons mentioned above, the source of training data required for the pedestrian detection model is less and more difficult than for general roads. Therefore, training surround-view footage for pedestrian detection models can be captured in large quantities in multiple specific areas using a fisheye lens, and these training surround-view images include pedestrian footage in various movements, at least from the upper body to the whole body, and in any direction of rotation.

[0030] In another embodiment, data augmentation may be performed on these training surround view videos to increase the amount of training data, for example, by splitting and / or angle-of-view transformation to obtain an augmented surround view video. Figure 7 is a schematic diagram of data augmentation of training surround view videos in one embodiment of the present disclosure. As shown in Figure 7, the training surround view video 700 is a video captured by the surround view video capture device 120, and by splitting, the original video can be divided into augmented surround view video 710 and augmented surround view video 720 to increase the amount of training data with different video sizes, resolutions, and orientations. Furthermore, since it is very important to train the pedestrian detection model to detect pedestrians with different rotation directions, the pedestrian H10 included in the training surround view video 700 can be rotated by angle-of-view transformation to obtain a pedestrian H10' with a different rotation direction from the original, as shown in the augmented surround view video 720. After video augmentation, a larger amount of video can be acquired to train the pedestrian detection model. Furthermore, data augmentation methods also include style conversion and processing of the degree of video alteration, and this disclosure does not impose any particular limitations on these.

[0031] After surrounding the pedestrian image in step S530, the process proceeds to step S540, where, as shown in Figure 6, the video analysis module 112 determines the detection points S on the detection frame F based on the relative position of the detection frame F with respect to the video center 410. The method for determining the detection points S involves dividing the surround view image 600 into multiple blocks with respect to the video center 410, for example, by dividing the surround view image 600 into eight areas at 45-degree intervals using boundary lines L (the boundary lines L in Figure 6 are schematic only and do not necessarily need to be drawn on the surround view image 600). If the video analysis module 112 determines that the detection frame F of pedestrian H6 is in the area to the left of the video center 410, it determines the detection point S of pedestrian H6 to be at the right edge of the detection frame F. If it determines that the detection frame F of pedestrian H7 is in the area above the video center 410, it determines the detection point S of pedestrian H7 to be at the lower edge of the detection frame F. If it determines that the detection frame F of pedestrian H8 is in the area to the upper right of the video center 410, it determines the detection point S of pedestrian H8 to be at the lower left edge of the detection frame F. Similarly, if only the upper body of pedestrian H9 is captured, and it determines that the detection frame F of pedestrian H9 is in the area to the lower right of the video center 410, it determines the detection point S of pedestrian H9 to be at the upper left edge of the detection frame F.

[0032] In one embodiment, the boundary line L can further divide the surround view image 600 into more blocks, for example, by dividing 360 degrees into 1-degree blocks, the position of the detection point S can be determined with greater precision. Alternatively, the detection point S may be defined as the point where the line connecting the center of the detection frame F and the image center 410 passes through the detection frame F, but this disclosure does not impose any particular limitations on this.

[0033] After determining the detection point S, the process proceeds to step S550, where the video analysis module 112 calculates the pedestrian distance between the detection point S and the video center 410, i.e., the distance between pedestrians H6-H9 and the video center 410. The method for calculating the pedestrian distance involves using the camera's internal and external parameters to convert video coordinates and spatial coordinates, and obtaining the relative position of the pedestrians in space with respect to the surround view video capture device 120 based on their positions in the video. Furthermore, when attempting to calculate the pedestrian distance using the camera's internal and external parameters, it is necessary to perform calibration, such as chessboard calibration, when installing the surround view video capture device 120 to obtain internal and external parameters such as focal length, coordinates, and distortion coefficient.

[0034] In one embodiment, when calculating pedestrian distance, the video analysis module 112 performs distortion correction processing on the video range corresponding to the detection point S to obtain an undeformed planar image and calculate the pedestrian distance. Compared to a method in which distortion correction processing is performed directly after acquiring surround view video 600, and then detection, direction confirmation, and pedestrian distance calculation are performed using a pedestrian detection model, performing the above steps on the unprocessed surround view video 600 and then performing distortion correction processing on a portion of the video after confirming the detection point S reduces the computational cost required for distortion correction processing. In particular, the areas to which the smart cockpit integrated system 100 of this disclosure is applied may be areas with underdeveloped communication infrastructure, where cloud resources cannot be used and only local servers are available, or where only small servers are installed for convenience, reducing the amount of computation contributes to improving system performance and expanding the range of applicable scenarios.

[0035] After calculating the pedestrian distance, the process proceeds to step S560, where the warning module 113 determines whether the pedestrian distance is less than or equal to the warning distance. If so, it proceeds to step S570 to generate a warning signal; otherwise, it repeats step S520, continuously receiving surround view video and performing subsequent operations such as pedestrian detection. However, different warning distances can be set to serve as the criteria for generating a warning signal. For example, as shown in Figure 6, different first warning distances R1 and second warning distances R2 can be set (the first and second warning distances R1 and R2 in Figure 6 are schematic and do not necessarily need to be depicted in the surround view video 600).

[0036] Specifically, in one embodiment, the first warning distance R1 can be set to a radius of 5 meters from the video center 410, and the second warning distance R2 can be set to a radius of 2 meters from the video center 410. As shown in Figure 6, if the corresponding detection point S of pedestrian H6 is located outside the first warning distance R1, it indicates that it is currently more than 5 meters away from the video center 410, and even if pedestrian H6 is detected, no corresponding warning signal is issued. If the corresponding detection point S of pedestrian H7 is between the first warning distance R1 and the second warning distance R2, it indicates that it is currently located between 2 and 5 meters from the video center 410, and therefore, the warning module 113 determines and issues a corresponding first warning signal. If the corresponding detection point S of pedestrian H8 is located exactly at the first warning distance R1, it indicates that it is currently located 5 meters from the video center 410, and therefore, for example, the warning module 113 determines and issues a corresponding first warning signal. If the corresponding detection point S of pedestrian H9 is located within the second warning distance R2, it indicates that it is currently located within 2 meters of the center of the video 410, meaning that pedestrian H9 is very close to the vehicle, and therefore the warning module 113 determines and issues a corresponding second warning signal.

[0037] Furthermore, in one embodiment, in addition to determining the position of the detection point S and the warning distance, the proportion of the detected body of the pedestrian is also considered. For example, if, after pedestrian H9 is detected, it is determined that the pedestrian image is not of the whole body, for example, if only the upper body is detected, there is a very high probability that the lower body is in the blind spot of the surround view video capture device 120, that is, the position of area C2 where pedestrian H2 is standing in Figure 3. Standing in the blind spot means that the pedestrian is very close to the vehicle, so the warning module 113 needs to issue a corresponding second warning signal.

[0038] In another embodiment, in steps S550-S570, the surround-view video detection server 110 not only calculates the position of the corresponding detection point S of the pedestrian and determines whether to issue a corresponding warning based on that position, but can also receive the vehicle's current direction of movement and speed from the vehicle system. After the pedestrian detection module 111 detects a pedestrian, the video analysis module 112 analyzes the predicted collision time of the pedestrian based on the vehicle's current direction of movement, speed, and the pedestrian's position, direction of movement, and distance. If the predicted collision time of the pedestrian is shorter than the collision time limit, the warning module 113 issues a corresponding warning signal.

[0039] After the warning module 113 emits a warning signal, it may transmit it to a warning device 130, such as a speaker or warning light, mounted on the vehicle, or to a display device 140, thereby prompting the driver inside the vehicle to stop and pay attention to surrounding pedestrians, issuing a warning to surrounding pedestrians outside the vehicle, or transmitting a warning to the control room of the vehicle or machine. The first and second warning signals can correspond to different forms of expression; for example, the first warning signal corresponds to the warning light continuously emitting a red light, while the more urgent second warning signal corresponds to the warning light emitting a red light and the speaker emitting an audible sound. The warning device 130 can immediately stop or avoid an accident in conjunction with the control device of the vehicle or machine. The warning distance and the number of corresponding warning signals may also be multiple, and this disclosure is not limited thereto.

[0040] In one embodiment, to more intuitively communicate the relationship between pedestrians and the vehicle and the corresponding warnings to the driver, a simulation coordinate center is set, for example, at the center of the vehicle or the driver's seat, and the warning module 113 coordinate-transforms the pedestrian's distance and position based on the actual distance between the video center 410 and this simulation coordinate center. For example, if a pedestrian is located directly behind the vehicle at a distance of 3 meters from the video center 410, and the actual distance between the video center 410 and the simulation coordinate center is 0.5 meters, the distance between the pedestrian and the simulation coordinate center is converted to 3.5 meters, and the corresponding spatial coordinates are calculated. In another embodiment, the position of the pedestrian on the boundary line L in Figure 6 can also be further converted to the pedestrian's position relative to the simulation coordinate center. Alternatively, in another embodiment, the video center 410 in Figure 6 can be replaced with the simulation coordinate center, and the pedestrian's position and distance can be obtained directly based on the relative position and distance between this simulation coordinate center and the detection frame.

[0041] This allows the warning module 113 to determine the content of the warning signal based on the simulated coordinate center. For example, if a pedestrian is standing within 3 meters to the right of the vehicle's center, the warning module 113 determines that the pedestrian is on the right and issues a second warning signal indicating the pedestrian is on the right. If the pedestrian is standing 4 meters to the left rear of the vehicle's center, the warning module 113 determines that the pedestrian is to the left rear and issues a first warning signal indicating the pedestrian is to the left rear, displaying a schematic screen on the display device 140 or indicating the corresponding position and color-coded light. For example, the vehicle could be schematicly represented, with the pedestrian's diagram flashing to the right and left rear of the vehicle diagram, and / or displayed with different colors or light codes. Furthermore, by improving the resolution of the directional classification in the video analysis module 112, the pedestrian's position can be indicated more accurately in the warning signal.

[0042] Furthermore, in one embodiment, the image from the surround view video capture device 120 extends outward from the image center 410, and considering the situation where the captured object exists in different rotational directions and is partially deformed, if it is necessary to view the actual image on a monitor, such an image is not intuitive to the human eye and is difficult to recognize. Therefore, step S580 can be executed, and after the surround view video detection server 110 receives the surround view video in step S520, the video conversion module 114 converts and generates multiple planar images by performing video division and distortion correction, and then executes step S590 to transmit and display the image on the display device 140, thereby making it easier for drivers and supervisors to monitor this surround view video.

[0043] Figure 8 is a schematic diagram of the display of surround view video in one embodiment of the present disclosure. As shown in Figure 8, the planar image 810 is the original 360-degree surround view video unfolded, and the video conversion module 114 can further divide the planar image 810 into a planar image 821 of the 180-degree front view, a planar image 822 of the 90-degree left rear view, and a planar image 823 of the 90-degree right rear view. The planar images 821, 822, and 823 are then displayed on the display device 140 to facilitate human monitoring. However, since data contained in the original surround view video may be lost due to image deformation correction and trimming during the processing of the surround view video, pedestrian detection, pedestrian distance calculation, and warning generation in steps S530 to S570 are all performed on the original surround view video.

[0044] According to the smart cockpit integrated system and pedestrian detection method disclosed herein, a surround-view video camera is used to capture images of the area around a vehicle or machine in a special region, and based on these images, it is possible to detect and determine whether a pedestrian is approaching, issue a warning, and further enhance the data to solve the problem of insufficient training data related to special regions.

[0045] Although this disclosure is disclosed in the embodiments described above, these embodiments are not intended to limit the disclosure, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of this disclosure; therefore, the scope of protection of this disclosure is as defined by the claims appended thereto. [Explanation of Symbols]

[0046] 100: Smart Cockpit Integrated System 110: Surround View Video Detection Server 111: Pedestrian detection module 112: Video Analysis Module 113: Warning Module 114: Video conversion module 120: Surround View Video Recording Device 130: Warning device 140:Display device 200: Vehicles 400, 600: Surround view video 410: Image focus 500: Surround View Image Detection Method 700: Surround view video for training 710, 720: Extended surround view video 810, 821, 822, 823: Plane image A, B, C1, C2: Range a,b,c1,c2:Angle D: Direction d1,d2: distance F: Detection frame G: Ground H1~H10, H10': Pedestrians L: Boundary line R1: First warning distance R2: 2nd warning distance S: Detection point S510~S590: Process

Claims

1. A smart cockpit integrated system, A surround view video recording device used to capture surround view video of the area around a vehicle, wherein the surround view video is projected downwards from the bottom of the surround view video recording device toward the air, A surround view video detection server connected to the surround view video recording device and receiving the surround view video, Equipped with, The surround view video detection server is A pedestrian detection module that, after detecting the surround view video using a pedestrian detection model, generates a detection frame surrounding the pedestrian video in the surround view video, A video analysis module connected to the pedestrian detection module determines a detection point on the detection frame based on the relative position of the detection frame with respect to the video center, and calculates the pedestrian distance between the detection point and the video center. A smart cockpit integrated system including a warning module connected to the video analysis module, which determines whether the pedestrian distance is less than or equal to the warning distance and generates a warning signal if so.

2. The smart cockpit integrated system according to claim 1, wherein the pedestrian image is the range from the upper body to the whole body of the pedestrian in any rotational direction in the surround view image.

3. The smart cockpit integrated system according to claim 2, wherein the warning module is further used to determine whether the pedestrian video does not correspond to the entire body of the pedestrian, and if so, to generate the warning signal.

4. The smart cockpit integrated system according to claim 1, wherein the video analysis module performs distortion correction processing on the corresponding range of the detection frame before calculating the pedestrian distance.

5. The smart cockpit integrated system according to claim 1, wherein the warning distance includes a first warning distance and a second warning distance, the warning signal includes a first warning signal and a second warning signal, and the warning module generates the first warning signal when it determines that the pedestrian distance is less than or equal to the first warning distance, and generates the second warning signal when it determines that the pedestrian distance is less than or equal to the second warning distance.

6. The smart cockpit integrated system according to claim 1, wherein the acquisition area for multiple training surround view images for training the pedestrian detection model includes at least one of a mine, a forest, a cargo dock, a construction site, farmland, and a warehouse.

7. The smart cockpit integrated system according to claim 1, wherein the multiple training surround view images for training the pedestrian detection model include multiple augmented surround view images that have undergone data augmentation by viewing angle transformation.

8. The system further includes a display device connected to the surround view video detection server, wherein the surround view video detection server further includes a video conversion module connected to the warning module. The smart cockpit integrated system according to claim 1, wherein the video conversion module is used to convert the surround view video into at least one planar video and transmit it to the display device for display.

9. A pedestrian detection method used in a smart cockpit integrated system, To capture surround view video of the vehicle, After detecting the surround view video, a detection frame is generated that surrounds the pedestrian video in the surround view video. Based on the relative position of the detection frame with respect to the center of the image, a detection point on the detection frame is determined, and the pedestrian distance between the detection point and the center of the image is calculated. The system determines whether the pedestrian distance is less than the warning distance, and if so, generates a warning signal. Includes, Furthermore, the surround view image is a pedestrian detection method in which the image extends upward into the air with the area below the surround view image capture device as the image center.

10. The pedestrian detection method according to claim 9, wherein the pedestrian image is the range from the upper body to the whole body of the pedestrian in any rotation direction in the surround view image.

Images