Pedestrian identification method and device, vehicle, electronic equipment and storage medium

Abstract

According to the pedestrian recognition method and device, the vehicle, the electronic equipment and the storage medium provided by the invention, the preset number of fisheye images are obtained, and the preset number of fisheye images are subjected to pedestrian recognition through the preset pedestrian recognition algorithm, so that the first pedestrian recognition information corresponding to each fisheye image is obtained; the preset number of fisheye images are images corresponding to different angles of the vehicle; performing projection transformation calculation on the preset number of fisheye images and the first pedestrian identification information to obtain a top view corresponding to each fisheye image and second pedestrian identification information corresponding to each top view; and carrying out image splicing on the preset number of top views, and carrying out information fusion processing on each piece of second pedestrian identification information in the top views after image splicing to obtain a target pedestrian identification result. Compared with the prior art, the lightweight pedestrian recognition model structure and the lightweight image transformation fusion model structure are used, so that the computing power demand and the energy consumption of the vehicle can be effectively reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method and device for pedestrian recognition, a vehicle, an electronic device, and a storage medium. Background Art

[0002] With the development of the times, the automotive industry is in an era of change, and the development and application of autonomous driving related technologies are in full swing. Autonomous driving is a cutting-edge technology that relies on computers and artificial intelligence technology to complete complete, safe and effective driving without human control.

[0003] In autonomous driving technology, pedestrian recognition algorithms are very important. The so-called pedestrian recognition refers to the detection and distance measurement of pedestrians around the vehicle through visual algorithms, using fisheye cameras installed in the front, back, left, and right directions to build the positional relationship of pedestrians around the vehicle. Pedestrians around the vehicle can be identified, so as to warn and record potential risk behaviors of some pedestrians approaching the vehicle.

[0004] However, the mainstream pedestrian recognition algorithm in related technologies is mainly implemented through 3D target detection under BEV (Bird's Eye View). The training of the algorithm often relies on the collection of point cloud data by lidar and 3D annotation of targets such as pedestrians. Due to the long data production link and high cost, it is difficult to deploy on vehicle chips with limited computing power, and the algorithm training data is difficult to construct. Summary of the invention

[0005] The present disclosure provides a method and device for pedestrian recognition, a vehicle, an electronic device and a storage medium. The main purpose is to solve the problem that the mainstream pedestrian recognition algorithm in the related art is mainly implemented through 3D target detection under BEV (Bird's Eye View), and the algorithm training often relies on laser radar to collect point cloud data and 3D annotation of targets such as pedestrians. Due to the long data production link and high cost, it is difficult to deploy on the vehicle chip with limited computing power, and the algorithm training data is difficult to build.

[0006] According to a first aspect of the present disclosure, a method for pedestrian recognition is provided, comprising:

[0007] Acquire a preset number of fisheye images, and perform pedestrian recognition on the preset number of fisheye images respectively using a preset pedestrian recognition algorithm to obtain first pedestrian recognition information corresponding to each fisheye image, wherein the preset number of fisheye images are images corresponding to different angles of the vehicle;

[0008] Performing projection transformation calculations on the preset number of fisheye images and the first pedestrian recognition information respectively to obtain a top view corresponding to each fisheye image and second pedestrian recognition information corresponding to each top view;

[0009] After image stitching is performed on a preset number of the bird's-eye views, information fusion processing is performed on each of the second pedestrian recognition information in the bird's-eye views after image stitching to obtain a target pedestrian recognition result.

[0010] Optionally, before acquiring a preset number of fisheye images and performing pedestrian recognition on the preset number of fisheye images respectively using a preset pedestrian recognition algorithm to obtain first pedestrian recognition information corresponding to each fisheye image, the method includes:

[0011] Based on a preset calibration plate, a preset number of calibration plate images are collected by a preset number of fisheye cameras, wherein the positions of the preset calibration plate are different in different calibration plate images;

[0012] The calibration plate image is input into a preset calibration interface for calibration to obtain internal parameters corresponding to each of the fisheye cameras.

[0013] Optionally, after inputting the calibration plate image into a preset calibration interface for calibration to obtain internal parameters corresponding to each of the fisheye cameras, the method includes:

[0014] Collecting a preset number of calibration images including the preset calibration plate and the preset rectangular frame through the preset number of fisheye cameras, wherein the preset calibration plate and the preset rectangular frame are arranged around the vehicle according to a preset positioning rule, and one fisheye camera corresponds to one preset calibration plate;

[0015] Based on the internal parameters, dedistorting the calibration image by using a preset dedistortion algorithm to obtain a dedistorted calibration image;

[0016] Obtaining at least four corner points of the preset calibration plate from each of the dedistorted calibration images, and obtaining corresponding at least four corner point coordinate values, wherein the corner point coordinate values ​​are coordinate values ​​determined with the preset corners of the dedistorted calibration image as the origin;

[0017] The corner point coordinate values ​​and the corresponding expected coordinate values ​​are input into a preset transformation algorithm to obtain an image transformation matrix corresponding to each fisheye image. The expected coordinate values ​​are the coordinate values ​​determined by the corner points of the preset calibration plate in the expected top view, with the preset corners of the expected top view as the origin.

[0018] Optionally, the dedistorting the calibration image by using a preset dedistortion algorithm based on the internal parameters to obtain the dedistorted calibration image includes:

[0019] Inputting the internal parameters into the preset de-distortion algorithm for parameter mapping processing to obtain image mapping relationship information corresponding to the fisheye camera;

[0020] Based on the image mapping relationship information, the calibration image is input into the preset dedistortion algorithm for dedistortion processing to obtain the dedistorted calibration image.

[0021] Optionally, performing pedestrian recognition on the preset number of fisheye images respectively by using a preset pedestrian recognition algorithm to obtain first pedestrian recognition information corresponding to each fisheye image includes:

[0022] Pedestrian image information is extracted from the four fisheye images by using a pedestrian detection algorithm in a preset pedestrian recognition algorithm, wherein the preset pedestrian recognition algorithm includes a pedestrian detection algorithm, a pedestrian key point algorithm, a pedestrian identification algorithm, and a pedestrian tracking algorithm;

[0023] Inputting the pedestrian image information into the pedestrian key point algorithm, positioning and identifying the pedestrian key points in the pedestrian image information, and obtaining pedestrian key point information, wherein the pedestrian key points are bone points on the pedestrian's body;

[0024] Inputting the pedestrian image information into the pedestrian identification algorithm, extracting pedestrian features from the pedestrian image information, and obtaining pedestrian identification information;

[0025] Inputting the pedestrian identification information into the pedestrian tracking algorithm, performing time-series tracking on the pedestrian corresponding to the pedestrian identification information, and obtaining pedestrian time-series information;

[0026] The pedestrian time sequence information, the pedestrian key point information and the pedestrian identification information corresponding to each fisheye image are merged to form the first pedestrian recognition information corresponding to each fisheye image.

[0027] Optionally, performing projection transformation calculation on the preset number of fisheye images and the first pedestrian recognition information respectively to obtain a top view corresponding to each fisheye image and second pedestrian recognition information corresponding to each top view includes:

[0028] Based on the image mapping relationship information, the preset number of fisheye images and the first pedestrian recognition information are input into the preset dedistortion algorithm for dedistortion processing to obtain a preset number of dedistorted fisheye images and dedistorted first pedestrian recognition information;

[0029] Based on the image transformation matrix, the dedistorted fisheye image is input into a preset projection algorithm to obtain the top view corresponding to each fisheye image, and the dedistorted first pedestrian recognition information is input into the preset projection algorithm to obtain the first pedestrian recognition information under the top view;

[0030] Based on the pedestrian key point information in the first pedestrian recognition information in the top view, the distance from the pedestrian to the vehicle is calculated in the top view to obtain relative distance information and relative position information between the pedestrian and the vehicle;

[0031] The relative distance information and the relative position information are combined with the first pedestrian recognition information in the top view to form the second pedestrian recognition information corresponding to each top view.

[0032] Optionally, the performing image stitching on a preset number of the overhead views includes:

[0033] Acquire an overlapping area between the adjacent first top view and the second top view, wherein the overlapping area includes a first area without image information in the first top view and a second area without image information in the second top view;

[0034] Calculate a first distance weight and a second distance weight according to a first distance between each pixel point in the overlapping area and the boundary line of the first area, and a second distance between each pixel point in the overlapping area and the boundary line of the second area, respectively, wherein the first distance weight is a ratio of the pixel value of the pixel point in the first top view to the value in the overlapping area, and the second distance weight is a ratio of the pixel value of the pixel point in the second top view to the value in the overlapping area;

[0035] Performing weighted summation according to the first distance weight and the second distance weight of each pixel point, and the first top view and the second top view to obtain a pixel value corresponding to each pixel point, and performing image stitching of the first top view and the second top view based on the pixel value;

[0036] The above image stitching process is repeated until all the top views are stitched together to obtain a stitched top view.

[0037] Optionally, performing information fusion processing on each of the second pedestrian recognition information in the top view after image splicing to obtain the target pedestrian recognition result includes:

[0038] Acquire the pedestrian timing information and the pedestrian identification information in the second pedestrian recognition information corresponding to each top view, wherein the second pedestrian recognition information includes the pedestrian timing information and the pedestrian identification information;

[0039] Based on the pedestrian time sequence information, the second pedestrian recognition information with the same pedestrian identification information is fused to obtain the target pedestrian recognition result.

[0040] According to a second aspect of the present disclosure, there is provided a pedestrian recognition device, comprising:

[0041] A first acquisition unit, used to acquire a preset number of fisheye images;

[0042] an identification unit, configured to perform pedestrian identification on the preset number of fisheye images respectively by using a preset pedestrian identification algorithm to obtain first pedestrian identification information corresponding to each fisheye image, wherein the preset number of fisheye images are images corresponding to different angles of the vehicle;

[0043] A projection unit, used to perform projection transformation calculation on the preset number of fisheye images and the first pedestrian recognition information respectively, to obtain a top view corresponding to each fisheye image and second pedestrian recognition information corresponding to each top view;

[0044] The fusion unit is used to perform information fusion processing on each of the second pedestrian recognition information in the top view after image stitching processing, so as to obtain a target pedestrian recognition result.

[0045] Optionally, the device further comprises:

[0046] A first acquisition unit is used to acquire a preset number of calibration plate images by using a preset number of fisheye cameras based on a preset calibration plate, wherein the positions of the preset calibration plates in different calibration plate images are different;

[0047] The calibration unit is used to input the calibration plate image into a preset calibration interface for calibration to obtain internal parameters corresponding to each of the fisheye cameras.

[0048] Optionally, the device further comprises:

[0049] A second acquisition unit is used to acquire a preset number of calibration images including the preset calibration plate and the preset rectangular frame through the preset number of fisheye cameras, wherein the preset calibration plate and the preset rectangular frame are arranged around the vehicle according to a preset positioning rule, and one fisheye camera corresponds to one preset calibration plate;

[0050] A processing unit, configured to perform a dedistortion process on the calibration image by using a preset dedistortion algorithm based on the internal parameters to obtain a dedistorted calibration image;

[0051] A second acquisition unit is used to acquire at least four corner points of the preset calibration plate from each of the dedistorted calibration images to obtain corresponding at least four corner point coordinate values, wherein the corner point coordinate values ​​are coordinate values ​​determined with the preset corners of the dedistorted calibration image as the origin;

[0052] An input unit is used to input the corner point coordinate values ​​and the corresponding expected coordinate values ​​into a preset transformation algorithm to obtain an image transformation matrix corresponding to each fisheye image. The expected coordinate values ​​are the coordinate values ​​determined by the corner points of the preset calibration plate in the expected top view, with the preset corners of the expected top view as the origin.

[0053] Optionally, the processing unit is further used for:

[0054] Inputting the internal parameters into the preset de-distortion algorithm for parameter mapping processing to obtain image mapping relationship information corresponding to the fisheye camera;

[0055] Based on the image mapping relationship information, the calibration image is input into the preset dedistortion algorithm for dedistortion processing to obtain the dedistorted calibration image.

[0056] Optionally, the identification unit includes:

[0057] An extraction module, used for extracting pedestrian image information from the four fisheye images by using a pedestrian detection algorithm in a preset pedestrian recognition algorithm, wherein the preset pedestrian recognition algorithm includes a pedestrian detection algorithm, a pedestrian key point algorithm, a pedestrian identification algorithm, and a pedestrian tracking algorithm;

[0058] An input module, used to input the pedestrian image information into the pedestrian key point algorithm, locate and identify the pedestrian key points in the pedestrian image information, and obtain pedestrian key point information, wherein the pedestrian key points are bone points on the pedestrian's body;

[0059] The input module is also used to input the pedestrian image information into the pedestrian identification algorithm, extract pedestrian features from the pedestrian image information, and obtain pedestrian identification information;

[0060] The input module is also used to input the pedestrian identification information into the pedestrian tracking algorithm, perform time-series tracking on the pedestrian corresponding to the pedestrian identification information, and obtain pedestrian time-series information;

[0061] The merging module is used to merge the pedestrian time sequence information, the pedestrian key point information and the pedestrian identification information corresponding to each fisheye image based on a preset integration algorithm to form the first pedestrian recognition information corresponding to each fisheye image.

[0062] Optionally, the projection unit includes:

[0063] An input module, for inputting the preset number of fisheye images and the first pedestrian recognition information into the preset dedistortion algorithm for dedistortion processing based on the image mapping relationship information, so as to obtain a preset number of dedistorted fisheye images and the first pedestrian recognition information after dedistortion;

[0064] The input module is also used to input the dedistorted fisheye image into a preset projection algorithm based on the image transformation matrix to obtain the top view corresponding to each fisheye image, and to input the dedistorted first pedestrian recognition information into the preset projection algorithm to obtain the first pedestrian recognition information under the top view.

[0065] a calculation module, configured to calculate the distance between the pedestrian and the vehicle in the top view based on the pedestrian key point information in the first pedestrian recognition information in the top view, and obtain relative distance information and relative position information between the pedestrian and the vehicle;

[0066] The merging module is used to merge the relative distance information and the relative position information with the first pedestrian recognition information in the top view to form the second pedestrian recognition information corresponding to each top view.

[0067] Optionally, the device further comprises:

[0068] A third acquisition unit is used to acquire an overlapping area between the adjacent first top view and the second top view, wherein the overlapping area includes a first area without image information in the first top view and a second area without image information in the second top view;

[0069] a calculation unit, configured to calculate a first distance weight and a second distance weight according to a first distance between each pixel point in the overlapping area and a boundary line of the first area, and a second distance between each pixel point in the overlapping area and a boundary line of the second area;

[0070] a stitching unit, configured to obtain a pixel value corresponding to each pixel point by performing weighted summation according to the first distance weight and the second distance weight of each pixel point, and the first top view and the second top view, and stitch the first top view and the second top view based on the pixel value;

[0071] The stitching unit is further used to repeat the above image stitching process until all the top views are stitched together to obtain a stitched top view.

[0072] Optionally, the fusion unit includes:

[0073] an acquisition module, configured to acquire the pedestrian timing information and the pedestrian identification information in the second pedestrian recognition information corresponding to each top view, wherein the second pedestrian recognition information includes the pedestrian timing information and the pedestrian identification information;

[0074] A fusion module is used to fuse the second pedestrian recognition information with the same pedestrian identification information based on the pedestrian time sequence information to obtain the target pedestrian recognition result.

[0075] According to a third aspect of the present disclosure, a vehicle is provided, wherein the vehicle includes the pedestrian recognition device as described in the second aspect of the present disclosure.

[0076] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0077] at least one processor; and

[0078] a memory communicatively connected to the at least one processor; wherein,

[0079] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0080] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0081] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0082] The method and device, vehicle, electronic device and storage medium for pedestrian recognition provided by the present disclosure obtain a preset number of fisheye images, and respectively perform pedestrian recognition on the preset number of fisheye images through a preset pedestrian recognition algorithm to obtain first pedestrian recognition information corresponding to each fisheye image, wherein the preset number of fisheye images are images corresponding to different angles of the vehicle; respectively perform projection transformation calculation on the preset number of fisheye images and the first pedestrian recognition information to obtain a top view corresponding to each fisheye image, and second pedestrian recognition information corresponding to each top view; after image splicing the preset number of top views, perform information fusion processing on each of the second pedestrian recognition information in the top view after image splicing to obtain a target pedestrian recognition result. Compared with the related art, the embodiment of the present disclosure uses a lightweight pedestrian recognition model structure and a lightweight image transformation fusion model structure, does not require the collection of cloud data and 3D annotation, has a shorter data production link, and has a lower difficulty in constructing algorithm training data. It has better effects in data production and engineering deployment, and can effectively reduce computing power requirements and vehicle energy consumption.

[0083] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0085] Figure 1 A flowchart of a method for pedestrian recognition provided by an embodiment of the present disclosure;

[0086] Figure 2 A schematic diagram of a surround-view BEV local map provided by an embodiment of the present disclosure;

[0087] Figure 3 A schematic diagram of a camera calibration process provided by an embodiment of the present disclosure;

[0088] Figure 4 A schematic diagram of a fisheye joint calibration process provided by an embodiment of the present disclosure;

[0089] Figure 5 A schematic diagram of fisheye joint calibration provided by an embodiment of the present disclosure;

[0090] Figure 6 A schematic top view of a fisheye image provided by an embodiment of the present disclosure;

[0091] Figure 7 A schematic diagram of splicing and fusion of a top view provided in an embodiment of the present disclosure;

[0092] Figure 8 A schematic diagram of the structure of a pedestrian recognition device provided by an embodiment of the present disclosure;

[0093] Fig. 9 A schematic diagram of the structure of another pedestrian recognition device provided by an embodiment of the present disclosure;

[0094] Fig.10 A schematic block diagram of an electronic device provided in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0095] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0096] The following describes a method and apparatus for pedestrian recognition, a vehicle, an electronic device, and a storage medium according to embodiments of the present disclosure with reference to the accompanying drawings.

[0097] Figure 1 A flowchart of a method for pedestrian recognition provided in an embodiment of the present disclosure.

[0098] like Figure 1 As shown, the method comprises the following steps:

[0099] Step 101, obtaining a preset number of fisheye images, and performing pedestrian recognition on the preset number of fisheye images respectively through a preset pedestrian recognition algorithm to obtain first pedestrian recognition information corresponding to each fisheye image, wherein the preset number of fisheye images are images corresponding to different angles of the vehicle respectively.

[0100] The present disclosure can be applied to any type of vehicle. Specifically, the embodiments of the present disclosure are not limited with respect to the type of vehicle.

[0101] In the embodiments of the present disclosure, the method of acquiring the fisheye image includes but is not limited to: acquiring through fisheye cameras deployed around the vehicle, for example: four-way surround-view fisheye cameras deployed in the front, back, left and right directions of the vehicle, etc. Specifically, the embodiments of the present disclosure do not limit the method of acquiring the fisheye image.

[0102] Among them, the preset pedestrian recognition algorithm is a custom-selected recognition algorithm, for example: HOG (Histogram of Oriented Gradient, feature extraction algorithm) + SVM (support vector machines, support vector machine) pedestrian detection algorithm, etc. It should be noted that the preset pedestrian recognition algorithm includes a pedestrian detection algorithm, a pedestrian key point algorithm, a pedestrian identification algorithm and a pedestrian tracking algorithm, etc., and correspondingly, the first pedestrian recognition information includes pedestrian timing information, the pedestrian key point information and the pedestrian identification information, etc. Specifically, the preset pedestrian recognition algorithm is not limited in the embodiment of the present disclosure.

[0103] Step 102 , performing projection transformation calculation on the preset number of fisheye images and the first pedestrian recognition information respectively to obtain a top view corresponding to each fisheye image and second pedestrian recognition information corresponding to each top view.

[0104] In the disclosed embodiment, the top view corresponding to the fisheye image is: fisheye images at different angles are transformed into the perspective of the object or scene viewed from above through a transformation matrix. The second pedestrian recognition information is: the information after the first pedestrian recognition information is transformed into the top view through a transformation matrix, mainly by projecting the pedestrian key point information, transforming the coordinates of the pedestrian key point information in the fisheye image to the top view, and obtaining the distance and position relationship of the pedestrian relative to the vehicle by calculating the distance from the pedestrian key point information to the circumscribed rectangle of the vehicle.

[0105] Projection transformation calculation means: assuming that the same camera is used to shoot the same plane at two positions A and B, two images are generated. The transformation relationship between the two images is the projection transformation, which is the process of transforming one image to another. For example: position A is the position where the fisheye camera is installed on the car body, and position B is a position above the car parallel to the ground. The fisheye image can be transformed into a bird's-eye view through the transformation matrix, that is, the picture taken by the fisheye camera installed on the car body is transformed to the picture taken by a camera above the car parallel to the ground.

[0106] Step 103: after performing image stitching on a preset number of the overhead views, perform information fusion processing on each of the second pedestrian recognition information in the overhead views after image stitching to obtain a target pedestrian recognition result.

[0107] In the embodiment of the present disclosure, after stitching a preset number of the top views, a BEV (Bird's Eye View) covering the vehicle body surrounding environment at 360° can be obtained, that is, a top view after image stitching, specifically, Figure 2 As shown, Figure 2 A schematic diagram of a surround-view BEV local map provided in an embodiment of the present disclosure.

[0108] Regarding the information fusion processing of each of the second pedestrian identification information in the overhead view after the image is stitched, it means that after the second pedestrian identification information is projected onto the overhead view after the image is stitched, there will be repeated content. Therefore, it is necessary to fuse the pedestrian identification information with repeated content projected under the BEV. After fusing the pedestrian identification information of the same pedestrian, unique pedestrian identification information corresponding to each pedestrian is obtained.

[0109] The method for pedestrian recognition provided by the present disclosure obtains a preset number of fisheye images, and performs pedestrian recognition on the preset number of fisheye images respectively through a preset pedestrian recognition algorithm to obtain first pedestrian recognition information corresponding to each fisheye image, wherein the preset number of fisheye images are images corresponding to different angles of the vehicle; performs projection transformation calculation on the preset number of fisheye images and the first pedestrian recognition information respectively to obtain a top view corresponding to each fisheye image, and second pedestrian recognition information corresponding to each top view; after image splicing the preset number of top views, information fusion processing is performed on each of the second pedestrian recognition information in the top view after image splicing to obtain a target pedestrian recognition result. Compared with the related art, the embodiment of the present disclosure uses a lightweight pedestrian recognition model structure and a lightweight image transformation fusion model structure, does not require the collection of cloud data and 3D annotation, has a shorter data production link, and has a lower difficulty in constructing algorithm training data. It has better effects in data production and engineering deployment, and can effectively reduce computing power requirements and vehicle energy consumption.

[0110] In one possible implementation of the embodiment of the present disclosure, the fisheye image needs to go through the steps of camera calibration, fisheye joint calibration, fisheye image dedistortion, projection transformation and image splicing fusion before the fused top view can be obtained. At the same time, the camera calibration and fisheye joint calibration need to be performed manually in advance. Therefore, regarding the camera calibration, the present disclosure provides a schematic diagram of the camera calibration process, such as Figure 3 As shown, including:

[0111] Step 301 : Based on a preset calibration plate, a preset number of calibration plate images are collected by a preset number of fisheye cameras, wherein the positions of the preset calibration plate are different in different calibration plate images.

[0112] In the embodiment of the present disclosure, the preset calibration plate is a custom selected calibration plate, such as a two-dimensional grid calibration plate, etc. The camera calibration method can be calibrated by, but not limited to, Zhang's calibration method. Specifically, the embodiment of the present disclosure does not limit the preset calibration plate and the camera calibration method.

[0113] The preset number is a custom-set number, for example, 15, 20, etc. The more calibration plate images are collected, the better. The calibration effect is best within a certain range. When using the preset calibration plate to collect calibration plate images, it is usually necessary to make the preset calibration plate appear in different positions of the screen and take multiple shots. Generally, about 20 calibration plate images are collected.

[0114] Step 302: input the calibration plate image into a preset calibration interface for calibration to obtain internal parameters corresponding to each of the fisheye cameras.

[0115] In the embodiment of the present disclosure, the preset calibration interface is a custom selected calibration interface, for example: a camera calibration interface provided by MATLAB or OpenCV, etc. The internal parameters include but are not limited to: camera focal length, optical center, radial distortion coefficient, etc. Specifically, the embodiment of the present disclosure does not limit the preset calibration interface and the internal parameters.

[0116] In one possible implementation of the embodiment of the present disclosure, regarding the fisheye joint calibration in the steps of the above-mentioned fisheye image, the present disclosure provides a flow chart of the fisheye joint calibration, such as Figure 4 As shown, including:

[0117] Step 401: collect a preset number of calibration images including the preset calibration plate and the preset rectangular frame through the preset number of fisheye cameras, wherein the preset calibration plate and the preset rectangular frame are set around the vehicle according to preset positioning rules, and one fisheye camera corresponds to one preset calibration plate.

[0118] In the disclosed embodiment, the construction of the vehicle's global surround view map (the bird's-eye view after fusion processing) requires converting independent fisheye images to the same coordinate system for fusion. This process requires obtaining the positional relationship between the fisheye cameras through fisheye joint calibration, and then fusing the captured images into a global surround view map.

[0119] The preset rectangular frame is a black rectangular frame selected by user, and the preset positioning rule is a rule selected by user. For the convenience of understanding, the present disclosure provides a schematic diagram of fisheye joint calibration, such as Figure 5As shown, when placing the preset calibration plate, two distances need to be determined. The first is the distance between the preset calibration plate and the vehicle body, and the second is the distance between the preset calibration plate and the border of the top view. These two distances can be determined according to the desired visual range of the top view. For example, we hope to obtain a top view with a visual range of 3m. After this visual range is determined, we place the calibration plate according to the 1:1 scale relationship between the real world and the panoramic view, that is, 1 pixel in the panoramic view represents 1cm in the real world. At the same time, the front, back, left, and right positions of the preset calibration plate are placed in the middle area of ​​the visual range, that is, symmetrical with the center of the vehicle. Figure 5 The middle black position is a black rectangular frame.

[0120] Step 402: Based on the internal parameters, dedistortion processing is performed on the calibration image by using a preset dedistortion algorithm to obtain a dedistorted calibration image.

[0121] In the embodiment of the present disclosure, the calibration image needs to be dedistorted to ensure the accuracy of the fisheye joint calibration. The preset dedistortion algorithm is a custom selected dedistortion algorithm, for example, the algorithm in opencv, etc. Specifically, the embodiment of the present disclosure does not limit the preset dedistortion algorithm.

[0122] Step 403: Obtain at least four corner points of the preset calibration plate from each of the dedistorted calibration images, and obtain corresponding at least four corner point coordinate values, wherein the corner point coordinate values ​​are coordinate values ​​determined with the preset corners of the dedistorted calibration image as the origin.

[0123] In the embodiments of the present disclosure, the corner points of the preset calibration plate are obtained, which need to be manually obtained by a technician. The obtained position is generally self-selected, but the selected point must be the corner point where black and white intersect in the preset calibration plate. The preset corners are the corners of the calibration image selected by the user, for example: the upper left corner of the calibration image, etc. Specifically, the embodiments of the present disclosure do not limit the preset corners.

[0124] Step 404, input the corner point coordinate values ​​and the corresponding expected coordinate values ​​into a preset transformation algorithm to obtain an image transformation matrix corresponding to each fisheye image, wherein the expected coordinate values ​​are coordinate values ​​determined by the corner points of the preset calibration plate in the expected top view with the preset corners of the expected top view as the origin.

[0125] In the embodiment of the present disclosure, the preset transformation algorithm is a transformation algorithm selected by a user, for example, findHomography or getPerspectiveTransform function in opencv, and the preset corner is the corner of the desired top view selected by the user, for example, the upper left corner of the desired top view (such as Figure 5 Specifically, the embodiments of the present disclosure do not limit the preset corners.

[0126] It should be noted here that the corners of the desired top view must be consistent with the corners of the calibration image in the above embodiment, that is, the upper left corner, the upper right corner, etc.

[0127] In one implementable manner of the disclosed embodiment, the fisheye image has a very large orientation range, resulting in relatively serious distortion and imaging distortion in the image. The ground, pedestrians, and vehicles in the fisheye image appear curved, especially in the four corners of the picture. Therefore, the fisheye image is dedistorted by internal parameters. In order to reduce the distortion and distortion of the fisheye image, it can be implemented in but not limited to the following manner: inputting the internal parameters into the preset dedistortion algorithm for parameter mapping processing to obtain the image mapping relationship information corresponding to the fisheye camera; based on the image mapping relationship information, inputting the calibration image into the preset dedistortion algorithm for dedistortion processing to obtain the dedistorted calibration image.

[0128] In the implementation of the present disclosure, when parameter mapping processing is performed through the preset dedistortion algorithm, it can be processed through the parameter mapping processing function in the preset dedistortion algorithm. Regarding the parameter mapping processing function, for example: the cv2.fisheye.initUndistortRectifyMap function provided by opencv, etc. When parameter mapping processing is performed through the preset dedistortion algorithm, it can be processed through the dedistortion processing function in the preset dedistortion algorithm. Regarding the dedistortion processing function, for example: the cv2.remap function provided by opencv, etc. Specifically, regarding the first preset mapping algorithm and the second preset mapping algorithm, the embodiment of the present disclosure does not limit it.

[0129] In one implementable method of the disclosed embodiment, when performing pedestrian recognition, it is necessary to transform the pedestrian recognition result into a top view to construct the relative position relationship between the pedestrian and the vehicle. In addition, pedestrian recognition also needs to track the same pedestrian as he moves around the vehicle. Therefore, in order to improve the effect of pedestrian recognition, it can be implemented in the following ways but not limited to: extracting pedestrian image information from the four fisheye images through a pedestrian detection algorithm in a preset pedestrian recognition algorithm, the preset pedestrian recognition algorithm includes a pedestrian detection algorithm, a pedestrian key point algorithm, a pedestrian identification algorithm and a pedestrian tracking algorithm; inputting the pedestrian image information into the pedestrian key point algorithm, locating and identifying the pedestrian key points in the pedestrian image information, and obtaining the pedestrian key points. The pedestrian key points are the bone points on the pedestrian's body; the pedestrian image information is input into the pedestrian identification algorithm, the pedestrian features in the pedestrian image information are extracted, and the pedestrian identification information is obtained; the pedestrian identification information is input into the pedestrian tracking algorithm, the pedestrian corresponding to the pedestrian identification information is tracked in time sequence, and the pedestrian time sequence information is obtained; the pedestrian time sequence information, the pedestrian key point information and the pedestrian identification information corresponding to each fisheye image are merged to form the first pedestrian recognition information corresponding to each fisheye image.

[0130] The pedestrian detection algorithm, pedestrian key point algorithm, pedestrian identification algorithm and pedestrian tracking algorithm in the preset pedestrian recognition algorithm are custom selected algorithms. For example, the pedestrian detection algorithm may be HOG (Histogram of Oriented Gradient, feature extraction algorithm), the pedestrian identification algorithm may be an algorithm in the pedestrian ReID model, the pedestrian key point algorithm may be an algorithm in the pedestrian key point model, the pedestrian tracking algorithm may be an algorithm based on DeepSort, etc. Specifically, the pedestrian detection algorithm, pedestrian key point algorithm, pedestrian identification algorithm and pedestrian tracking algorithm in the preset pedestrian recognition algorithm are not limited in the embodiments of the present disclosure.

[0131] The pedestrian key point information includes but is not limited to: 16 skeletal points of the pedestrian (head, neck, left shoulder, right shoulder, left elbow, left wrist, right elbow, right wrist, left hip, right hip, left knee, right knee, left heel, right heel, left toe, right toe), etc. The preset integration algorithm is a custom selection integration algorithm, for example: the pedestrian time sequence information, the pedestrian key point information and the pedestrian identification information are digitally integrated, etc.

[0132] In order to facilitate understanding of the implementation process of the disclosed embodiment, an example is provided for illustration: the pedestrians around the car are located by the pedestrian detection model to obtain the pedestrian position; then, according to the pedestrian detection result, the pedestrian image is extracted from the original image and sent to the pedestrian key point model and the pedestrian ReID model respectively. Among them, the pedestrian key point model will locate the 16 skeletal points of the pedestrian (head, neck, left shoulder, right shoulder, left elbow, left wrist, right elbow, right wrist, left hip, right hip, left knee, right knee, left heel, right heel, left toe, right toe); and the pedestrian ReID model extracts the features of the pedestrian to obtain the pedestrian identification information, and sends it to the pedestrian tracking algorithm based on DeepSort to achieve the time-series tracking of pedestrians in the monocular image, and output the pedestrian time-series information of different pedestrians. Finally, the detection frame, pedestrian identification information, pedestrian key points and pedestrian time-series information corresponding to the pedestrian in the single fisheye image are integrated into a whole, which is used as the first pedestrian recognition information.

[0133] In one possible implementation of the embodiment of the present disclosure, as a refinement of the above step 102, it is also necessary to perform a dedistortion process on the fisheye image and then perform a projection transformation to ensure that the top view is not distorted and distorted. Therefore, in order to improve the effect of the projection transformation, it can be implemented in the following manner but not limited to: based on the image mapping relationship information, the preset number of fisheye images and the first pedestrian recognition information are input into the preset dedistortion algorithm for dedistortion processing to obtain a preset number of dedistorted fisheye images and the dedistorted first pedestrian recognition information; based on the image transformation matrix, the dedistorted fisheye images The fisheye image is input into a preset projection algorithm to obtain the top view corresponding to each fisheye image, and the first pedestrian recognition information after dedistortion is input into the preset projection algorithm to obtain the first pedestrian recognition information in the top view; based on the pedestrian key point information in the first pedestrian recognition information in the top view, the distance from the pedestrian to the vehicle is calculated in the top view to obtain the relative distance information and relative position information between the pedestrian and the vehicle; the relative distance information and the relative position information are merged with the first pedestrian recognition information in the top view to form the second pedestrian recognition information corresponding to each top view.

[0134] In the implementation of the present disclosure, the preset projection algorithm is a custom-selected projection algorithm, for example, the remap function provided by opencv, etc. Specifically, the present disclosure embodiment does not limit the preset projection algorithm.

[0135] Among them, the calculation of the relative distance information and the relative position information refers to projecting the pedestrian key point information in the first pedestrian recognition information, converting the key point coordinates in the pedestrian key point information in the original fisheye image to the BEV view, and then calculating the distance from the key point of the pedestrian's footsteps to the circumscribed rectangle of the vehicle based on the key point of the pedestrian's footsteps in the pedestrian key point information to obtain the relative distance information and relative position information between the pedestrian and the vehicle. Finally, the relative distance information and the relative position information are integrated with the first pedestrian recognition information in the overhead view as a whole, and together serve as the second pedestrian recognition information.

[0136] In an implementable manner of the embodiment of the present disclosure, before the fused top view and the second pedestrian recognition information are fused, a preset number of the top views need to be fused to complete the construction of the vehicle-surround BEV local map. Therefore, in order to increase the effect of pedestrian recognition, the following method may also be used but is not limited to: obtaining an overlapping area between adjacent first top views and second top views, wherein in the overlapping area, there are a first area without image information in the first top view and a second area without image information in the second top view; calculating a first distance weight and a second distance weight according to a first distance between each pixel point in the overlapping area and a boundary line of the first area, and a second distance between each pixel point in the overlapping area and a boundary line of the second area; performing weighted summation according to the first distance weight and the second distance weight of each pixel point, and the first top view and the second top view to obtain a pixel value corresponding to each pixel point, and performing image stitching of the first top view and the second top view based on the pixel value; repeating the above image stitching process until all the top views are stitched together to obtain the top view after image stitching.

[0137] In the embodiment of the present disclosure, for ease of understanding, the embodiment of the present disclosure provides a schematic diagram of a top view of a fisheye image, such as Figure 6 As shown in the figure, after the projection transformation is completed, a top view of the fisheye image is obtained. The converted top view often has a black triangular area. When stitching, Figure 7 As shown, Figure 7 A schematic diagram of a top view splicing fusion provided in an embodiment of the present disclosure is provided, and the repeated areas in the two images are found based on the coordinates of the black rectangular areas in the two top views, such as Figure 7As shown in the white rectangular area in the middle. The distances from all pixels to the AB and CD edges in the repeated area are calculated, and the weights are calculated based on the two distances. The weight obtained from the distance from the pixel to the AB edge represents the weight of the pixel in the front fisheye top view, while the weight obtained from the distance from the pixel to the CD edge represents the weight of the pixel in the left fisheye top view. The weighted sum of the two is the pixel value taken at the pixel after the fusion.

[0138] In one implementable method of the embodiment of the present disclosure, since there will be repeated content during image fusion, when the second pedestrian recognition information is fused, after the second pedestrian recognition information is projected onto the fused overhead view, the pedestrian recognition information with repeated content projected onto the BEV needs to be fused. Therefore, in order to avoid duplication of pedestrian recognition information, it can be implemented in but not limited to the following manner: obtaining the pedestrian timing information and the pedestrian identification information in the second pedestrian recognition information corresponding to each overhead view, the second pedestrian recognition information including the pedestrian timing information and the pedestrian identification information; based on the pedestrian timing information, the second pedestrian recognition information with the same pedestrian identification information is fused to obtain the target pedestrian recognition result.

[0139] In an embodiment of the present disclosure, regarding the merge processing of the second pedestrian identification information with the same pedestrian identification information, for example: when the pedestrian identification information is the same, the second pedestrian identification information with a later time in the pedestrian timing information is used to cover the second pedestrian identification information with an earlier time in the pedestrian timing information.

[0140] In summary, the embodiments of the present disclosure can achieve the following effects:

[0141] 1. The disclosed embodiment uses a lightweight pedestrian recognition model structure and a lightweight image transformation fusion model structure, which eliminates the need to collect cloud data and 3D annotations. The data production link is short, the algorithm training data construction difficulty is low, and the effect is better in data production and engineering deployment, which can effectively reduce computing power requirements and vehicle energy consumption.

[0142] 2. The embodiment of the present disclosure performs pedestrian recognition by using the pedestrian detection algorithm, pedestrian key point algorithm, pedestrian identification algorithm and pedestrian tracking algorithm in the preset pedestrian recognition algorithm, so as to realize the positioning, ranging and tracking of pedestrians around the vehicle.

[0143] Corresponding to the above-mentioned pedestrian recognition method, the present invention also provides a pedestrian recognition device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.

[0144] Figure 8 A schematic diagram of a pedestrian recognition device provided by an embodiment of the present disclosure is shown in FIG. Figure 8 As shown, including:

[0145] A first acquisition unit 801 is used to acquire a preset number of fisheye images;

[0146] The recognition unit 802 is used to perform pedestrian recognition on the preset number of fisheye images respectively by using a preset pedestrian recognition algorithm to obtain first pedestrian recognition information corresponding to each fisheye image, wherein the preset number of fisheye images are images corresponding to different angles of the vehicle;

[0147] The projection unit 803 is used to perform projection transformation calculation on the preset number of fisheye images and the first pedestrian recognition information respectively to obtain a top view corresponding to each fisheye image and second pedestrian recognition information corresponding to each top view;

[0148] The fusion unit 804 is used to perform information fusion processing on each of the second pedestrian recognition information in the top views after image stitching processing, so as to obtain a target pedestrian recognition result.

[0149] The pedestrian recognition device provided by the present disclosure obtains a preset number of fisheye images, and respectively performs pedestrian recognition on the preset number of fisheye images through a preset pedestrian recognition algorithm to obtain first pedestrian recognition information corresponding to each fisheye image, wherein the preset number of fisheye images are images corresponding to different angles of the vehicle; respectively performs projection transformation calculation on the preset number of fisheye images and the first pedestrian recognition information to obtain a top view corresponding to each fisheye image, and second pedestrian recognition information corresponding to each top view; after image splicing the preset number of top views, information fusion processing is performed on each of the second pedestrian recognition information in the top view after image splicing to obtain a target pedestrian recognition result. Compared with the related art, the embodiment of the present disclosure uses a lightweight pedestrian recognition model structure and a lightweight image transformation fusion model structure, does not need to collect cloud data and 3D annotation, has a shorter data production link, and has a lower difficulty in constructing algorithm training data. It has better effects in data production and engineering deployment, and can effectively reduce computing power requirements and vehicle energy consumption.

[0150] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Fig. 9 As shown, the device also includes:

[0151] A first acquisition unit 805 is configured to acquire a preset number of calibration plate images by using a preset number of fisheye cameras based on a preset calibration plate, wherein the positions of the preset calibration plates in different calibration plate images are different;

[0152] The calibration unit 806 is used to input the calibration plate image into a preset calibration interface for calibration to obtain internal parameters corresponding to each of the fisheye cameras.

[0153] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Fig. 9 As shown, the device also includes:

[0154] A second acquisition unit 807 is used to acquire a preset number of calibration images including the preset calibration plate and the preset rectangular frame through the preset number of fisheye cameras, wherein the preset calibration plate and the preset rectangular frame are arranged around the vehicle according to a preset positioning rule, and one fisheye camera corresponds to one preset calibration plate;

[0155] A processing unit 808 is used to perform dedistortion processing on the calibration image based on the internal parameters by using a preset dedistortion algorithm to obtain a dedistorted calibration image;

[0156] A second acquisition unit 809 is used to acquire at least four corner points of the preset calibration plate from each of the dedistorted calibration images, and obtain corresponding at least four corner point coordinate values, wherein the corner point coordinate values ​​are coordinate values ​​determined with the preset corners of the dedistorted calibration image as the origin;

[0157] The input unit 810 is used to input the corner point coordinate values ​​and the corresponding expected coordinate values ​​into a preset transformation algorithm to obtain an image transformation matrix corresponding to each fisheye image. The expected coordinate values ​​are the coordinate values ​​determined by the corner points of the preset calibration plate in the expected top view, with the preset corners of the expected top view as the origin.

[0158] Furthermore, in a possible implementation of the embodiment of the present disclosure, the processing unit 808 is further configured to:

[0159] Inputting the internal parameters into the preset de-distortion algorithm for parameter mapping processing to obtain image mapping relationship information corresponding to the fisheye camera;

[0160] Based on the image mapping relationship information, the calibration image is input into the preset dedistortion algorithm for dedistortion processing to obtain the dedistorted calibration image.

[0161] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Fig. 9 As shown, the identification unit 802 includes:

[0162] An extraction module 8021 is used to extract pedestrian image information from the four fisheye images by using a pedestrian detection algorithm in a preset pedestrian recognition algorithm, wherein the preset pedestrian recognition algorithm includes a pedestrian detection algorithm, a pedestrian key point algorithm, a pedestrian identification algorithm, and a pedestrian tracking algorithm;

[0163] An input module 8022 is used to input the pedestrian image information into the pedestrian key point algorithm, locate and identify the pedestrian key points in the pedestrian image information, and obtain pedestrian key point information, where the pedestrian key points are bone points on the pedestrian's body;

[0164] The input module 8022 is further used to input the pedestrian image information into the pedestrian identification algorithm, extract pedestrian features from the pedestrian image information, and obtain pedestrian identification information;

[0165] The input module 8022 is further used to input the pedestrian identification information into the pedestrian tracking algorithm, perform time-series tracking on the pedestrian corresponding to the pedestrian identification information, and obtain pedestrian time-series information;

[0166] The merging module 8023 is used to merge the pedestrian time sequence information, the pedestrian key point information and the pedestrian identification information corresponding to each fisheye image based on a preset integration algorithm to form the first pedestrian recognition information corresponding to each fisheye image.

[0167] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Fig. 9 As shown, the projection unit 803 includes:

[0168] An input module 8031 ​​is used to input the preset number of fisheye images and the first pedestrian recognition information into the preset dedistortion algorithm for dedistortion processing based on the image mapping relationship information, so as to obtain a preset number of dedistorted fisheye images and the first pedestrian recognition information after dedistortion;

[0169] The input module 8031 ​​is also used to input the dedistorted fisheye image into a preset projection algorithm based on the image transformation matrix to obtain the top view corresponding to each fisheye image, and input the dedistorted first pedestrian recognition information into the preset projection algorithm to obtain the first pedestrian recognition information under the top view.

[0170] A calculation module 8032 is used to calculate the distance between the pedestrian and the vehicle in the top view based on the pedestrian key point information in the first pedestrian recognition information in the top view, so as to obtain relative distance information and relative position information between the pedestrian and the vehicle;

[0171] The merging module 8033 is used to merge the relative distance information and the relative position information with the first pedestrian recognition information in the top view to form the second pedestrian recognition information corresponding to each top view.

[0172] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Fig. 9 As shown, the device also includes:

[0173] A third acquisition unit 811 is used to acquire an overlapping area between adjacent first top view and second top view, wherein the overlapping area includes a first area without image information in the first top view and a second area without image information in the second top view;

[0174] A calculation unit 812, configured to calculate a first distance weight and a second distance weight according to a first distance between each pixel point in the overlapping area and a boundary line of the first area, and a second distance between each pixel point in the overlapping area and a boundary line of the second area;

[0175] a stitching unit 813, configured to perform weighted summation according to the first distance weight and the second distance weight of each pixel point, and the first top view and the second top view to obtain a pixel value corresponding to each pixel point, and stitch the first top view and the second top view based on the pixel value;

[0176] The stitching unit 813 is further used to repeat the above image stitching process until all the top views are stitched together to obtain a stitched top view.

[0177] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Fig. 9 As shown, the fusion unit 804 includes:

[0178] An acquisition module 8041 is used to acquire the pedestrian timing information and the pedestrian identification information in the second pedestrian recognition information corresponding to each top view, wherein the second pedestrian recognition information includes the pedestrian timing information and the pedestrian identification information;

[0179] The fusion module 8042 is used to fuse the second pedestrian recognition information with the same pedestrian identification information based on the pedestrian time sequence information to obtain the target pedestrian recognition result.

[0180] In an embodiment of the present disclosure, a vehicle is also provided, wherein the vehicle is equipped with a pedestrian recognition device.

[0181] It should be noted that the above explanation of the method embodiment is also applicable to the device of the embodiment of the present disclosure, and the principle is the same, which is no longer limited in the embodiment of the present disclosure.

[0182] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0183] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0184] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 1002 or a computer program loaded from a storage unit 1008 to a RAM (Random Access Memory) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An I / O (Input / Output) interface 1005 is also connected to the bus 1004.

[0185] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0186] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as a method for pedestrian recognition. For example, in some embodiments, the method for pedestrian recognition may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the aforementioned pedestrian recognition method in any other appropriate manner (for example, by means of firmware).

[0187] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0188] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0189] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory) or a flash memory, an optical fiber, a CD-ROM (Compact Dis sc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0190] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0191] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0192] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0193] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0194] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0195] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for pedestrian recognition, characterized in that: include: Acquire a preset number of fisheye images, and perform pedestrian recognition on the preset number of fisheye images respectively using a preset pedestrian recognition algorithm to obtain first pedestrian recognition information corresponding to each fisheye image, wherein the preset number of fisheye images are images corresponding to different angles of the vehicle; Performing projection transformation calculations on the preset number of fisheye images and the first pedestrian recognition information respectively to obtain a top view corresponding to each fisheye image and second pedestrian recognition information corresponding to each top view; After image stitching is performed on a preset number of the bird's-eye views, information fusion processing is performed on each of the second pedestrian recognition information in the bird's-eye views after image stitching to obtain a target pedestrian recognition result.

2. The method according to claim 1, characterized in that: Before obtaining a preset number of fisheye images and performing pedestrian recognition on the preset number of fisheye images respectively using a preset pedestrian recognition algorithm to obtain first pedestrian recognition information corresponding to each fisheye image, the method includes: Based on a preset calibration plate, a preset number of calibration plate images are collected by a preset number of fisheye cameras, wherein the positions of the preset calibration plate are different in different calibration plate images; The calibration plate image is input into a preset calibration interface for calibration to obtain internal parameters corresponding to each of the fisheye cameras.

3. The method according to claim 2, characterized in that After the calibration plate image is input into a preset calibration interface for calibration to obtain internal parameters corresponding to each of the fisheye cameras, the method includes: Collecting a preset number of calibration images including the preset calibration plate and the preset rectangular frame through the preset number of fisheye cameras, wherein the preset calibration plate and the preset rectangular frame are arranged around the vehicle according to a preset positioning rule, and one fisheye camera corresponds to one preset calibration plate; Based on the internal parameters, dedistorting the calibration image by using a preset dedistortion algorithm to obtain a dedistorted calibration image; Obtaining at least four corner points of the preset calibration plate from each of the dedistorted calibration images, and obtaining corresponding at least four corner point coordinate values, wherein the corner point coordinate values ​​are coordinate values ​​determined with the preset corners of the dedistorted calibration image as the origin; The corner point coordinate values ​​and the corresponding expected coordinate values ​​are input into a preset transformation algorithm to obtain an image transformation matrix corresponding to each fisheye image. The expected coordinate values ​​are the coordinate values ​​determined by the corner points of the preset calibration plate in the expected top view, with the preset corners of the expected top view as the origin.

4. The method according to claim 3, characterized in that The step of performing dedistortion processing on the calibration image by using a preset dedistortion algorithm based on the internal parameters to obtain a dedistorted calibration image includes: Inputting the internal parameters into the preset de-distortion algorithm for parameter mapping processing to obtain image mapping relationship information corresponding to the fisheye camera; Based on the image mapping relationship information, the calibration image is input into the preset dedistortion algorithm for dedistortion processing to obtain the dedistorted calibration image.

5. The method according to claim 1, characterized in that The performing pedestrian recognition on the preset number of fisheye images respectively by using a preset pedestrian recognition algorithm to obtain first pedestrian recognition information corresponding to each fisheye image includes: Pedestrian image information is extracted from the four fisheye images by using a pedestrian detection algorithm in a preset pedestrian recognition algorithm, wherein the preset pedestrian recognition algorithm includes a pedestrian detection algorithm, a pedestrian key point algorithm, a pedestrian identification algorithm, and a pedestrian tracking algorithm; Inputting the pedestrian image information into the pedestrian key point algorithm, positioning and identifying the pedestrian key points in the pedestrian image information, and obtaining pedestrian key point information, wherein the pedestrian key points are bone points on the pedestrian's body; Inputting the pedestrian image information into the pedestrian identification algorithm, extracting pedestrian features from the pedestrian image information, and obtaining pedestrian identification information; Inputting the pedestrian identification information into the pedestrian tracking algorithm, performing time-series tracking on the pedestrian corresponding to the pedestrian identification information, and obtaining pedestrian time-series information; The pedestrian time sequence information, the pedestrian key point information and the pedestrian identification information corresponding to each fisheye image are merged to form the first pedestrian recognition information corresponding to each fisheye image.

6. The method according to claim 4, characterized in that The performing projection transformation calculation on the preset number of fisheye images and the first pedestrian recognition information respectively to obtain a top view corresponding to each fisheye image and the second pedestrian recognition information corresponding to each top view includes: Based on the image mapping relationship information, the preset number of fisheye images and the first pedestrian recognition information are input into the preset dedistortion algorithm for dedistortion processing to obtain a preset number of dedistorted fisheye images and dedistorted first pedestrian recognition information; Based on the image transformation matrix, the dedistorted fisheye image is input into a preset projection algorithm to obtain the top view corresponding to each fisheye image, and the dedistorted first pedestrian recognition information is input into the preset projection algorithm to obtain the first pedestrian recognition information under the top view; Based on the pedestrian key point information in the first pedestrian recognition information in the top view, the distance from the pedestrian to the vehicle is calculated in the top view to obtain relative distance information and relative position information between the pedestrian and the vehicle; The relative distance information and the relative position information are combined with the first pedestrian recognition information in the top view to form the second pedestrian recognition information corresponding to each top view.

7. The method according to claim 1, characterized in that The image stitching of a preset number of the top views comprises: Acquire an overlapping area between the adjacent first top view and the second top view, wherein the overlapping area includes a first area without image information in the first top view and a second area without image information in the second top view; Calculate a first distance weight and a second distance weight according to a first distance between each pixel point in the overlapping area and a boundary line of the first area, and a second distance between each pixel point in the overlapping area and a boundary line of the second area; Performing weighted summation according to the first distance weight and the second distance weight of each pixel point, and the first top view and the second top view to obtain a pixel value corresponding to each pixel point, and performing image stitching of the first top view and the second top view based on the pixel value; The above image stitching process is repeated until all the top views are stitched together to obtain a stitched top view.

8. The method according to claim 5, characterized in that The performing information fusion processing on each second pedestrian recognition information in the top view after image splicing to obtain the target pedestrian recognition result includes: Acquire the pedestrian timing information and the pedestrian identification information in each of the second pedestrian recognition information, wherein the second pedestrian recognition information includes the pedestrian timing information and the pedestrian identification information; Based on the pedestrian time sequence information, the second pedestrian recognition information with the same pedestrian identification information is fused to obtain the target pedestrian recognition result.

9. A pedestrian recognition device, characterized in that: include: A first acquisition unit, used to acquire a preset number of fisheye images; an identification unit, configured to perform pedestrian identification on the preset number of fisheye images respectively by using a preset pedestrian identification algorithm to obtain first pedestrian identification information corresponding to each fisheye image, wherein the preset number of fisheye images are images corresponding to different angles of the vehicle; A projection unit, used to perform projection transformation calculation on the preset number of fisheye images and the first pedestrian recognition information respectively, to obtain a top view corresponding to each fisheye image and second pedestrian recognition information corresponding to each top view; The fusion unit is used to perform information fusion processing on each of the second pedestrian recognition information in the top view after image stitching processing, so as to obtain a target pedestrian recognition result.

10. A vehicle, characterized in that: The vehicle comprises the pedestrian recognition device as claimed in claim 9.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

13. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images