Traffic modeling method, device and system
The real-life model of traffic scenes is constructed through image acquisition equipment, which solves the error and high cost problems introduced by high-precision maps, and realizes low-cost and accurate traffic modeling, which is suitable for a variety of traffic scenarios.
Patent Information
- Application Number
- CN202310517816.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-09
- Publication Date
- 2024-11-19
AI Technical Summary
The existing traffic modeling solutions rely on high-precision maps, resulting in high production costs, large errors and difficulty in automatic updates online, affecting the accuracy and versatility of modeling results.
Acquire images through multiple image acquisition devices, convert and stitch the field of view, build panoramic bird's eye view images and add depth information to generate real-life model images, avoiding the use of high-precision maps and auxiliary information.
It reduces the difficulty and cost of implementation of traffic modeling, improves the accuracy and adaptability of modeling results, and is suitable for various traffic scenarios, especially large scenarios such as highways and airports.
Smart Images

Figure CN118982568A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent transportation, and in particular to a traffic modeling method, device and system. Background Art
[0002] Traffic modeling refers to the construction of a visual model of a traffic road scene (such as a traffic intersection), which can help users intuitively understand the operating conditions of traffic roads and is of great significance to improving the operational efficiency, service level, and safety of urban traffic.
[0003] In the current traffic modeling scheme, the visualization model of the traffic road scene is constructed by matching the image of the traffic road scene with the high-precision map of the traffic road. However, the production process of high-precision maps requires a lot of manual participation. Due to human factors, errors are inevitable in high-precision maps, which easily affect the modeling results. Summary of the invention
[0004] The present application provides a traffic modeling method, device and system. The technical solution of the present application is as follows.
[0005] In a first aspect, a traffic modeling method is provided, the method comprising: acquiring a plurality of images, the plurality of images being respectively obtained by a plurality of image acquisition devices performing image acquisition on a target road scene at a first moment; performing field of view conversion on the plurality of images to obtain a plurality of bird's-eye view images corresponding one to one to the plurality of images; splicing the plurality of bird's-eye view images to obtain a panoramic bird's-eye view image of the target road scene; and adding depth information to the panoramic bird's-eye view image to obtain a real-scene model image of the target road scene.
[0006] Since the multiple images are obtained by the multiple image acquisition devices respectively acquiring images of the target road scene, the multiple images are all camera field of view images. Camera field of view images refer to images directly captured by an image acquisition device (such as a camera), and camera field of view images are also called camera perspective images or similar names. Bird's-eye view images are images observed from a bird's-eye view, and bird's-eye view images are also called bird's-eye field of view images, bird's-eye perspective images, or overhead images, and similar names.
[0007] The technical solution provided by the present application constructs a real-life model image of the target road scene based on the image acquired by the image acquisition device, that is, constructs a real-life model of the target road scene. In the process of constructing the model, there is no need to use a high-precision map, so the error introduced by the high-precision map can be avoided from affecting the modeling results.
[0008] Optionally, performing field of view conversion on the multiple images includes: performing field of view conversion on a first image among the multiple images according to a field of view conversion matrix of a first image acquisition device among the multiple image acquisition devices, the first image being acquired by the first image acquisition device. The field of view conversion matrix is used to convert the image acquired by the first image acquisition device into a bird's-eye view image.
[0009] Optionally, the field of view conversion matrix is obtained by a camera field of view image (e.g., camera field of view image A1) and a field of view conversion model, wherein the camera field of view image A1 is acquired by a first image acquisition device, and the field of view conversion model is obtained by training based on training data, wherein the training data includes a correspondence between a preset camera field of view image and a preset field of view conversion matrix. The field of view conversion model is used to calculate the field of view conversion matrix of the image acquisition device based on the image acquired by the image acquisition device (i.e., the camera field of view image).
[0010] The technical solution provided in the present application determines the field of view conversion matrix of the first image acquisition device based on the field of view conversion model and the camera field of view image acquired by the first image acquisition device, and can facilitate field of view conversion of the first image acquired by the first image acquisition device according to the field of view conversion matrix to obtain a bird's-eye view image corresponding to the first image.
[0011] Optionally, the camera field of view image A1 is the first image; or, the camera field of view image A1 is acquired by a first image acquisition device before the first moment, and the posture of the first image acquisition device when acquiring the camera field of view image A1 is consistent with the posture when acquiring the first image. Among them, the posture includes position and posture, the position refers to the coordinates of the image acquisition device in the world coordinate system, and the posture refers to the orientation of the image acquisition device in the world coordinate system. The world coordinate system can describe the position of the image acquisition device in the real world, and can also describe the position of objects (such as vehicles, pedestrians, road features, etc.) in the image acquired by the image acquisition device in the real world.
[0012] Optionally, the method also includes: dividing the camera field of view image A1 into multiple image segments; obtaining a field of view conversion matrix corresponding to each of the multiple image segments through the field of view conversion model; and determining the field of view conversion matrix of the first image acquisition device based on the field of view conversion matrix corresponding to the multiple image segments.
[0013] The technical solution provided by the present application can reduce the conversion error caused by the visual difference of near large and far small because the camera field of view image A1 is segmented and the field of view conversion matrix of the first image acquisition device is calculated based on the multiple image segments obtained by segmentation. "Near large and far small" means that the proportion of objects farther away from the first image acquisition device in the camera field of view image acquired by the first image acquisition device is smaller than the proportion of objects closer to the first image acquisition device in the camera field of view image.
[0014] Optionally, the field of view conversion model includes a feature extraction unit, a feature conversion unit and a matrix calculation unit, and for a first image segment among the multiple image segments: the feature extraction unit is used to extract image features of the first image segment; the feature conversion unit is used to determine a bird's-eye view feature corresponding to the first image segment based on the image features of the first image segment; and the matrix calculation unit is used to calculate a field of view conversion matrix corresponding to the first image segment based on the image features of the first image segment and the bird's-eye view feature corresponding to the first image segment.
[0015] Optionally, the field of view conversion matrix of the first image acquisition device is determined according to the field of view conversion matrix corresponding to the multiple image segments, including: splicing the field of view conversion matrices corresponding to the multiple image segments according to the positions of the multiple image segments in the camera field of view image A1 to obtain the field of view conversion matrix of the first image acquisition device.
[0016] Optionally, stitching the multiple bird's-eye view images includes: converting the first bird's-eye view image into a pixel coordinate system corresponding to the second bird's-eye view image, the first bird's-eye view image and the second bird's-eye view image being two images among the multiple bird's-eye view images; and fusing the first bird's-eye view image and the second bird's-eye view image in the pixel coordinate system corresponding to the second bird's-eye view image.
[0017] Optionally, converting the first bird's-eye view image to a pixel coordinate system corresponding to the second bird's-eye view image includes: converting the first bird's-eye view image to a pixel coordinate system corresponding to the second bird's-eye view image according to an image conversion matrix from the first bird's-eye view image to the second bird's-eye view image, the image conversion matrix being obtained based on matching pixel pairs in two reference bird's-eye view images, and the matching pixel pairs being obtained based on a graph neural network (GNN) model.
[0018] Optionally, the GNN model includes a feature extraction unit and a matching unit, wherein the feature extraction unit is used to extract image features of the two reference bird's-eye view images, and the image features of any image in the two reference bird's-eye view images include features of each pixel in the any image; and the matching unit is used to determine matching pixel pairs in the two reference bird's-eye view images according to the image features of the two reference bird's-eye view images. For example, the matching unit is used to: determine matching pixel pairs in the two reference bird's-eye view images according to the similarity between the features of the pixel points in the two reference bird's-eye view images, and the similarity between the features of the pixel points in any matching pixel pair in the two reference bird's-eye view images is greater than a similarity threshold.
[0019] Optionally, the two reference bird's-eye view images are respectively the first bird's-eye view image and the second bird's-eye view image; or, the two reference bird's-eye view images respectively correspond to two image acquisition devices among the multiple image acquisition devices, and the two camera field of view images corresponding to the two reference bird's-eye view images are acquired by the two image acquisition devices at a second moment, and the second moment is before the first moment.
[0020] Optionally, adding depth information to the panoramic bird's-eye view image includes: determining at least two original pixel points corresponding to a first pixel point in the panoramic bird's-eye view image in the multiple images; and setting a depth value of the first pixel point according to the depth parallax of the at least two original pixel points. The at least two original pixel points are located in at least two images of the multiple images in a one-to-one correspondence, and after the at least two images are converted according to the perspective conversion matrix and the image conversion matrix, the at least two original pixel points overlap to form the first pixel point, that is, the first pixel point is a pixel point at the overlapping position of the at least two original pixel points.
[0021] The technical solution provided by the present application is that the panoramic bird's-eye view image of the target road scene is a plane view. Adding depth information to the panoramic bird's-eye view image can realize the three-dimensionalization of objects (such as vehicles, pedestrians, road features, etc.) in the panoramic bird's-eye view image. The objects in the real-scene model image obtained after adding depth information to the panoramic bird's-eye view image are three-dimensional objects.
[0022] Optionally, the method further includes: performing traffic business analysis on the target road scene according to the real scene model image. For example, performing object detection and object tracking on the target road scene according to the real scene model image, analyzing the traffic flow of the target road scene, analyzing the congestion situation of the target road scene, etc. After performing traffic business analysis on the target road scene, signal light control optimization strategies, traffic event warnings, etc. can also be provided to the traffic management unit (such as the traffic police brigade) according to the analysis results.
[0023] Optionally, adjacent image acquisition devices among the multiple image acquisition devices have overlapping fields of view, and the fields of view of the multiple image acquisition devices cover the target road scene. The multiple image acquisition devices are at least 4 image acquisition devices.
[0024] In a second aspect, a traffic modeling device is provided, comprising at least one functional module, wherein the at least one functional module interacts with each other to implement the method provided in the first aspect or any optional implementation of the first aspect. The at least one functional module may be implemented based on software, hardware, or a combination of software and hardware, and the at least one functional module may be arbitrarily combined or divided based on a specific implementation, and all or part of the at least one functional module is located in the same computing device.
[0025] In a third aspect, a traffic modeling device is provided, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory so that the traffic modeling device performs the traffic modeling method provided in the first aspect or any optional method of the first aspect.
[0026] In a fourth aspect, a traffic modeling system is provided, which includes the traffic modeling device provided in the second aspect or the third aspect.
[0027] Optionally, the traffic modeling system includes at least one computing device, the traffic modeling apparatus is located in the at least one computing device, and all or part of the functional modules of the traffic modeling apparatus are located in the same computing device.
[0028] Optionally, the at least one computing device includes an edge computing device and a cloud computing device, and a portion of the functional modules of the traffic modeling device are located in the edge computing device, and another portion of the functional modules are located in the cloud computing device.
[0029] Optionally, the traffic modeling system also includes multiple image acquisition devices, which are used to provide camera field of view images of the traffic road scene to the at least one computing device, and the at least one computing device is used to construct a real-life model image of the traffic road scene based on the camera field of view images of the traffic road scene to achieve traffic modeling.
[0030] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed, the method provided in the first aspect or any optional manner of the first aspect is implemented.
[0031] In a sixth aspect, a computer program product is provided, which includes a program or code, and when the program or code is executed, it implements the method provided in the first aspect or any optional manner of the first aspect.
[0032] In a seventh aspect, a chip is provided, which includes a programmable logic circuit and / or program instructions, and when the chip is running, it is used to implement the method provided by the first aspect or any optional manner of the first aspect.
[0033] The technical effects of the second to seventh aspects mentioned above can refer to the technical effects of the first aspect and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 is a schematic diagram of a traffic modeling system provided in an embodiment of the present application;
[0035] Figure 2is a schematic diagram of the shooting areas of adjacent image acquisition devices provided in an embodiment of the present application;
[0036] Figure 3 is a schematic diagram of another traffic modeling system provided in an embodiment of the present application;
[0037] Figure 4 is a schematic diagram of another traffic modeling system provided in an embodiment of the present application;
[0038] Figure 5 is a flow chart of a traffic modeling method provided in an embodiment of the present application;
[0039] Figure 6 is a schematic diagram of converting the field of view of a first image provided by an embodiment of the present application;
[0040] Figure 7 is a schematic diagram of segmenting a camera field of view image provided by an embodiment of the present application;
[0041] Figure 8 is a schematic diagram of a field of view conversion model provided in an embodiment of the present application;
[0042] Fig. 9 is a schematic diagram of a GNN model provided in an embodiment of the present application;
[0043] Fig.10 is a schematic diagram of stitching bird's-eye view images Q1 to Q4 provided in an embodiment of the present application;
[0044] Fig.11 is a schematic diagram of a traffic modeling device provided in an embodiment of the present application;
[0045] Fig.12 It is a schematic diagram of another traffic modeling device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] The implementation methods of the present application will be further described below in conjunction with the accompanying drawings.
[0047] Traffic modeling in the field of smart transportation refers to the construction of a visualization model of traffic road scenes, which is also called a traffic road model. The traffic road model can help users intuitively understand the operation status of traffic roads, and is of great significance to improving the operational efficiency, service level, and safety assurance of urban transportation. Among them, traffic road scenes include intersections (such as crossroads, T-junctions, etc.), road sections (i.e., a section extending on the traffic road), etc. The traffic road model includes multiple virtual objects used to represent multiple physical objects in the traffic road scene. The multiple virtual objects correspond to the multiple physical objects one by one. Each virtual object is distributed in the traffic road model according to the state of the corresponding physical object. Each virtual object can be an object model used to represent the corresponding physical object, or it can be an image of the physical object. Among them, the physical object includes moving objects and background objects. The moving object refers to an object that is moving in the traffic road scene or an object that is stationary but movable for a period of time, such as a motor vehicle, a pedestrian, a non-motor vehicle, an animal, etc. The background object refers to a static object in the traffic road scene, such as a road marking line, a warning sign, a traffic signal pole, a guard booth, surrounding buildings of the road, a roadside tree, a flower bed, etc.
[0048] The current traffic modeling solutions include digital modeling solutions and real-scene modeling solutions. The digital modeling solution is currently the most widely used traffic modeling solution, and the real-scene modeling solution is currently mainly used in the automotive field. In the current digital modeling solution, the visual model of the traffic road scene is constructed by matching the image of the traffic road scene with the high-precision map. In the current real-scene modeling solution, the image of the traffic road scene is combined with auxiliary information such as radar data and high-precision maps to perform modeling. Among them, a high-precision map is a map with an accuracy of up to centimeters. The positioning coordinates of the high-precision map belong to the world coordinate system. The high-precision map is usually used for vehicle positioning, road feature positioning, etc. At present, when making a high-precision map, it is necessary to obtain map data through laser radar or drone aerial photography, and draw a high-precision map based on the map data. The process of making a high-precision map requires a lot of manpower, material resources and financial resources. The production of high-precision maps is difficult and costly, and the production process is greatly affected by human factors. For example, in the current scheme for making high-precision maps, the production cost per kilometer of road section is about 25,000 yuan, and the current business volume is 100+ kilometers. The production cost of each traffic intersection is about 30,000 yuan, and the current business volume is 600+ intersections. With the increase in subsequent business volume, the cost of making high-precision maps is expected to increase exponentially. For another example, the current scheme for making high-precision maps requires manual drawing of lane lines and manual map correction. Affected by human factors, errors are inevitable in high-precision maps, and the quality of high-precision maps varies. These problems with high-precision maps make the current traffic modeling scheme that relies on high-precision maps difficult to implement, costly, and with poor modeling results.
[0049] In addition, the process of updating high-precision maps requires a lot of manual participation, which makes it impossible for high-precision maps to be automatically updated online in a timely manner. When the background objects of the traffic road scene change, the high-precision map cannot be updated in a timely manner based on these changes, which may cause the modeling results obtained based on the high-precision map to not match the actual road scene, and thus cause errors in the upper-level traffic business data. In addition, different manufacturers have different ways of making high-precision maps, and a large number of compatibility problems are prone to occur during the delivery and use of high-precision maps, which makes it difficult for traffic modeling solutions that rely on high-precision maps to be universal. In addition, the current digital modeling solution requires matching the image of the traffic road scene with the high-precision map, and the errors introduced in the matching process will also affect the modeling results. The current real-scene modeling solution needs to be combined with auxiliary information such as radar data, which will also introduce errors and lead to an increase in modeling costs, ultimately resulting in large errors and high costs in the constructed traffic road model. In addition, the current real-scene modeling solution can only model roads within a preset range around the vehicle, and the coverage range is small.
[0050] The embodiment of the present application provides a traffic modeling scheme, which is executed by a computing device, the computing device obtains multiple images obtained by multiple image acquisition devices for image acquisition of a target road scene at a first moment, the computing device converts the field of view of the multiple images to obtain multiple bird's-eye images corresponding to the multiple images, the computing device splices the multiple bird's-eye images to obtain a panoramic bird's-eye image of the target road scene, and the computing device adds depth information to the panoramic bird's-eye image to obtain a real-life model image of the target road scene, thereby achieving modeling of the target road scene. The embodiment of the present application does not need to use high-precision maps and other auxiliary information in the process of traffic modeling, which can avoid the high-precision maps and these other auxiliary information affecting the modeling results. In addition, since the traffic modeling scheme provided by the embodiment of the present application does not rely on high-precision maps, the implementation difficulty of the traffic modeling scheme is small, the implementation cost is low, and it can be adapted to various traffic scenes, especially to large traffic scenes such as highways, ports, airports, and stations. It can provide a large amount of effective information data for these large traffic scenes, and provide a reliable technical foundation for realizing traffic management, human resource scheduling efficiency, and intelligent automation of operation and maintenance construction.
[0051] The technical solution of the embodiment of the present application is introduced below, and the application scenario of the embodiment of the present application is first introduced.
[0052] The application scenario of the embodiment of the present application provides a traffic modeling system, including multiple image acquisition devices and at least one computing device, wherein the multiple image acquisition devices are respectively connected to the at least one computing device. The multiple image acquisition devices are used to acquire images of a target road scene at the same time (e.g., a first time) to obtain multiple images, and the at least one computing device is used to model the target road scene based on the multiple images to obtain a real-scene model image of the target road scene.
[0053] For example, Figure 1 It is a schematic diagram of a traffic modeling system provided in an embodiment of the present application. Figure 1 The traffic modeling system includes n image acquisition devices 101-10n (i.e., image acquisition device 101, image acquisition device 102, image acquisition device 103...image acquisition device 10n) and a computing device 200 as an example, where n is an integer greater than 1, for example, n≥4. The image acquisition devices 101-10n are respectively connected to the computing device 200, and the image acquisition devices 101-10n are used to acquire images of the target road scene at a first moment to obtain n images P1-Pn (i.e., image P1, image P2, image P3...image Pn) of the target road scene, and the computing device 200 is used to model the target road scene based on the images P1-Pn to obtain a real-scene model image of the target road scene. For example, the computing device 200 performs field of view conversion on images P1~Pn to obtain n bird's-eye view images Q1~Qn corresponding one to one to images P1~Pn, the computing device 200 splices the bird's-eye view images Q1~Qn to obtain a panoramic bird's-eye view image of the target road scene, and the computing device 200 adds depth information to the panoramic bird's-eye view image to obtain a real-scene model image of the target road scene.
[0054] Optionally, the image acquisition devices 101-10n are used to perform video acquisition on the target road scene to obtain n video streams, each of the n video streams is acquired by one of the image acquisition devices 101-10n, and the image acquisition devices 101-10n have the same time and frequency for acquiring videos. For example, the synchronous shooting of the image acquisition devices 101-10n is achieved by using camera synchronization technology. Each of the n video streams includes multiple images corresponding to multiple moments, and the n video streams include n images corresponding to each of the multiple moments. The computing device 200 is used to model the target road scene based on the n images corresponding to each moment in the n video streams to obtain a real-scene model image corresponding to each moment, and the real-scene model images corresponding to the multiple moments are arranged in chronological order to obtain a real-scene model video of the target road scene.
[0055] In the embodiment of the present application, the adjacent image acquisition devices among the image acquisition devices 101 to 10n have overlapping fields of view, and the fields of view of the image acquisition devices 101 to 10n cover the target road scene, thereby ensuring that the images acquired by the adjacent image acquisition devices include the same road features (for example, the same lane lines), and the n bird's-eye views corresponding to the n images acquired by the image acquisition devices 101 to 10n at each moment can be spliced to form a panoramic bird's-eye view image of the target road scene. The overlapping fields of view of adjacent image acquisition devices means that the shooting areas of the adjacent image acquisition devices have overlapping areas, for example Figure 2 As shown, the image acquisition device 101 is adjacent to the image acquisition device 102, and the shooting area V1 of the image acquisition device 101 and the shooting area V2 of the image acquisition device 102 have an overlap area V12, and the overlap area V12 is the common viewing area of the image acquisition device 101 and the image acquisition device 102, and the common viewing area is also called the field of view overlap area. Among them, the field of view of the image acquisition devices 101-10n covers the target road scene, which means that the overall shooting area of the image acquisition devices 101-10n covers the complete target road scene, and the images captured by the image acquisition devices 101-10n can cover the complete actual features of the target road scene after splicing.
[0056] Optionally, the installation heights of the image acquisition devices 101 to 10n are substantially the same, and the image acquisition devices 101 to 10n are arranged in a circular arrangement, a fan-shaped arrangement, a straight line arrangement, or other irregular arrangements, and the arrangement of the image acquisition devices 101 to 10n can be designed according to the actual situation of the target road scene. For example, if the target road scene is an intersection, the image acquisition devices 101 to 10n are at least 4 image acquisition devices, and one image acquisition device can be installed at each of the four corners of the intersection (e.g., the southeast corner, the southwest corner, the northeast corner, and the northwest corner), and the shooting direction of each image acquisition device is toward the intersection. For another example, if the target road scene is a straight extension section, at least two image acquisition devices can be installed on both sides of the straight extension section, so that the image acquisition devices on each side of the straight extension section are evenly arranged, and the image acquisition devices on both sides of the straight extension section are symmetrical or staggered, which is not limited in the embodiments of the present application.
[0057] Among them, the image acquisition device is a camera device such as a fisheye camera, a ball camera, a wide-angle camera, a gun camera, etc. The computing device is a variety of possible devices with computing capabilities. Optionally, the traffic modeling system provided in the embodiment of the present application includes multiple computing devices, and the multiple computing devices include edge computing devices and cloud computing devices. The edge computing devices are respectively connected to the cloud computing devices and the above-mentioned multiple image acquisition devices, and the edge computing devices and the cloud computing devices cooperate to perform traffic modeling based on the images collected by the multiple image acquisition devices. The edge computing device can be an edge server, such as an edge computing box, and the edge computing device is deployed at a position close to the image acquisition device, such as deployed in the target road scene and on the side of the road. The cloud computing device can be a cloud computing server, and the cloud computing device is deployed in the cloud. The image acquisition device and the edge computing device and the edge computing device and the cloud computing device can communicate based on various possible security protocols to ensure information security. In some embodiments, the image acquisition device is called an end-side device, the edge computing device is called an edge-side device, and the cloud computing device is called a cloud-side device. The architecture of the traffic modeling system is an "end-edge-cloud" architecture.
[0058] For example, Figure 3 1 is a schematic diagram of another traffic modeling system provided by an embodiment of the present application. The traffic modeling system includes image acquisition devices 101-10n, edge computing devices 201 and cloud computing devices 202, and the edge computing devices 201 are connected to the image acquisition devices 101-10n and the cloud computing devices 202, respectively. The edge computing devices 201 and the cloud computing devices 202 cooperate to model the target road scene based on the images P1-Pn obtained by the image acquisition devices 101-10n at the first moment to obtain the real-life model image of the target road scene. For example, the edge computing device 201 performs field of view conversion on the images P1~Pn to obtain bird's-eye images Q1~Qn, the edge computing device 201 splices the bird's-eye images Q1~Qn to obtain a panoramic bird's-eye image of the target road scene, and the edge computing device 201 adds depth information to the panoramic bird's-eye image to obtain a real-scene model image of the target road scene; the cloud computing device 202 calculates the perspective conversion matrix, image conversion matrix (such as homography matrix), etc. required for traffic modeling, and transmits these matrices to the edge computing device 201, and the edge computing device 201 models the target road scene according to these matrices and the images P1~Pn of the target road scene.
[0059] In an optional embodiment, the traffic modeling system further includes a display device, which is connected to the computing device and can be used to display a real-life model image of the target road scene. The display device can be a visualization device such as a mobile phone, a tablet computer, a car computer, a portable computer, a desktop computer, a monitoring screen, etc. Optionally, the traffic modeling system includes an edge computing device and a cloud computing device, and the display device is connected to at least one of the edge computing device and the cloud computing device, for example Figure 4 As shown, the traffic modeling system also includes a display device 300, which is connected to the edge computing device 201, and is used to display the real scene model image obtained by the edge computing device 201 and the cloud computing device 202. The display device can be deployed in a traffic management unit (such as a traffic police brigade) or in other locations.
[0060] In an optional embodiment, the computing device (such as the edge computing device 201 or the cloud computing device 202) is also used to perform traffic business analysis on the target road scene according to the real-life model image of the target road scene. For example, the computing device performs object detection (or object detection) and object tracking (or object tracking) on the target road scene according to the real-life model image, analyzes the traffic flow of the target road scene, analyzes the congestion of the target road scene, and analyzes whether the target road scene has traffic accidents, traffic congestion, traffic anomalies, etc. The display device is also used to display the results of the traffic business analysis. The computing device can also provide traffic management units with signal light control optimization strategies, traffic event alarms, etc. based on the results of the traffic business analysis, so as to facilitate the traffic management units to implement relevant traffic management measures (such as adjusting the timing of signal lights, dispatching traffic management human resources for traffic command, dispatching traffic management human resources for accident handling, etc.), so as to improve the operating efficiency of road traffic, improve traffic management efficiency, and reduce urban congestion. The computing device can also determine at least one image acquisition device whose shooting area covers the location where the traffic accident occurred in the real-life model image of the target road scene (referred to as the traffic accident location) based on the location where the traffic accident occurred, and control the at least one image acquisition device to capture a video of the target road scene. The computing device can store the video captured by the at least one image acquisition device to provide auxiliary judgment basis for the resolution of the traffic accident.
[0061] The above is an introduction to the application scenarios of the embodiments of the present application. The following is an introduction to the method embodiments of the present application.
[0062] Figure 5 is a flow chart of a traffic modeling method provided in an embodiment of the present application. The traffic modeling method is executed by a computing device. The computing device is Figure 1 The computing device 200 in the embodiment of the present invention is Figure 3 or Figure 4The edge computing device 201 or the cloud computing device 202 in the embodiment, or a part of the steps of the traffic modeling method is executed by the edge computing device 201, and another part of the steps is executed by the cloud computing device 202. Figure 5 As shown, the method includes the following steps S501 to S504.
[0063] S501. Acquire multiple images, where the multiple images are respectively acquired by multiple image acquisition devices performing image acquisition on a target road scene at a first moment.
[0064] Among them, there is an overlap in the field of view of adjacent image acquisition devices among the multiple image acquisition devices, and the field of view of the multiple image acquisition devices covers the target road scene. The multiple images correspond one-to-one to the multiple image acquisition devices, and each of the multiple images is obtained by the corresponding image acquisition device performing image acquisition on the target road scene at the first moment. The image directly taken or collected by the image acquisition device (such as a camera device) can be called a camera field of view image, so the multiple images are all camera field of view images. The camera field of view image can also be called a camera perspective image or a similar name.
[0065] In an example, the multiple images are n images P1~Pn, the multiple image acquisition devices are n image acquisition devices 101~10n, images P1~Pn correspond one-to-one to image acquisition devices 101~10n, images P1~Pn are all camera field of view images, image P1 is obtained by image acquisition device 101 performing image acquisition on the target road scene at the first moment, image P2 is obtained by image acquisition device 102 performing image acquisition on the target road scene at the first moment, and so on.
[0066] S502. Perform field of view conversion on the multiple images to obtain multiple bird's-eye view images corresponding to the multiple images one by one.
[0067] In an optional embodiment, the computing device performs field of view conversion on the first image among the multiple images according to the field of view conversion matrix G1 of the first image acquisition device among the multiple image acquisition devices to obtain a bird's-eye view image corresponding to the first image. The first image is acquired by the first image acquisition device, and the first image acquisition device is any device among the multiple image acquisition devices.
[0068] In one embodiment, the computing device multiplies the first image with the field of view conversion matrix G1 to obtain a bird's-eye view image corresponding to the first image. For example, the field of view conversion matrix G1 includes matrix elements corresponding to each pixel in the first image, and the computing device multiplies the features of each pixel in the first image with the matrix elements corresponding to each pixel. The features of each pixel in the first image include the coordinates of each pixel in the pixel coordinate system corresponding to the first image, the pixel value of each pixel, etc. Among them, the image coordinate system is a coordinate system with the upper left vertex of the camera field of view image (the image acquired by the image acquisition device) as the coordinate origin, and the direction of one of the x-axis and y-axis of the image coordinate system (for example, the x-axis) is the length direction of the camera field of view image, and the direction of the other coordinate axis (for example, the y-axis) is the width direction of the camera field of view image. The pixel coordinate system corresponding to the first image is a coordinate system with the upper left vertex of the first image as the coordinate origin, and the direction of the x-axis of the pixel coordinate system corresponding to the first image can be the length direction of the first image, and the direction of the y-axis can be the width direction of the first image. In one example, a schematic diagram of field of view conversion of the first image is shown as follows. Figure 6 shown.
[0069] Among them, the field of view conversion matrix G1 is obtained by the camera field of view image A1 and the field of view conversion model. The camera field of view image A1 is obtained by the first image acquisition device performing image acquisition on the target road scene. For example, the camera field of view image A1 is the first image; or, the camera field of view image A1 is acquired by the first image acquisition device before the first moment, and the posture when the first image acquisition device acquires the camera field of view image A1 is consistent with the posture when the first image acquisition device acquires the first image. The posture includes position and posture. The position of the image acquisition device refers to the coordinates of the image acquisition device in the world coordinate system, and the posture of the image acquisition device refers to the orientation of the image acquisition device in the world coordinate system. The moment when the first image acquisition device acquires the camera field of view image A1 can be during the day, and when the first image acquisition device acquires the camera field of view image A1, there are fewer vehicles in the target road scene, so that the camera field of view image A1 includes more road features, reducing the impact of vehicles, pedestrians, etc. blocking the road surface. Among them, the field of view conversion model is an artificial intelligence (AI) model, and the field of view conversion model can be a convolutional neural network (CNN) model, a recurrent neural network (RNN) model or a long short-term memory (LSTM) model, etc.
[0070] Optionally, before S502, the computing device obtains the field of view conversion matrix G1 according to the camera field of view image A1 and the field of view conversion model. In one embodiment, the computing device divides the camera field of view image A1 into multiple image segments, and the computing device obtains the field of view conversion matrix corresponding to each of the multiple image segments through the field of view conversion model, and the computing device determines the field of view conversion matrix G1 according to the field of view conversion matrices corresponding to the multiple image segments. For example, the computing device obtains the field of view conversion matrix G1 by splicing the field of view conversion matrices corresponding to the multiple image segments according to their positions in the camera field of view image A1.
[0071] In the embodiment of the present application, the computing device may segment the camera field of view image A1 in any possible manner. For example, the computing device segments the camera field of view image A1 into multiple image segments along the length direction of the camera field of view image A1; or, the computing device segments the camera field of view image A1 into multiple image segments along the width direction of the camera field of view image A1. Alternatively, the computing device segments the camera field of view image A1 along the length direction and the width direction of the camera field of view image A1 respectively. In one example, Figure 7 is a schematic diagram of a segmented camera field of view image A1 provided in an embodiment of the present application, such as Figure 7 As shown, the computing device divides the camera field of view image A1 into 7 image segments A11-A17 along the length direction x of the camera field of view image A1. The image segments A11-A17 are in strip shape. The length directions of the image segments A11-A17 are parallel to the width direction y of the camera field of view image A1. The width directions of the image segments A11-A17 are parallel to the length direction x of the camera field of view image A1. The computing device obtains the field of view conversion matrix corresponding to each image segment in the image segments A11-A17 through the field of view conversion model. For example, the image segments A11-A17 correspond to the field of view conversion matrices G11-G17 one by one. The field of view conversion matrix G11 corresponds to the image segment A11. The field of view conversion matrix G12 corresponds to the image segment A12. By analogy, the field of view conversion matrix G17 corresponds to the image segment A17. The number of rows of the field of view conversion matrices G11 to G17 is equal. The computing device sequentially splices the field of view conversion matrices G11 to G17 according to the positions of the image segments A11 to A17 in the camera field of view image A1 to obtain the field of view conversion matrix G1. The number of rows of the field of view conversion matrices G1 and G11 to G17 is equal, and the number of columns of the field of view conversion matrix G1 is equal to the sum of the number of columns of the field of view conversion matrices G11 to G17.
[0072] In an optional embodiment, the computing device obtains the field of view conversion matrix corresponding to each image segment of the multiple image segments through the field of view conversion model, including: the computing device inputs each image segment into the field of view conversion model, so that the field of view conversion model calculates the field of view conversion matrix corresponding to each image segment according to each image segment, and outputs the field of view conversion matrix corresponding to each image segment. In one embodiment, Figure 8 is a schematic diagram of a field of view conversion model provided by an embodiment of the present application, wherein the field of view conversion model comprises a feature extraction unit, a feature conversion unit and a matrix calculation unit, wherein the feature extraction unit, the feature conversion unit and the matrix calculation unit are connected in sequence, and the feature extraction unit, the feature conversion unit and the matrix calculation unit respectively comprise at least one convolutional layer ( Figure 8 ). The feature extraction unit is used to extract the image features of the camera field of view image, the feature conversion unit is used to convert the image features into bird's-eye features, and the matrix calculation unit is used to calculate the field of view conversion matrix according to the image features and the bird's-eye features. For the first image segment among the multiple image segments: the feature extraction unit is used to extract the image features of the first image segment, and the image features of the first image segment include the features of each pixel in the first image segment; the feature conversion unit is used to determine the bird's-eye features corresponding to the first image segment according to the image features of the first image segment, and the bird's-eye features corresponding to the first image segment include the bird's-eye features corresponding to each pixel in the first image segment; the matrix calculation unit is used to calculate the field of view conversion matrix corresponding to the first image segment according to the image features of the first image segment and the bird's-eye features corresponding to the first image segment, for example, calculate the mapping relationship between the image features of the first image segment and the bird's-eye features corresponding to the first image segment, the mapping relationship is the field of view conversion matrix corresponding to the first image segment, and the mapping relationship includes the mapping relationship between the features of each pixel in the first image segment and the bird's-eye features corresponding to the each pixel. The first image segment is any image segment among the multiple image segments.
[0073] In an embodiment of the present application, a computing device divides the camera field of view image A1 into multiple image segments, and the computing device obtains the field of view conversion matrix corresponding to the multiple image segments, and splices the field of view conversion matrix corresponding to the multiple image segments to obtain the field of view conversion matrix G1. The computing device obtains the field of view conversion matrix G1 in this way, which can reduce the error caused by the visual difference between near and far, and improve the accuracy of the field of view conversion matrix G1. Among them, "near and far" means that the proportion of objects farther away from the first image acquisition device in the camera field of view image (such as camera field of view image A1) collected by the first image acquisition device is smaller than the proportion of objects closer to the first image acquisition device in the camera field of view image. In other embodiments, the computing device may not divide the camera field of view image A1, and the computing device directly obtains the field of view conversion matrix G1 based on the camera field of view image A1 and the field of view conversion model. For example, the computing device inputs the camera field of view image A1 into the field of view conversion model, so that the field of view conversion model calculates the field of view conversion matrix G1 based on the camera field of view image A1, and outputs the field of view conversion matrix G1.
[0074] In an optional embodiment, before the computing device performs field of view conversion on the first image, the computing device preprocesses the first image to improve the picture quality of the first image, and the computing device performs field of view conversion on the preprocessed first image. The preprocessing includes image denoising, color balancing, illumination correction, distortion correction, etc. For example, the computing device preprocesses the first image according to the camera intrinsic parameters of the first image acquisition device. The camera intrinsic parameters are inherent properties of the image acquisition device (such as a camera), and the camera intrinsic parameters generally include distortion coefficients, denoising coefficients, color balancing coefficients, etc.
[0075] The above description uses the example of the computing device performing field of view conversion on the first image. The first image is any one of the above multiple images. The computing device can perform field of view conversion on each of the multiple images based on a field of view conversion process similar to the first image. For example, the multiple images are n images P1 to Pn, the multiple image acquisition devices are n image acquisition devices 101 to 10n, the images P1 to Pn correspond one to one with the image acquisition devices 101 to 10n, the first image is image P1, and the first image acquisition device is image acquisition device 101; based on the field of view conversion process similar to the first image, the computing device performs field of view conversion on image P2 according to the field of view conversion matrix of image acquisition device 102, and the computing device performs field of view conversion on image P3 according to the field of view conversion matrix of image acquisition device 103, and so on. In addition, before the computing device performs field of view conversion on images P1 to Pn, the computing device performs pre-processing such as picture denoising, color balancing, illumination correction, and distortion correction on images P1 to Pn respectively. For example, the computing device preprocesses the image P1 according to the camera intrinsic parameters of the image acquisition device 101, the computing device preprocesses the image P2 according to the camera intrinsic parameters of the image acquisition device 102, and so on.
[0076] S503: stitching the multiple bird's-eye view images to obtain a panoramic bird's-eye view image of the target road scene.
[0077] The panoramic bird's-eye view image is a panoramic image of a target road scene, and the panoramic bird's-eye view image is a real scene image.
[0078] In an optional embodiment, the plurality of bird's-eye view images include a first bird's-eye view image and a second bird's-eye view image, and S503 includes: the computing device converts the first bird's-eye view image into a pixel coordinate system corresponding to the second bird's-eye view image, and the computing device fuses the first bird's-eye view image and the second bird's-eye view image in the pixel coordinate system corresponding to the second bird's-eye view image. Fusing the first bird's-eye view image and the second bird's-eye view image is actually fusing the overlapping area of the first bird's-eye view image and the second bird's-eye view image.
[0079] In one embodiment, the computing device converts the first bird's-eye view image into a pixel coordinate system corresponding to the second bird's-eye view image according to an image conversion matrix H12 from the first bird's-eye view image to the second bird's-eye view image. For example, the computing device multiplies the first bird's-eye view image by the image conversion matrix H12 to convert the first bird's-eye view image into a pixel coordinate system corresponding to the second bird's-eye view image. For example, the image conversion matrix H12 includes matrix elements corresponding to each pixel point in the first bird's-eye view image, and the computing device multiplies the feature of each pixel point in the first bird's-eye view image by the matrix element corresponding to each pixel point.
[0080] In one embodiment, the first bird's-eye view image is a bird's-eye view image corresponding to the first image in the multiple images described in S501-S502, and the second bird's-eye view image is a bird's-eye view image corresponding to the second image in the multiple images. The first image is obtained by a first image acquisition device performing image acquisition on a target road scene at a first moment, and the first image corresponds to the first image acquisition device, so the first bird's-eye view image corresponds to the first image acquisition device. The second image is obtained by a second image acquisition device performing image acquisition on a target road scene at a first moment, and the second image corresponds to the second image acquisition device, so the second bird's-eye view image corresponds to the second image acquisition device. The image conversion matrix H12 is an image conversion matrix from the first image acquisition device to the second image acquisition device, and the image conversion matrix H12 may be a homography matrix, and the image conversion matrix H12 is used to describe the mapping relationship between the pixel coordinate system of the first image acquisition device and the pixel coordinate system of the second image acquisition device. Since the first bird's-eye view image corresponds to the first image acquisition device and the second bird's-eye view image corresponds to the second image acquisition device, the embodiment of the present application describes the image conversion matrix H12 as an image conversion matrix from the first bird's-eye view image to the second bird's-eye view image.
[0081] The image coordinate system is a coordinate system with the upper left vertex of the camera field of view image as the coordinate origin, and the direction of one of the x-axis and the y-axis of the image coordinate system is the length direction of the camera field of view image, and the direction of the other coordinate axis is the width direction of the camera field of view image. For example, the pixel coordinate system corresponding to the first bird's-eye view image is the pixel coordinate system corresponding to the first image (for example, referred to as the first pixel coordinate system), the first pixel coordinate system is a coordinate system with the upper left vertex of the first image as the coordinate origin, the direction of the x-axis of the first pixel coordinate system can be the length direction of the first image, and the direction of the y-axis can be the width direction of the first image. The pixel coordinate system corresponding to the second bird's-eye view image is the pixel coordinate system corresponding to the second image (for example, referred to as the second pixel coordinate system), the second pixel coordinate system is a coordinate system with the upper left vertex of the second image as the coordinate origin, the direction of the x-axis of the second pixel coordinate system can be the length direction of the second image, and the direction of the y-axis can be the width direction of the second image.
[0082] In an embodiment of the present application, the image conversion matrix H12 is obtained based on matching pixel pairs in two reference bird's-eye images, and the matching pixel pairs are obtained based on a GNN model. The two reference bird's-eye images are respectively a first bird's-eye image and a second bird's-eye image. Alternatively, the two reference bird's-eye images correspond to two image acquisition devices among the multiple image acquisition devices described above, respectively, and the two reference bird's-eye images are obtained by converting the field of view of two camera field of view images obtained by the two image acquisition devices performing image acquisition on the target road scene at a second moment, and before the first moment, one of the two image acquisition devices is an image acquisition device corresponding to the first bird's-eye image (i.e., the first image acquisition device), and the other image acquisition device is an image acquisition device corresponding to the second bird's-eye image (i.e., the second image acquisition device). For example, the two reference bird's-eye images are respectively a reference bird's-eye image C1 and a reference bird's-eye image C2. In one example, one of the reference bird's-eye view image C1 and the reference bird's-eye view image C2 is the first bird's-eye view image, and the other image is the second bird's-eye view image, for example, the reference bird's-eye view image C1 is the first bird's-eye view image, and the reference bird's-eye view image C2 is the second bird's-eye view image. In another example, the reference bird's-eye view image C1 is obtained by performing a field of view conversion on the camera field of view image D1, and the reference bird's-eye view image C2 is obtained by performing a field of view conversion on the camera field of view image D2. One of the camera field of view image D1 and the camera field of view image D2 is obtained by performing an image acquisition on a target road scene at a second moment by a first image acquisition device, and the other image is obtained by performing an image acquisition on a target road scene at a second moment by a second image acquisition device, for example, the camera field of view image D1 is acquired by the first image acquisition device, and the camera field of view image D2 is acquired by the second image acquisition device.
[0083] In an optional embodiment, before S503, the computing device obtains the image conversion matrix H12 based on the two reference bird's-eye view images (i.e., the reference bird's-eye view image C1 and the reference bird's-eye view image C2) and the GNN model. For example, the computing device determines the matching pixel pairs in the two reference bird's-eye view images through the GNN model, and the computing device calculates the image conversion matrix H12 based on the matching pixel pairs in the two reference bird's-eye view images. Optionally, the computing device inputs the two reference bird's-eye view images into the GNN model, so that the GNN model calculates the matching pixel pairs in the two reference bird's-eye view images, and outputs the matching pixel pairs in the two reference bird's-eye view images. The two reference bird's-eye view images include a plurality of matching pixel pairs, each matching pixel pair includes two pixels, one of the two pixels is located in the reference bird's-eye view image C1, and the other pixel is located in the reference bird's-eye view image C2. For any pixel point pair among the multiple matching pixel point pairs, the GNN model outputs the corresponding relationship between the coordinates of the two pixel points in the any pixel point pair, and the coordinates of any pixel point among the two pixel points are the coordinates of the any pixel point in the pixel coordinate system corresponding to the reference bird's-eye view image to which it belongs.
[0084] In one embodiment, Fig. 9is a schematic diagram of a GNN model provided in an embodiment of the present application, the GNN model includes a feature extraction unit and a matching unit, and the feature extraction unit is connected to the matching unit. The feature extraction unit is used to extract image features of two images, and the matching unit is used to determine matching pixel pairs in the two images according to the image features of the two images. In an embodiment of the present application, the feature extraction unit is used to extract image features of the above-mentioned two reference bird's-eye images (that is, reference bird's-eye image C1 and reference bird's-eye image C2), and the image features of any image in the two reference bird's-eye images include the features of each pixel in the any image; the matching unit is used to determine matching pixel pairs in the two reference bird's-eye images according to the image features of the two reference bird's-eye images. For example, the matching unit is used to calculate the similarity between the features of the pixels in the two reference bird's-eye images, and determine matching pixel pairs in the two reference bird's-eye images according to the similarity between the features of the pixels in the two reference bird's-eye images, and the similarity between the features of the two pixels in any matching pixel pair in the two reference bird's-eye images is greater than the similarity threshold. The matching unit calculates the similarity between the features of the pixels in the two reference bird's-eye view images as follows: for each pixel in the reference bird's-eye view image C1, the matching unit calculates the similarity between the features of each pixel and the features of each pixel in the reference bird's-eye view image C2. For example, the similarity between the features of two pixels is represented by the distance between the feature description subvectors of the features of the two pixels, and the distance between the feature description subvectors of the features of the two pixels in any matching pixel pair is less than a preset distance. The matching unit calculates the feature description subvector of the features of each pixel in the reference bird's-eye view image C1 and the feature description subvector of the features of each pixel in the reference bird's-eye view image C2. For each pixel in the reference bird's-eye view image C1, the matching unit calculates the distance between the feature description subvector of the features of each pixel and the feature description subvector of the features of each pixel in the reference bird's-eye view image C2. The matching unit determines the pixel pairs in the reference bird's-eye view image C1 and the reference bird's-eye view image C2 whose corresponding distance is less than the preset distance as matching pixel pairs. The embodiment of the present application uses a GNN model to calculate the matching pixel pairs in the two reference bird's-eye view images. Benefiting from the structure of the GNN model itself, the GNN model can extract more effective road key features in the two reference bird's-eye view images during the process of calculating the matching pixel pairs, reduce data noise interference, and make the final output matching pixel pairs more accurate.
[0085] As an implementation of S503, each of the above-mentioned multiple bird's-eye images corresponds to a pixel coordinate system, and the multiple bird's-eye images include a target bird's-eye image and at least one non-target bird's-eye image. The computing device converts the at least one non-target bird's-eye image into the pixel coordinate system corresponding to the target bird's-eye image, and the computing device fuses the multiple bird's-eye images in the pixel coordinate system corresponding to the target bird's-eye image to obtain a panoramic bird's-eye image of the target road scene. For example, the computing device converts each non-target bird's-eye image into the pixel coordinate system corresponding to the target bird's-eye image according to the image conversion matrix from each non-target bird's-eye image in the at least one non-target bird's-eye image to the target bird's-eye image. The target bird's-eye image is any one of the multiple bird's-eye images, or the target bird's-eye image corresponds to the target image in the multiple images described in S501-S502 (the multiple images are all camera field of view images), and the target image is the image with the largest amount of road information contained in the multiple images. Optionally, the target bird's-eye view image is the second bird's-eye view image, and the first bird's-eye view image is any one of the at least one non-target bird's-eye view image.
[0086] In one embodiment, the multiple bird's-eye images are n bird's-eye images Q1-Qn, the bird's-eye image Q2 is a target bird's-eye image, the bird's-eye image Q1 and the bird's-eye images Q3-Qn are all non-target bird's-eye images, the computing device converts the bird's-eye image Q1 to a pixel coordinate system corresponding to the bird's-eye image Q2 according to an image conversion matrix from the bird's-eye image Q1 to the bird's-eye image Q2, the computing device converts the bird's-eye image Q3 to a pixel coordinate system corresponding to the bird's-eye image Q2 according to an image conversion matrix from the bird's-eye image Q3 to the bird's-eye image Q2, the computing device converts the bird's-eye image Q4 to a pixel coordinate system corresponding to the bird's-eye image Q2 according to an image conversion matrix from the bird's-eye image Q4 to the bird's-eye image Q2, and so on. After the computing device converts the bird's-eye image Q1 and the bird's-eye images Q3-Qn to the pixel coordinate system corresponding to the bird's-eye image Q2, the computing device fuses the bird's-eye images Q1-Qn in the pixel coordinate system corresponding to the bird's-eye image Q2 to obtain a panoramic bird's-eye image of the target road scene. In an example, the bird's-eye image Q1~Qn correspond one-to-one to the image acquisition devices 101~10n, the image conversion matrix from the bird's-eye image Q1 to the bird's-eye image Q2 is the homography matrix from the image acquisition device 101 to the image acquisition device 102, the image conversion matrix from the bird's-eye image Q3 to the bird's-eye image Q2 is the homography matrix from the image acquisition device 103 to the image acquisition device 102, the image conversion matrix from the bird's-eye image Q4 to the bird's-eye image Q2 is the homography matrix from the image acquisition device 104 to the image acquisition device 102, and so on.
[0087] In one example, the target road scene is an intersection, and the multiple bird's-eye view images are four bird's-eye view images Q1-Q4. The bird's-eye view images Q1-Q4 correspond to four camera field of view images one by one. The four camera field of view images are obtained by four image acquisition devices installed at four corners of the intersection to acquire images of the target road scene at the first moment. The process of the computing device splicing the bird's-eye view images Q1-Q4 is as follows: Fig.10 When stitching, the computing device converts the bird's-eye view images Q1 to Q4 into the same pixel coordinate system (for example, the pixel coordinate system is the pixel coordinate system corresponding to the bird's-eye view image Q2), and the computing device fuses the overlapping areas of the bird's-eye view images Q1 to Q4 in the pixel coordinate system to obtain a panoramic bird's-eye view image of the target road scene.
[0088] As another implementation of S503, each of the above-mentioned multiple bird's-eye images corresponds to a pixel coordinate system, the first bird's-eye image and the second bird's-eye image are two adjacent bird's-eye images in the multiple bird's-eye images, and the computing device converts the first bird's-eye image to the pixel coordinate system corresponding to the second bird's-eye image, and fuses the first bird's-eye image and the second bird's-eye image in the pixel coordinate system corresponding to the second bird's-eye image to obtain a first fused image. Optionally, the second bird's-eye image is also adjacent to the third bird's-eye image, and after the computing device obtains the first fused image, the computing device converts the first fused image to the pixel coordinate system corresponding to the third bird's-eye image (for example, the computing device converts the first fused image to the pixel coordinate system corresponding to the third bird's-eye image according to the image conversion matrix from the second bird's-eye image to the third bird's-eye image), and the computing device fuses the first fused image and the third bird's-eye image in the pixel coordinate system corresponding to the third bird's-eye image to obtain a second fused image. Optionally, the third bird's-eye view image is also adjacent to the fourth bird's-eye view image. After the computing device obtains the second fused image, the computing device converts the second fused image into a pixel coordinate system corresponding to the fourth bird's-eye view image (for example, the computing device converts the second fused image into a pixel coordinate system corresponding to the fourth bird's-eye view image according to an image conversion matrix from the third bird's-eye view image to the fourth bird's-eye view image). The computing device fuses the second fused image and the fourth bird's-eye view image in the pixel coordinate system corresponding to the fourth bird's-eye view image to obtain a third fused image. This is repeated until the computing device completes the fusion of the multiple bird's-eye view images in the same pixel coordinate system. The fused image obtained after the last fusion is a panoramic bird's-eye view image of the target road scene.
[0089] In one embodiment, the plurality of bird's-eye images are n bird's-eye images Q1-Qn, and the bird's-eye images Q1-Qn are adjacent in sequence and the bird's-eye image Q1 is adjacent to the bird's-eye image Qn. First, the computing device transforms the bird's-eye image Q1 into a pixel coordinate system corresponding to the bird's-eye image Q2 according to the image transformation matrix from the bird's-eye image Q1 to the bird's-eye image Q2, and fuses the bird's-eye image Q1 and the bird's-eye image Q2 in the pixel coordinate system corresponding to the bird's-eye image Q2 to obtain a fused image Q12. Then, the computing device transforms the fused image Q12 into a pixel coordinate system corresponding to the bird's-eye image Q3 according to the image transformation matrix from the bird's-eye image Q2 to the bird's-eye image Q3, and fuses the fused image Q12 and the bird's-eye image Q3 in the pixel coordinate system corresponding to the bird's-eye image Q3 to obtain a fused image Q123. Next, the computing device converts the fused image Q123 to the pixel coordinate system corresponding to the bird's-eye image Q4 according to the image conversion matrix from the bird's-eye image Q3 to the bird's-eye image Q4, and fuses the fused image Q123 and the bird's-eye image Q4 in the pixel coordinate system corresponding to the bird's-eye image Q4 to obtain a fused image Q1234; and so on; finally, the computing device converts the fused image Q123...(n-1) to the pixel coordinate system corresponding to the bird's-eye image Qn according to the image conversion matrix from the bird's-eye image Q(n-1) to the bird's-eye image Qn, and fuses the fused image Q123...(n-1) and the bird's-eye image Qn in the pixel coordinate system corresponding to the bird's-eye image Qn to obtain a panoramic bird's-eye image of the target road scene. In an example, the bird's-eye image Q1~Qn correspond one-to-one to the image acquisition devices 101~10n, the image conversion matrix from the bird's-eye image Q1 to the bird's-eye image Q2 is the homography matrix from the image acquisition device 101 to the image acquisition device 102, the image conversion matrix from the bird's-eye image Q2 to the bird's-eye image Q3 is the homography matrix from the image acquisition device 102 to the image acquisition device 103, and the image conversion matrix from the bird's-eye image Q3 to the bird's-eye image Q4 is the homography matrix from the image acquisition device 103 to the image acquisition device 104, and so on.
[0090] S504. Add depth information to the panoramic bird's-eye view image to obtain a real-scene model image of the target road scene.
[0091] The computing device can add depth information to the panoramic bird's-eye view image to realize stereoscopic objects in the panoramic bird's-eye view image. The objects in the real-scene model image obtained by adding the depth information are stereoscopic objects, such as three-dimensional objects.
[0092] In an optional embodiment, the depth information is a depth value, and S504 includes: a computing device determines at least two original pixels corresponding to a first pixel in the panoramic bird's-eye view image in the multiple images described in S501-S502, the at least two original pixels being located in at least two images of the multiple images in a one-to-one correspondence, the computing device calculating the depth disparity of the at least two original pixels, and the computing device setting the depth value of the first pixel according to the depth disparity of the at least two original pixels to add depth information to the first pixel. The first pixel is located in the splicing area of the panoramic bird's-eye view image, so the first pixel can correspond to the at least two original pixels in the at least two images.
[0093] In one embodiment, the first pixel point is located in the splicing area of the first bird's-eye view image and the second bird's-eye view image. In the above S503, when the computing device splices the first bird's-eye view image and the second bird's-eye view image, the computing device uses the image conversion matrix H12 from the first bird's-eye view image to the second bird's-eye view image to convert the first bird's-eye view image to the pixel coordinate system corresponding to the second bird's-eye view image. In addition, the first bird's-eye view image corresponds to the first image of the multiple images described in S501-S502, and the second bird's-eye view image corresponds to the second image of the multiple images. The first image is obtained by the first image acquisition device performing image acquisition on the target road scene at the first moment, and the second image is obtained by the second image acquisition device performing image acquisition on the target road scene at the first moment. The first bird's-eye view image is obtained by the computing device performing field of view conversion on the first image according to the field of view conversion matrix G1 of the first image acquisition device, and the second bird's-eye view image is obtained by the computing device performing field of view conversion on the second image according to the field of view conversion matrix G2 of the second image acquisition device. The computing device determines the two original pixel points corresponding to the first pixel point in the first image and the second image according to the image conversion matrix H12, the field of view conversion matrix G1, and the field of view conversion matrix G2. Specifically, the computing device determines two pixel points corresponding to the first pixel point in the first bird's-eye view image and the second bird's-eye view image according to the image conversion matrix H12. For the sake of convenience of description, the two pixel points are both referred to as bird's-eye view pixel points. One of the two bird's-eye view pixel points (for example, bird's-eye view pixel point 1) is located in the first bird's-eye view image, and the other pixel point (for example, bird's-eye view pixel point 2) is located in the second bird's-eye view image. The computing device determines the original pixel point 1 corresponding to the bird's-eye view pixel point 1 in the first image according to the field of view conversion matrix G1, and the computing device determines the original pixel point 2 corresponding to the bird's-eye view pixel point 2 in the second image according to the field of view conversion matrix G2. The original pixel point 1 is the original pixel point corresponding to the first pixel point in the first image, and the original pixel point 2 is the original pixel point corresponding to the first pixel point in the second image. In one embodiment, the computing device calculates the transposed matrix H12T of the image conversion matrix H12, the transposed matrix H12T includes the matrix elements corresponding to the first pixel point, the computing device multiplies the features of the first pixel point (e.g., the coordinates of the first pixel point in the panoramic bird's-eye view image) with the matrix elements corresponding to the first pixel point in the transposed matrix H12T, and the computing device determines the bird's-eye view pixel point 1 in the first bird's-eye view image and the bird's-eye view pixel point 2 in the second bird's-eye view image based on the product result. Also, the computing device calculates the transposed matrix G1T of the field of view conversion matrix G1, the transposed matrix G1T includes the matrix elements corresponding to the bird's-eye view pixel point 1, the computing device multiplies the features of the bird's-eye view pixel point 1 (e.g., the coordinates of the bird's-eye view pixel point 1 in the first bird's-eye view image) with the matrix elements corresponding to the bird's-eye view pixel point 1 in the transposed matrix G1T, and the computing device determines the original pixel point 1 corresponding to the bird's-eye view pixel point 1 in the first image based on the product result.Furthermore, the computing device calculates a transposed matrix G2T of the field of view transformation matrix G2, wherein the transposed matrix G2T includes matrix elements corresponding to the bird's-eye view pixel point 2, and the computing device multiplies the characteristics of the bird's-eye view pixel point 2 (for example, the coordinates of the bird's-eye view pixel point 2 in the second bird's-eye view image) by the matrix elements corresponding to the bird's-eye view pixel point 2 in the transposed matrix G2T, and the computing device determines the original pixel point 2 corresponding to the bird's-eye view pixel point 2 in the second image based on the product result.
[0094] In one embodiment, the at least two original pixel points corresponding to the first pixel point are two original pixel points, the computing device calculates the depth disparity of the two original pixel points, and the computing device sets the depth disparity of the two original pixel points as the depth value of the first pixel point. In another embodiment, the at least two original pixel points corresponding to the first pixel point are k original pixel points, k is an integer greater than 2; for each of the k original pixel points, the computing device calculates the depth disparity of each original pixel point and each of the remaining k-1 original pixel points, and the computing device can obtain k(k-1) / 2 depth disparities; the computing device sets the average value of the k(k-1) / 2 depth disparities as the depth value of the first pixel point. The depth disparity between any original pixel point and another original pixel point among the at least two original pixel points is the line of sight distance between the any original pixel point and the other original pixel point. For example, the any original pixel point and the other original pixel point are both located on the first line of sight, and the line of sight distance between the any original pixel point and the other original pixel point is the length of the line of sight segment between the any original pixel point and the other original pixel point on the first line of sight. For any two original pixel points of the at least two original pixel points, the computing device may calculate the depth disparity of the two original pixel points using a triangulation method. In some embodiments, the triangulation method is also referred to as a triangulation disparity method.
[0095] In the above, the calculation device sets a depth value for the first pixel as an example. For each pixel in the stitching area of the panoramic bird's-eye view image, the calculation device sets a depth value for each pixel using a method similar to the first pixel. For each pixel in the non-stitching area of the panoramic bird's-eye view image, the calculation device sets a preset depth value for each pixel. After the calculation device sets the depth value for the pixel in the panoramic bird's-eye view image, a real-life model image of the target road scene can be obtained.
[0096] In summary, in the traffic modeling method provided in the embodiment of the present application, after the computing device acquires multiple images obtained by multiple image acquisition devices performing image acquisition on the target road scene at the first moment, the computing device performs field of view conversion on the multiple images to obtain multiple bird's-eye images corresponding to the multiple images one by one, the computing device splices the multiple bird's-eye images to obtain a panoramic bird's-eye image of the target road scene, and the computing device adds depth information to the panoramic bird's-eye image to obtain a real-life model image of the target road scene, thereby achieving modeling of the target road scene. It can be seen that the embodiment of the present application does not need to use high-precision maps and other auxiliary information in the process of traffic modeling, and can avoid the errors introduced by high-precision maps and these other auxiliary information affecting the modeling results. Since the traffic modeling scheme provided in the embodiment of the present application does not rely on high-precision maps, the implementation difficulty and cost of the traffic modeling scheme are low, and it can be applied to various traffic scenarios.
[0097] In addition, the coordinate system of the traffic road model constructed by the traffic modeling solution provided in the embodiment of the present application is a pixel coordinate system. The modeling process only involves the pixel coordinate system, and there is no need to convert data between the world coordinate system and the pixel coordinate system, which can avoid the errors caused by data conversion between the world coordinate system and the pixel coordinate system. Moreover, the traffic road model constructed by the present application is a real-scene model, which is more intuitive than a digital model. The present application can directly monitor and record traffic conditions in real time.
[0098] In an optional embodiment, after the computing device obtains the real-life model image of the target road scene, the computing device performs traffic business analysis on the target road scene according to the real-life model image. For example, the computing device performs object detection on the target road scene according to the real-life model image, analyzes the traffic flow of the target road scene, analyzes the congestion of the target road scene, and analyzes whether traffic accidents, traffic congestion, traffic anomalies, and other traffic events occur in the target road scene. The real-life model image described in S504 is constructed by the computing device based on multiple images collected by multiple image acquisition devices at the first moment, and the real-life model image is the real-life model image corresponding to the first moment. The computing device can use methods similar to those of S501-S504 above to construct the real-life model image corresponding to each of the multiple moments. The computing device can obtain the real-life model video of the target road scene according to the real-life model images corresponding to the multiple moments. The computing device can also perform object detection, object tracking, traffic flow analysis, traffic event analysis, etc. according to the real-life model video.
[0099] In an optional embodiment, the computing device also provides the traffic management unit with a signal light control optimization strategy, a traffic incident warning, etc. according to the traffic business analysis result of the target road scene, so that the traffic management unit can implement relevant traffic management measures. For example, the traffic management unit can adjust the timing of the signal light according to the signal light control optimization strategy provided by the computing device, and timely dispatch traffic management human resources to perform traffic command, accident handling, etc. according to the traffic incident warning provided by the computing device. When the traffic business analysis result includes a traffic accident in the target road scene, the computing device can also determine at least one image acquisition device whose shooting area covers the traffic accident location among the above-mentioned multiple image acquisition devices according to the real-life model image of the target road scene, and control the at least one image acquisition device to acquire a video of the target road scene. The computing device can store the video acquired by the at least one image acquisition device to provide auxiliary judgment basis for solving the traffic accident. Since the computing device constructs the real-life model image of the target road scene according to the image acquired by the multiple image acquisition devices through image acquisition of the target road scene, and controls at least one image acquisition device among the multiple image acquisition devices to acquire a video of the target road scene according to the analysis result obtained by performing traffic business analysis on the real-life model image, a self-closed loop of the data link is realized.
[0100] In an optional embodiment, the computing device is connected to a display device, and the computing device also transmits a real-life model image of the target road scene to the display device, and / or transmits an analysis result obtained by performing traffic business analysis based on the real-life model image to the display device, and the display device can display the real-life model image and / or the analysis result, and realize visualization of the real-life model image and / or the analysis result, so that the staff can directly understand the operation status of the target road scene. In some embodiments, the computing device packages and stores the real-life model image and the analysis result to realize richer extended functions.
[0101] In an optional embodiment, after the above-mentioned multiple image acquisition devices are deployed in the target road scene, the multiple image acquisition devices perform video acquisition on the target road scene to obtain multiple video streams corresponding to the multiple image acquisition devices one by one, and the multiple image acquisition devices construct the real-life model image of the target road scene in real time based on the multiple video streams. Specifically, each video stream in the multiple video streams includes multiple images corresponding to multiple moments, and the multiple video streams include multiple images corresponding to each of the multiple moments. The computing device constructs the real-life model image corresponding to each moment in real time based on the multiple images corresponding to each of the multiple moments. Take the example that the multiple image acquisition devices are n image acquisition devices 101~10n, and each moment corresponds to n images, and the n images correspond to the image acquisition devices 101~10n one by one. Since the field of view conversion matrix of the image acquisition device and the image conversion matrix between different image acquisition devices are related to the posture of the image acquisition device but have nothing to do with the content of the image acquired by the image acquisition device, therefore, in the process of the computing device constructing the real-scene model image according to the n images corresponding to the earliest moment among the multiple moments, the computing device calculates the field of view conversion matrix of the image acquisition devices 101~10n according to the n images and the field of view conversion model, and the computing device calculates the image conversion matrix between different image acquisition devices in the image acquisition devices 101~10n according to the n bird's-eye view images that correspond one to one to the n images. After the earliest moment, if the position and posture of the image acquisition devices 101-10n have not changed (for example, for any moment after the earliest moment, if the position and posture of the image acquisition devices 101-10n when acquiring images at any moment are consistent with the position and posture of the image acquisition devices 101-10n when acquiring images at the earliest moment, then the position and posture of the image acquisition devices 101-10n have not changed), the computing device does not need to calculate the field of view conversion matrix of the image acquisition devices 101-10n and the image conversion matrix between the image acquisition devices 101-10n, but directly uses the field of view conversion matrix and the image conversion matrix calculated at the earliest moment to construct the real scene model image. After the earliest moment, if the position and posture of at least one of the image acquisition devices 101-10n changes, the computing device recalculates the field of view conversion matrix of the at least one image acquisition device, recalculates the image conversion matrix between the at least one image acquisition device and the image conversion matrix between each image acquisition device in the at least one image acquisition device and other image acquisition devices, and the computing device uses the recalculated field of view conversion matrix and image conversion matrix to construct the real scene model image. In this way, on the one hand, the amount of calculation for the computing device to construct the real-scene model image can be reduced, and on the other hand, the field of view conversion matrix and image conversion matrix used by the computing device to construct the real-scene model image each time can be matched with the posture of the image acquisition device to ensure the accuracy of the constructed real-scene model image.
[0102] In an optional embodiment, the computing device includes an edge computing device and a cloud computing device, and the cloud computing device calculates the field of view conversion matrix of the multiple image acquisition devices and the image conversion matrix between different image acquisition devices in the multiple image acquisition devices, and transmits the field of view conversion matrix of the multiple image acquisition devices and the image conversion matrix between different image acquisition devices in the multiple image acquisition devices to the edge computing device. In each process of constructing a real-scene model image, the edge computing device performs field of view conversion on the image acquired by each image acquisition device according to the field of view conversion matrix of each image acquisition device, and the edge computing device converts the bird's-eye view images corresponding to the different image acquisition devices into the same pixel coordinate system according to the image conversion matrix between different image acquisition devices, and fuses the multiple bird's-eye view images in the same pixel coordinate system, and adds depth information to the panoramic bird's-eye view image obtained by the fusion. As a result, the edge computing device does not need to transmit images to the cloud computing device in each modeling process, and the cloud computing device does not need to transmit images to the edge computing device, which can reduce the amount of data transmitted between the edge computing device and the cloud computing device, saving transmission resources.
[0103] In an optional embodiment, the computing device also constructs a model database, and stores real-life model images of the target road scene in the model database. In practical applications, the computing device can use a modeling method similar to that of the target road scene to construct real-life model images of multiple road scenes, and store the real-life model images of the multiple road scenes in the model database. In one embodiment, the computing device can also store the field of view conversion matrix of each image acquisition device in each of the multiple road scenes and the image conversion matrix between different image acquisition devices in each road scene in the model database, etc. The computing device updates the relevant matrix in the model database after the posture of the image acquisition device changes. The process of updating the relevant matrix by the computing device does not require human intervention, and the relevant matrix can be updated in a timely manner.
[0104] In the embodiment of the present application, the above-mentioned field of view conversion model is obtained by training based on training data. The field of view conversion model can be trained by the above-mentioned computing device, or it can be trained by other devices and transplanted to the computing device. The embodiment of the present application takes the training of the field of view conversion model by the computing device as an example. For example, before S502, the computing device trains the field of view conversion model.
[0105] In one embodiment, the training data for training the field of view conversion model includes the correspondence between a preset camera field of view image and a preset field of view conversion matrix. The computing device obtains an initial field of view conversion model, and the computing device inputs the preset camera field of view image into the initial field of view conversion model, so that the initial field of view conversion model calculates the field of view conversion matrix according to the preset camera field of view image and outputs the calculated field of view conversion matrix. The computing device adjusts the model parameters of the initial field of view conversion model according to the matching degree between the field of view conversion matrix output by the initial field of view conversion model and the preset field of view conversion matrix, and obtains the field of view conversion model after adjusting the parameters. The computing device inputs the preset camera field of view image into the field of view conversion model after adjusting the parameters, so that the field of view conversion model after adjusting the parameters calculates the field of view conversion matrix according to the preset camera field of view image and outputs the calculated field of view conversion matrix. The computing device continues to adjust the model parameters of the field of view conversion model after adjusting the parameters according to the matching degree between the field of view conversion matrix output by the field of view conversion model after adjusting the parameters and the preset field of view conversion matrix. This is repeated until the matching degree between the field of view conversion matrix output by the field of view conversion model after adjusting the parameters and the preset field of view conversion matrix meets the preset condition, and the computing device obtains the above-mentioned field of view conversion model. The preset condition includes that the matching degree between the field of view conversion matrix output by the field of view conversion model after adjusting the parameters and the preset field of view conversion matrix is greater than the preset matching degree.
[0106] The embodiment of the present application is explained by taking the example of a computing device training a field of view conversion model according to a set of correspondences between a preset camera field of view image and a preset field of view conversion matrix. In practical applications, the training data used to train the field of view conversion model includes multiple sets of correspondences between preset camera field of view images and preset field of view conversion matrices, and each set of correspondences includes a preset camera field of view image and a preset field of view conversion matrix corresponding to the preset camera field of view image. The computing device trains the field of view conversion model according to the multiple sets of correspondences, thereby ensuring the reliability of the trained field of view conversion model.
[0107] In the embodiment of the present application, the above-mentioned GNN model is trained based on the training data. The GNN model can be trained by the above-mentioned computing device, or it can be trained by other devices and transplanted to the computing device. The embodiment of the present application takes the training of the GNN model by the computing device as an example. For example, before S503, the computing device trains the GNN model.
[0108] In one embodiment, the training data for training the GNN model includes at least one set of correspondences, each set of correspondences in the at least one set of correspondences records two bird's-eye view images and at least one set of matching pixel pairs corresponding to the two bird's-eye view images, and each set of matching pixel pairs in the at least one set of matching pixel pairs includes two pixels, and the two pixels are respectively located in the two bird's-eye view images. Take the example of a computing device training the GNN model according to one set of correspondences in the at least one set of correspondences (for example, referred to as the first correspondence). The computing device obtains an initial GNN model, and the computing device inputs the two bird's-eye images recorded in the first corresponding relationship into the initial GNN model, so that the initial GNN model calculates at least one group of matching pixel pairs based on the two bird's-eye images and outputs the at least one group of matching pixel pairs, and the computing device adjusts the model parameters of the initial GNN model according to the degree of matching between the pixel pairs output by the initial GNN model and the pixel pairs recorded in the first corresponding relationship to obtain the GNN model after parameter adjustment; the computing device inputs the two bird's-eye images in the first corresponding relationship into the GNN model after parameter adjustment, so that the GNN model after parameter adjustment calculates at least one group of matching pixel pairs based on the two bird's-eye images and outputs the at least one group of matching pixel pairs, and the computing device continues to adjust the model parameters of the GNN model after parameter adjustment according to the degree of matching between the pixel pairs output by the GNN model after parameter adjustment and the pixel pairs recorded in the first corresponding relationship. This is repeated until the degree of matching between the pixel pairs output by the GNN model after adjusting the parameters and the pixel pairs recorded in the first correspondence relationship meets the preset conditions, and the computing device obtains the above-mentioned GNN model. The preset conditions include that the degree of matching between the pixel pairs output by the GNN model after adjusting the parameters and the pixel pairs recorded in the first correspondence relationship is greater than the preset matching degree. The pixel pairs output by the GNN model specifically refers to the information of the pixel pairs output by the GNN model, such as the correspondence between the coordinates of the two pixels in the pixel pair, and the degree of matching between the pixel pairs output by the GNN model and the pixel pairs recorded in the first correspondence relationship refers to the degree of matching between the coordinates of the pixels in the pixel pairs output by the GNN model and the coordinates of the pixels in the pixel pairs recorded in the first correspondence relationship.
[0109] The embodiment of the present application takes the example of a computing device training a GNN model according to a set of corresponding relationships. In practical applications, the computing device trains the GNN model according to the multiple sets of corresponding relationships, thereby ensuring the reliability of the trained GNN model.
[0110] The above embodiment introduces the technical solution of the present application by taking the modeling of road scenes as an example. The technical solution of the present application can also be applied to large-scale transportation scenes such as ports, airports, and stations. It can provide a large amount of effective information data for these large-scale transportation scenes, and provide a reliable technical foundation for realizing traffic management, human resource scheduling efficiency, and intelligent automation of operation and maintenance construction.
[0111] The above is an introduction to the method embodiment of the present application. The following is an introduction to the device embodiment of the present application. The device of the present application can be used to execute the method of the present application. For details not disclosed in the device embodiment of the present application, please refer to the method embodiment.
[0112] Fig.11 1 is a schematic diagram of a traffic modeling device 1100 provided in an embodiment of the present application. The traffic modeling device 1100 is applied to a computing device. For example, the traffic modeling device 1100 is the computing device or a functional component in the computing device. The traffic modeling device 1100 can be used to execute the above-mentioned traffic modeling method embodiment. Fig.11 As shown, the traffic modeling device 1100 includes an image acquisition module 1110 , a conversion module 1120 , a splicing module 1130 and an adding module 1140 .
[0113] The image acquisition module 1110 is used to acquire multiple images, where the multiple images are respectively acquired by multiple image acquisition devices performing image acquisition on a target road scene at a first moment.
[0114] The conversion module 1120 is used to perform field of view conversion on the multiple images to obtain multiple bird's-eye view images corresponding to the multiple images.
[0115] The stitching module 1130 is used to stitch the multiple bird's-eye view images to obtain a panoramic bird's-eye view image of the target road scene.
[0116] The adding module 1140 is used to add depth information to the panoramic bird's-eye view image to obtain a real-scene model image of the target road scene.
[0117] Optionally, the conversion module 1120 is used to perform field of view conversion on a first image among the multiple images according to a field of view conversion matrix of a first image acquisition device among the multiple image acquisition devices, where the first image is acquired by the first image acquisition device.
[0118] Optionally, the field of view conversion matrix is obtained by a camera field of view image and a field of view conversion model, the camera field of view image is acquired by a first image acquisition device, and the field of view conversion model is trained based on training data, and the training data includes a correspondence between a preset camera field of view image and a preset field of view conversion matrix.
[0119] Optionally, the camera field of view image is the first image; or, the camera field of view image is acquired by a first image acquisition device before a first moment, and the posture of the first image acquisition device when acquiring the camera field of view image is consistent with the posture when acquiring the first image.
[0120] Optional, such as Fig.11 As shown, the traffic modeling device 1100 also includes a matrix acquisition module 1150, which is used to: divide the camera field of view image into multiple image segments; obtain the field of view conversion matrix corresponding to each image segment in the multiple image segments through the field of view conversion model; and determine the field of view conversion matrix of the first image acquisition device according to the field of view conversion matrix corresponding to the multiple image segments.
[0121] Optionally, the stitching module 1130 is used to: convert the first bird's-eye view image into a pixel coordinate system corresponding to the second bird's-eye view image, the first bird's-eye view image and the second bird's-eye view image being two images among the multiple bird's-eye view images; and fuse the first bird's-eye view image and the second bird's-eye view image in the pixel coordinate system corresponding to the second bird's-eye view image.
[0122] Optionally, the stitching module 1130 is used to convert the first bird's-eye view image into a pixel coordinate system corresponding to the second bird's-eye view image according to an image transformation matrix from the first bird's-eye view image to the second bird's-eye view image, wherein the image transformation matrix is obtained based on matching pixel pairs in two reference bird's-eye view images, and the matching pixel pairs are obtained based on a graph neural network (GNN) model.
[0123] Optionally, the two reference bird's-eye view images are respectively the first bird's-eye view image and the second bird's-eye view image; or, the two reference bird's-eye view images respectively correspond to two image acquisition devices among the multiple image acquisition devices, and the two camera field of view images corresponding to the two reference bird's-eye view images are acquired by the two image acquisition devices at a second moment, and the second moment is before the first moment.
[0124] Optionally, a module 1140 is added, which is used to: determine at least two original pixel points corresponding to a first pixel point in the panoramic bird's-eye view image in the multiple images; and set a depth value of the first pixel point according to the depth disparity of the at least two original pixel points.
[0125] Optional, such as Fig.11 As shown, the traffic modeling device 1100 further includes an analysis module 1160 for performing traffic business analysis on the target road scene according to the real scene model image.
[0126] Optionally, adjacent image acquisition devices among the multiple image acquisition devices have overlapping fields of view, and the fields of view of the multiple image acquisition devices cover the target road scene.
[0127] The functional implementation of the image acquisition module 1110 may refer to the relevant description in S501 above. The functional implementation of the conversion module 1120 and the matrix acquisition module 1150 may refer to the relevant description in S502 above. The functional implementation of the splicing module 1130 may refer to the relevant description in S503 above. The functional implementation of the adding module 1140 may refer to the relevant description in S504 above.
[0128] The functional implementations of the image acquisition module 1110, the conversion module 1120, the stitching module 1130, the adding module 1140, the matrix acquisition module 1150 and the analysis module 1160 have been described in detail in the above method embodiments and will not be repeated here.
[0129] In summary, the traffic modeling device provided by the embodiment of the present application includes an image acquisition module, a conversion module, a splicing module and an adding module. After the image acquisition module acquires multiple images obtained by multiple image acquisition devices performing image acquisition on the target road scene at the first moment, the conversion module performs field of view conversion on the multiple images to obtain multiple bird's-eye images corresponding to the multiple images one by one. The splicing module splices the multiple bird's-eye images to obtain a panoramic bird's-eye image of the target road scene. The adding module adds depth information to the panoramic bird's-eye image to obtain a real-life model image of the target road scene, thereby achieving modeling of the target road scene. It can be seen that the embodiment of the present application does not need to use high-precision maps and other auxiliary information in the process of traffic modeling. Therefore, it can avoid the errors introduced by high-precision maps and these other auxiliary information affecting the modeling results.
[0130] The embodiment of the present application provides a traffic modeling device, which includes a memory and a processor. The memory is used to store a computer program. The processor is used to execute the computer program stored in the memory so that the traffic modeling device performs all or part of the steps of the traffic modeling method provided in the above embodiment.
[0131] As an example, Fig.12 1 is a schematic diagram of another traffic modeling device 1200 provided in an embodiment of the present application. The traffic modeling device 1200 is applied to a computing device. For example, the traffic modeling device 1200 is the computing device or a functional component in the computing device. The traffic modeling device 1200 can be used to execute the above-mentioned traffic modeling method embodiment. Fig.12 As shown, the traffic modeling device 1200 includes a processor 1201 , a memory 1202 , a bus 1203 , a communication interface 1204 , and an input / output device 1205 . The processor 1201 , the memory 1202 , the communication interface 1204 , and the input / output device 1205 are connected via the bus 1203 . Fig.12The processor 1201 and the memory 1202 are described as being independent of each other. The processor 1201 and the memory 1202 may also be integrated together.
[0132] The memory 1202 is used to store computer programs, including operating systems and program codes. The memory 1202 is a storage medium of various types, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), flash memory, register, optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), disk or other magnetic storage device.
[0133] The processor 1201 is a general-purpose processor or a special-purpose processor. A general-purpose processor is a processor that performs specific steps and / or operations by reading and executing a computer program stored in a memory. The general-purpose processor may use the computer program stored in the memory in the process of performing the above steps and / or operations. The computer program is executed to implement Fig.12The related functions of each module in the illustrated embodiment. A general-purpose processor is, for example, but not limited to, a central processing unit (CPU). A dedicated processor is a processor specially designed to perform specific steps and / or operations, and a dedicated processor is, for example, but not limited to, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. Processor 1201 may be a single-CPU processor or a multi-CPU processor. Processor 1201 includes at least one circuit to execute all or part of the steps of the traffic modeling method provided in the above-mentioned embodiment.
[0134] The communication interface 1204 is used for the traffic modeling device 1200 to communicate with other devices. The communication interface 1204 includes a physical interface and a logical interface. The physical interface is used to realize the interconnection between the traffic modeling device 1200 and other devices, and the logical interface is an interface inside the traffic modeling device 1200, and the logical interface is used to realize the interconnection of devices inside the traffic modeling device 1200.
[0135] The input / output device 1205 includes an input / output (I / O) interface, a device such as a display connected to the traffic modeling device 1200 through the I / O interface, and a device such as a display connected to the processor 1201 through a bus. The processor 1201 can receive input commands or data through the input / output device 1205 and output processed data. For example, the input / output device 1205 includes a display, and the display can be used to display the intermediate results and / or final results generated by the processor 1201 executing the above-mentioned traffic modeling method, such as displaying the above-mentioned camera field of view image, the bird's-eye view image corresponding to the camera field of view image, the panoramic bird's-eye view image, the traffic business analysis result, etc.
[0136] The bus 1203 is any type of communication bus for interconnecting the internal components of the traffic modeling device 1200, such as a system bus. The embodiment of the present application takes the above components in the traffic modeling device 1200 being interconnected through the bus 1203 as an example, and the above components in the traffic modeling device 1200 are connected to each other using other connection methods, such as the above components in the traffic modeling device 1200 being interconnected through a logical interface in the traffic modeling device 1200.
[0137] The above devices may be arranged on independent chips, or at least partially or completely on the same chip. Whether to arrange each device independently on different chips or to integrate them on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above devices.
[0138] Fig.12 The traffic modeling device 1200 shown is merely exemplary. During implementation, the traffic modeling device 1200 may include other components, which are not listed here one by one. Fig.12 The traffic modeling device 1200 shown can model the target road scene by executing all or part of the steps of the traffic modeling method provided by the above embodiment.
[0139] The present application also provides a traffic modeling system, comprising: Fig.11 or Fig.12 The traffic modelling device shown.
[0140] Optionally, the traffic modeling system includes at least one computing device, the traffic modeling apparatus is located in the at least one computing device, and all or part of the functional modules of the traffic modeling apparatus are located in the same computing device.
[0141] Optionally, the at least one computing device includes an edge computing device and a cloud computing device, and all or part of the functional modules of the traffic modeling device are located in the edge computing device or the cloud computing device. For example, part of the functional modules of the traffic modeling device are located in the edge computing device, and another part of the functional modules are located in the cloud computing device.
[0142] Optionally, the traffic modeling system includes multiple image acquisition devices, which are used to acquire multiple images of the target road scene; the traffic modeling device is used to construct a real-scene model image of the target road scene based on the multiple images.
[0143] For example, the traffic modeling system is as follows Figure 1 , Figure 3 or Figure 4 Any one shown.
[0144] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed (for example, executed by a traffic modeling apparatus, a computing device, one or more processors, etc.), all or part of the steps of the traffic modeling method provided in the above method embodiment are implemented.
[0145] An embodiment of the present application provides a computer program product, which includes a program or code. When the program or code is executed (for example, executed by a traffic modeling device, a computing device, one or more processors, etc.), it implements all or part of the steps of the traffic modeling method provided in the above method embodiment.
[0146] An embodiment of the present application provides a chip, which includes a programmable logic circuit and / or program instructions. When the chip is running, it is used to implement all or part of the steps of the traffic modeling method provided in the above method embodiment.
[0147] The embodiments of the present application are described by taking the example that the computing device and the image acquisition device are different devices, that is, taking the example that the device for acquiring images and the device for performing traffic modeling based on the images are different devices. In some embodiments, the image acquisition device can integrate computing functions to assume the role of the computing device for traffic modeling, so that traffic modeling can be achieved without the need for additional computing devices. For example, as the capabilities of the image acquisition device increase, computing functions can be integrated into the image acquisition device so that the image acquisition device can use images and perform traffic modeling based on the images, and the embodiments of the present application are not limited to this.
[0148] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product, and the computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium, or a semiconductor medium (e.g., a solid-state hard disk), etc.
[0149] It should be understood that the term "at least one" in this application refers to one or more, and "multiple" refers to two or more. The term "and / or" in this application is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate clear description, in this application, the words "first", "second", "third" and the like are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second", "third" and the like do not limit the quantity and execution order.
[0150] The different types of embodiments such as the method embodiments and device embodiments provided in the embodiments of the present application can refer to each other, and the embodiments of the present application are not limited thereto. The sequence of operations of the method embodiments provided in the embodiments of the present application can be appropriately adjusted, and the operations can also be increased or decreased in response to the situation. Any technician familiar with the technical field can easily think of the changed methods within the technical scope disclosed in the present application, and they should be included in the protection scope of the present application, so they will not be repeated.
[0151] In the corresponding embodiments provided in the present application, it should be understood that the disclosed devices and the like can be implemented by other configuration methods. For example, the device embodiments described above are only illustrative, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0152] The modules described as separate components may or may not be physically separated, and the components described as modules may or may not be physical modules, and may be located in one place or distributed on multiple devices. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0153] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A traffic modeling method, characterized in that: The method comprises: Acquire a plurality of images, wherein the plurality of images are respectively acquired by a plurality of image acquisition devices performing image acquisition on a target road scene at a first moment; Performing field of view conversion on the multiple images to obtain multiple bird's-eye view images corresponding to the multiple images one by one; splicing the multiple bird's-eye view images to obtain a panoramic bird's-eye view image of the target road scene; Depth information is added to the panoramic bird's-eye view image to obtain a real-scene model image of the target road scene.
2. The method according to claim 1, characterized in that The converting the field of view of the multiple images includes: The field of view conversion is performed on a first image among the multiple images according to a field of view conversion matrix of a first image acquisition device among the multiple image acquisition devices; wherein the first image is acquired by the first image acquisition device.
3. The method according to claim 2, characterized in that The field of view conversion matrix is obtained by a camera field of view image and a field of view conversion model. The camera field of view image is acquired by the first image acquisition device. The field of view conversion model is trained based on training data. The training data includes a correspondence between a preset camera field of view image and a preset field of view conversion matrix.
4. The method according to claim 3, characterized in that: The camera field of view image is the first image; or, the camera field of view image is acquired by the first image acquisition device before the first moment, and the posture of the first image acquisition device when acquiring the camera field of view image is consistent with the posture when acquiring the first image.
5. The method according to claim 3 or 4, characterized in that: The method further comprises: Segmenting the camera field of view image into a plurality of image segments; Acquire a field of view conversion matrix corresponding to each image segment in the plurality of image segments through the field of view conversion model; The field of view conversion matrix of the first image acquisition device is determined according to the field of view conversion matrices corresponding to the multiple image segments.
6. The method according to any one of claims 1 to 5, characterized in that: The stitching of the multiple bird's-eye view images comprises: Converting the first bird's-eye view image into a pixel coordinate system corresponding to a second bird's-eye view image, wherein the first bird's-eye view image and the second bird's-eye view image are two images of the plurality of bird's-eye view images; The first bird's-eye view image and the second bird's-eye view image are fused in a pixel coordinate system corresponding to the second bird's-eye view image.
7. The method according to claim 6, characterized in that The converting the first bird's-eye view image into a pixel coordinate system corresponding to the second bird's-eye view image includes: According to the image conversion matrix from the first bird's-eye view image to the second bird's-eye view image, the first bird's-eye view image is converted into a pixel coordinate system corresponding to the second bird's-eye view image, the image conversion matrix is obtained based on matching pixel pairs in two reference bird's-eye view images, and the matching pixel pairs are obtained based on a graph neural network GNN model.
8. The method according to claim 7, characterized in that The two reference bird's-eye view images are respectively the first bird's-eye view image and the second bird's-eye view image; or, the two reference bird's-eye view images respectively correspond to two image acquisition devices among the multiple image acquisition devices, and the two camera field of view images corresponding to the two reference bird's-eye view images are acquired by the two image acquisition devices at a second moment, and the second moment is before the first moment.
9. The method according to any one of claims 1 to 8, characterized in that: The adding depth information to the panoramic bird's-eye view image comprises: Determine at least two original pixel points corresponding to a first pixel point in the panoramic bird's-eye view image in the multiple images; The depth value of the first pixel is set according to the depth disparity of the at least two original pixel points.
10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: A traffic business analysis is performed on the target road scene according to the real scene model image.
11. The method according to any one of claims 1 to 10, characterized in that: Adjacent image acquisition devices among the multiple image acquisition devices have overlapping fields of view, and the fields of view of the multiple image acquisition devices cover the target road scene.
12. A traffic modeling device, characterized in that: The device comprises: An image acquisition module, used to acquire multiple images, wherein the multiple images are respectively acquired by multiple image acquisition devices performing image acquisition on a target road scene at a first moment; A conversion module, used for performing field of view conversion on the multiple images to obtain multiple bird's-eye view images corresponding to the multiple images one by one; A stitching module, used for stitching the multiple bird's-eye view images to obtain a panoramic bird's-eye view image of the target road scene; The adding module is used to add depth information to the panoramic bird's-eye view image to obtain a real-scene model image of the target road scene.
13. The device according to claim 12, characterized in that The conversion module is used to perform field of view conversion on a first image among the multiple images according to a field of view conversion matrix of a first image acquisition device among the multiple image acquisition devices; wherein the first image is acquired by the first image acquisition device.
14. The device according to claim 13, characterized in that The field of view conversion matrix is obtained by a camera field of view image and a field of view conversion model. The camera field of view image is acquired by the first image acquisition device. The field of view conversion model is trained based on training data. The training data includes a correspondence between a preset camera field of view image and a preset field of view conversion matrix.
15. The device according to claim 14, characterized in that The camera field of view image is the first image; or, the camera field of view image is acquired by the first image acquisition device before the first moment, and the posture of the first image acquisition device when acquiring the camera field of view image is consistent with the posture when acquiring the first image.
16. The device according to claim 14 or 15, characterized in that The device also includes: Matrix acquisition module, used to: Segmenting the camera field of view image into a plurality of image segments; Acquire a field of view conversion matrix corresponding to each image segment in the plurality of image segments through the field of view conversion model; The field of view conversion matrix of the first image acquisition device is determined according to the field of view conversion matrices corresponding to the multiple image segments.
17. The device according to any one of claims 12 to 16, characterized in that: The splicing module is used for: Converting the first bird's-eye view image into a pixel coordinate system corresponding to a second bird's-eye view image, wherein the first bird's-eye view image and the second bird's-eye view image are two images of the plurality of bird's-eye view images; The first bird's-eye view image and the second bird's-eye view image are fused in a pixel coordinate system corresponding to the second bird's-eye view image.
18. The device according to claim 17, characterized in that The stitching module is used to convert the first bird's-eye view image into a pixel coordinate system corresponding to the second bird's-eye view image according to an image conversion matrix from the first bird's-eye view image to the second bird's-eye view image, wherein the image conversion matrix is obtained based on matching pixel pairs in two reference bird's-eye view images, and the matching pixel pairs are obtained based on a graph neural network GNN model.
19. The device according to claim 18, characterized in that The two reference bird's-eye view images are respectively the first bird's-eye view image and the second bird's-eye view image; or, the two reference bird's-eye view images respectively correspond to two image acquisition devices among the multiple image acquisition devices, and the two camera field of view images corresponding to the two reference bird's-eye view images are acquired by the two image acquisition devices at a second moment, and the second moment is before the first moment.
20. The device according to any one of claims 12 to 19, characterized in that The adding module is used for: Determine at least two original pixel points corresponding to a first pixel point in the panoramic bird's-eye view image in the multiple images; The depth value of the first pixel is set according to the depth disparity of the at least two original pixel points.
21. The device according to any one of claims 12 to 20, characterized in that: The device also includes: The analysis module is used to perform traffic business analysis on the target road scene according to the real scene model image.
22. The device according to any one of claims 12 to 21, characterized in that Adjacent image acquisition devices among the multiple image acquisition devices have overlapping fields of view, and the fields of view of the multiple image acquisition devices cover the target road scene.
23. A traffic modeling device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory so that the traffic modeling device performs the method according to any one of claims 1 to 11.
24. A traffic modeling system, characterized in that: It comprises a traffic modeling device as described in any one of claims 12-23.
25. The system according to claim 24, characterized in that The system includes an edge computing device and a cloud computing device, and all or part of the functional modules of the traffic modeling device are located in the edge computing device or the cloud computing device.
26. The system according to claim 24 or 25, characterized in that The system includes a plurality of image acquisition devices, and the plurality of image acquisition devices are used to acquire a plurality of images of a target road scene; The traffic modeling device is used to construct a real-scene model image of the target road scene based on the multiple images.
27. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 11 is implemented.
28. A computer program product, characterized in that The computer program product comprises a program or a code, and when the program or the code is executed, the method according to any one of claims 1 to 11 is implemented.
