Vehicle identification method, system, device and storage medium based on multi-camera information fusion

Abstract

The present invention provides a vehicle identification method, system, device and storage medium for multi-camera information fusion, the method comprising the following steps: image acquisition devices are respectively arranged at both ends of a channel to collect forward images and backward images; a first image coordinate system and a second image coordinate system are respectively established based on the forward image and the backward image, and a channel space stereo coordinate system is established based on the channel to establish a mapping relationship; vehicle image recognition is performed on the forward image and the backward image; when at least two vehicles appear in the image, the image areas occupied by the front and rear of each vehicle in the forward image and the backward image are obtained in the channel space stereo coordinate system respectively corresponding to each other; the front pattern and the rear pattern of each vehicle are paired based on the spatial position relationship of each front and rear. The present invention can automatically identify scenes where multiple vehicles pass by based on multiple cameras, reduces the overall cost, greatly improves the accuracy and reaction speed of recognition, and has a high robustness of vehicle separation effect.

Description

Background Art

[0002] With the development of computer technology, the automatic identification technology of vehicles and containers is becoming more and more popular in my country's ports and docks, which can greatly reduce labor costs, speed up work efficiency and improve work accuracy. Among these automatic identification technologies, one of the core tasks is to solve the problem of following vehicles, that is, to assign the identified information to the corresponding vehicle when the vehicles are close to each other. Usually, the recognition task is completed by multiple cameras at different angles (such as recognizing license plates, recognizing container numbers, etc.), and the final recognition result needs to fuse this information together. When the distance between the front and rear vehicles is very close, the scenes seen by each camera are different (some are the pictures of the front vehicle, and some are the pictures of the rear vehicle), so it is easy to confuse the recognition results of the front and rear vehicles.

[0003] The current solutions include visual target tracking algorithms and laser radar-based vehicle segmentation solutions. The former has a single application scenario and can only be applied to simple operation scenarios, and the effect is unstable in complex actual operations. The latter requires additional laser radar equipment assistance, which is costly and increases technical complexity.

[0004] In view of this, the present invention provides a vehicle identification method, system, device and storage medium for multi-camera information fusion.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] In response to the problems in the prior art, the purpose of the present invention is to provide a vehicle identification method, system, device and storage medium with multi-camera information fusion, which overcomes the difficulties of the prior art and can automatically identify scenes of multiple vehicles passing by based on multiple cameras, thereby reducing the overall cost, greatly improving the accuracy and reaction speed of identification, and achieving high robustness of vehicle classification effect.

[0007] An embodiment of the present invention provides a vehicle identification method using multi-camera information fusion, comprising the following steps:

[0008] Image acquisition devices are respectively arranged at both ends of the channel to acquire forward images and backward images;

[0009] Establishing a first image coordinate system and a second image coordinate system based on the forward image and the backward image respectively, establishing the channel space stereo coordinate system based on the channel, and establishing a mapping relationship between each pixel in the first image coordinate system and the second image coordinate system and the spatial stereo coordinates;

[0010] Performing vehicle image recognition on the front image and the rear image;

[0011] When at least two vehicles appear in the image, the channel space three-dimensional coordinate system corresponding to the image areas occupied by the front and rear of each vehicle in the forward image and the backward image is obtained;

[0012] The front pattern and the rear pattern of each vehicle are paired based on the spatial position relationship between the front and rear of the vehicle.

[0013] Preferably, the image acquisition devices are respectively arranged at both ends of the channel to acquire the forward image and the backward image, including:

[0014] The channel is a one-way lane. An image acquisition device arranged in the outbound direction of the channel acquires forward images of vehicles passing through the channel, and an image acquisition device arranged in the incoming direction of the channel acquires rearward images of vehicles passing through the channel.

[0015] Preferably, the establishing of the first image coordinate system and the second image coordinate system based on the forward image and the backward image respectively, the establishing of the channel space stereo coordinate system based on the channel, and the establishing of the mapping relationship between each pixel in the first image coordinate system and the second image coordinate system and the spatial stereo coordinates include:

[0016] Establishing a first image coordinate system and a second image coordinate system based on the forward image and the backward image respectively;

[0017] Selecting four common feature points of the channel that are not on the same straight line, and recording the coordinates of each common feature point in the first image coordinate system and the second image coordinate system respectively;

[0018] The perspective transformation matrix of each of the image acquisition devices is obtained, and a mapping relationship between each pixel in the first image coordinate system and the second image coordinate system and the spatial stereo coordinates is established.

[0019] Preferably, the performing vehicle image recognition on the forward image and the backward image comprises:

[0020] Performing vehicle image recognition on the forward image, and identifying a coordinate range and first text information of a vehicle front region image within a pixel region corresponding to the vehicle portion;

[0021] The rearward image is subjected to vehicle image recognition, and the coordinate range of the vehicle rear region image within the pixel region corresponding to the vehicle portion and the second text information are identified.

[0022] Preferably, when at least two vehicles appear in the image, obtaining the channel space three-dimensional coordinate system corresponding to the image areas occupied by the front and rear of each vehicle in the forward image and the backward image, respectively, includes:

[0023] Mapping the image area occupied by the front region of the vehicle into the channel space three-dimensional coordinate system according to the perspective transformation matrix of the image acquisition device that acquires the forward image, and establishing a set of front region images based on at least two image areas occupied by the front of the vehicle that are separated from each other;

[0024] The image area occupied by the rear area of ​​the vehicle is mapped into the channel space stereo coordinate system according to the perspective transformation matrix of the image acquisition device that acquires the rearward image, and a set of rear area images is established based on at least two image areas occupied by the rear of the vehicle that are separated from each other.

[0025] Preferably, the pairing of the front pattern and the rear pattern of each vehicle based on the spatial position relationship between the front and rear of each vehicle includes:

[0026] Sequentially taking out one of the vehicle front area images from the set of vehicle front area images, and taking out a vehicle rear area image whose distance from the vehicle front along the direction in the channel space stereo coordinate system is the smallest from the vehicle rear area image from the set of vehicle rear area images, to form a whole vehicle image set;

[0027] The first text information and the second text information are added to a whole vehicle image set including the corresponding vehicle front area image and vehicle rear area image to generate vehicle identification information.

[0028] Preferably, the image area occupied by the front area of ​​the vehicle is mapped into the channel space three-dimensional coordinate system according to the perspective transformation matrix of the image acquisition device for acquiring the forward image, and a set of front area images is established based on at least two image areas occupied by the front of the vehicle that are separated from each other, and further includes:

[0029] Identifying a vehicle type corresponding to the vehicle head area image, and obtaining a preset length range of the vehicle based on the vehicle type;

[0030] The pairing of the front pattern and the rear pattern of each vehicle based on the spatial position relationship between the front and rear of each vehicle includes:

[0031] Based on the position of each of the vehicle front area images in the channel space three-dimensional coordinate system, a vehicle rear area pattern is matched in a preset length range corresponding to the range of the vehicle front area image along the direction of the channel to form a whole vehicle image set;

[0032] The first text information and the second text information are added to a whole vehicle image set including the corresponding vehicle front area image and vehicle rear area image to generate vehicle identification information.

[0033] An embodiment of the present invention further provides a vehicle identification system with multi-camera information fusion, which is used to implement the above-mentioned vehicle identification method with multi-camera information fusion. The vehicle identification system with multi-camera information fusion includes:

[0034] An image acquisition module, wherein image acquisition devices are respectively arranged at both ends of the channel to acquire forward images and backward images;

[0035] A mapping relationship module, which establishes a first image coordinate system and a second image coordinate system based on the forward image and the backward image, respectively, and establishes the channel space stereo coordinate system based on the channel, and establishes a mapping relationship between each pixel in the first image coordinate system and the second image coordinate system and the spatial stereo coordinates;

[0036] An image recognition module performs vehicle image recognition on the forward image and the backward image;

[0037] A partition recognition module, when at least two vehicles appear in the image, obtains the channel space three-dimensional coordinate system corresponding to the image areas occupied by the front and rear of each vehicle in the forward image and the backward image;

[0038] The pattern matching module matches the front pattern and the rear pattern of each vehicle based on the spatial position relationship between the front and rear of the vehicle.

[0039] An embodiment of the present invention further provides a vehicle identification device for multi-camera information fusion, comprising:

[0040] processor;

[0041] a memory in which executable instructions for the processor are stored;

[0042] The processor is configured to execute the steps of the above-mentioned vehicle identification method of multi-camera information fusion by executing executable instructions.

[0043] An embodiment of the present invention further provides a computer-readable storage medium for storing a program, which, when executed, implements the steps of the above-mentioned vehicle identification method using multi-camera information fusion.

[0044] The vehicle identification method, system, device and storage medium of the multi-camera information fusion of the present invention can automatically identify scenes of multiple vehicles passing by based on multiple cameras, thereby reducing overall costs, greatly improving the accuracy and reaction speed of identification, and achieving high robustness of vehicle classification effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0046] Figure 1It is a flow chart of the vehicle identification method of multi-camera information fusion of the present invention.

[0047] Figure 2 It is a schematic diagram of an implementation process of the vehicle identification method of multi-camera information fusion of the present invention.

[0048] Figures 3 to 9 It is a schematic diagram of another implementation process of the vehicle identification method of multi-camera information fusion of the present invention.

[0049] Fig.10 It is a structural schematic diagram of a vehicle identification system with multi-camera information fusion according to the present invention.

[0050] Fig.11 It is a structural schematic diagram of a vehicle identification device with multi-camera information fusion according to the present invention. And

[0051] Fig.12 It is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the present application. The present application can also be implemented or applied through other different specific implementation methods, and the details in the present application can also be modified or changed in various ways according to different viewpoints and application systems without departing from the spirit of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0053] The following is a detailed description of the embodiments of the present application with reference to the accompanying drawings so that those skilled in the art can easily implement the present application. The present application can be embodied in a variety of different forms and is not limited to the embodiments described herein.

[0054] In the representations of the present application, the representations with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials or characteristics represented in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the represented specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples represented in the present application and the features of the different embodiments or examples, unless they contradict each other.

[0055] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the representation of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0056] In order to clearly describe the present application, components irrelevant to the description are omitted, and the same reference symbols are given to the same or similar components throughout the specification.

[0057] Throughout the specification, when a device is said to be "connected" to another device, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a device is said to "include" a certain component, unless otherwise stated, it does not exclude other components, but means that other components may be included.

[0058] When a device is said to be "on" another device, it may be directly on the other device, but there may also be other devices between it. In contrast, when a device is said to be "directly" on another device, there are no other devices between it.

[0059] Although the terms first, second, etc. are used to represent various elements in the present invention in some examples, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first interface and the second interface, etc. are represented. Moreover, as used in the present invention, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising" and "including" indicate the existence of features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". An exception to this definition will occur only when a combination of elements, functions, steps or operations are inherently mutually exclusive in some manner.

[0060] The technical terms used herein are only used to refer to specific embodiments and are not intended to limit the present application. The singular form used herein also includes the plural form unless the sentence clearly indicates the contrary meaning. The meaning of "including" used in the specification is to specify specific characteristics, regions, integers, steps, operations, elements and / or components, and does not exclude the existence or addition of other characteristics, regions, integers, steps, operations, elements and / or components.

[0061] Although not defined differently, all terms, including technical and scientific terms used herein, have the same meaning as those generally understood by those skilled in the art to which this application belongs. Terms defined in commonly used dictionaries are additionally interpreted as having meanings consistent with the relevant technical literature and the contents of the current disclosure, and unless defined, shall not be overly interpreted as ideal or very formal meanings.

[0062] Figure 1 FIG. 1 is a flow chart of the vehicle identification method of the present invention using multi-camera information fusion. Figure 1 As shown, an embodiment of the present invention provides a vehicle identification method using multi-camera information fusion, comprising the following steps:

[0063] S110, image acquisition devices are respectively arranged at both ends of the channel to acquire forward images and backward images.

[0064] S120, establishing a first image coordinate system and a second image coordinate system based on the forward image and the backward image respectively, establishing a channel space stereo coordinate system based on the channel, and establishing a mapping relationship between each pixel in the first image coordinate system and the second image coordinate system and the spatial stereo coordinates.

[0065] S130: Perform vehicle image recognition on the front image and the rear image.

[0066] S140: When at least two vehicles appear in the image, obtain the channel space stereo coordinate system corresponding to the image areas occupied by the front and rear ends of each vehicle in the forward image and the backward image.

[0067] S150: Pairing the front pattern and the rear pattern of each vehicle based on the spatial position relationship between the front and rear of each vehicle.

[0068] The technical solution of the present invention maps the original multiple camera images to the same coordinate system after specific transformation, and uses the detected vehicle and container information to represent the vehicle's 3D position information in reality in this coordinate system, thereby dividing different vehicles into different coordinate ranges to achieve the ultimate purpose of vehicle division.

[0069] In a preferred embodiment, in step S110, the channel is a one-way lane, and an image acquisition device arranged in the outgoing direction of the channel collects forward images of vehicles passing through the channel, and an image acquisition device arranged in the incoming direction of the channel collects rearward images of vehicles passing through the channel, but the present invention is not limited thereto.

[0070] In a preferred embodiment, step S120 includes:

[0071] S121. Establish a first image coordinate system and a second image coordinate system based on the forward image and the backward image respectively.

[0072] S122, selecting four common feature points of the channel that are not on the same straight line, and recording the coordinates of each common feature point in the first image coordinate system and the second image coordinate system respectively.

[0073] S123, obtaining the perspective transformation matrix of each image acquisition device, and establishing a mapping relationship between each pixel in the first image coordinate system and the second image coordinate system and the spatial stereo coordinates, but not limited to this.

[0074] In a preferred embodiment, step S130 includes:

[0075] S131, performing vehicle image recognition on the forward image, and identifying the coordinate range and first text information of the vehicle front area image within the pixel area corresponding to the vehicle part.

[0076] S132, performing vehicle image recognition on the rearward image, and identifying the coordinate range of the vehicle rear area image within the pixel area corresponding to the vehicle portion and the second text information, but not limited thereto.

[0077] In a preferred embodiment, step S140 includes:

[0078] S141. Map the image area occupied by the front area of ​​the vehicle into the channel space stereo coordinate system according to the perspective transformation matrix of the image acquisition device that acquires the forward image, and establish a set of front area images based on the image areas occupied by at least two separate front areas.

[0079] S142. Map the image area occupied by the rear area of ​​the vehicle into the channel space three-dimensional coordinate system according to the perspective transformation matrix of the image acquisition device that acquires the rear image, and establish a set of rear area images based on the image areas occupied by at least two separated rear areas, but not limited to this.

[0080] In a preferred embodiment, step S150 includes:

[0081] S151. Take out one vehicle front area image in turn from the set of vehicle front area images, and take out a vehicle rear area image from the set of vehicle rear area images whose distance from the vehicle front along the direction in the channel space stereo coordinate system is the smallest, to form a whole vehicle image set.

[0082] S152: Add the first text information and the second text information to a whole vehicle image set including corresponding front region images and rear region images to generate vehicle identification information, but the present invention is not limited thereto.

[0083] In a preferred embodiment, step S131 also includes identifying the corresponding vehicle type according to the image of the front area of ​​the vehicle, and obtaining a preset length range of the vehicle based on the vehicle type.

[0084] Step S150 includes:

[0085] S153, matching a vehicle rear area pattern within a preset length range corresponding to the vehicle front area image in the channel space stereoscopic coordinate system based on the position of each vehicle front area image along the channel direction to form a whole vehicle image set.

[0086] S154: Add the first text information and the second text information to a whole vehicle image set including corresponding front region images and rear region images to generate vehicle identification information, but the present invention is not limited thereto.

[0087] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a technical solution for intelligently identifying vehicles in real time and locating their front and rear positional relationships based on the original multi-camera images, thereby solving the problem of difficulty in separating vehicles during the automatic recognition process. The advantages of the present invention are mainly reflected in the following aspects:

[0088] (1) The technology has low cost and can be based on existing detection cameras. It does not require the assistance of additional equipment such as lidar to confirm the relative position of vehicles in 3D space.

[0089] (2) The vehicle division effect is highly robust. The vehicle range is distinguished according to the real scene coordinates, and it is compatible with complex operation scenarios such as parking, reversing, and fast passing.

[0090] (3) It has strong scalability and can be applied to automatic detection processes such as license plate detection, box number detection, and weight limit recognition.

[0091] Figure 2 FIG. 1 is a schematic diagram of an implementation process of the vehicle identification method of the present invention using multi-camera information fusion. Figure 2 As shown, an implementation process of the present invention mainly includes:

[0092] First, in step S1, target detection is performed on each camera input, mainly detecting the front and rear of the vehicle and the front and rear box facades of the container on the vehicle, and obtaining the detection frame information of each target;

[0093] The specific steps of the S1 target detection step above are:

[0094] S11. Collect video or picture data of vehicles passing by in corresponding scenes.

[0095] S12. Manually label the data, with the specific targets being the front of the vehicle, the frame, the front door, and the rear door.

[0096] S13. For the labeled data, select 80% of the data to train the You-Only-Look-OnceV5 (YOLOV5) neural network model. Among them, YOLO (You Only Look Once), proposed by R. Joseph et al. in 2015, is the pioneering work of the YOLO series and the first one-stage detector in the field of deep learning. It is extremely fast and can run at 155fps when mAP=52.7% on VOC. As a one-stage detector, YOLO does not generate a suggestion box. It directly divides the image into S×S grid cells, and each grid detects the target whose center point falls into it (if the center point of the target falls into a grid, it is considered that this grid "contains" the target, and this grid is responsible for detecting the target). YOLO performs target detection end-to-end, directly from input to output, one-step, without intermediate steps. Moreover, YOLO training is also end-to-end.

[0097] S14. Use the trained model to perform target detection on each camera image to obtain rectangular detection frames of different targets. The coordinate format of the detection frames is [X1, Y1, X2, Y2], which respectively represent the coordinates of the upper left corner and the lower right corner of the target.

[0098] Next, in step S2, the detection frame information of each camera is mapped in a fixed coordinate system, so that the vehicle information of each camera is placed in the same coordinate system to obtain a unified vehicle position relationship;

[0099] The specific steps of the above S2 coordinate mapping steps are:

[0100] S21, calibrate each camera, select four common areas, and record their pixel positions in each camera, as shown in the attached figure Figure 3 , and map them to the same four coordinates, recording the corresponding four coordinate pixel positions.

[0101] S22. Based on the relative coordinates obtained in the previous step, use the SVD decomposition method to calculate according to the following formula, where x, y represent the pixel coordinates of the original point, and X, Y, and Z represent the pixel coordinates after the change (because it is 2D image information, the value of z is generally 1). Solving the equations can obtain the perspective transformation matrix A of each camera (a total of 9 coefficients, a33 defaults to 1). SVD (Singular Value Decomposition) is a tool of linear algebra, which reveals the most essential transformation of the matrix. Using SVD to decompose the matrix can obtain the matrix elements representing the most essential changes of the matrix. This is like a composite number can be expressed as the product of several prime numbers, and decomposing the composite number can obtain the prime factors representing the composite number; a complex periodic signal can be expressed as the sum of several simple sine waves and cosine waves, and using Fourier transform can obtain a simple wave representing the signal; the linear transformation represented by the complex matrix can be combined by the linear transformation represented by several simple matrices, and these simple matrices can be found using SVD.

[0102]

[0103] S23. According to the perspective change matrix A obtained in the previous step, the detection frames obtained in step S1 are respectively calculated according to the following formulas to map their pixel coordinates in the unified coordinate system, where x and y represent the original pixel coordinates of the detection frame, and X′, Y′, and Z′ represent the mapped pixel coordinates (Z′ defaults to 1)

[0104]

[0105] Then in step S3, the mapped coordinate information is passed into the logic algorithm, and the vehicle status and position range at the current moment are updated through logic filtering and matching based on the vehicle information at the previous moment;

[0106] The specific steps of the S3 logic algorithm are as follows:

[0107] S31, marking the vehicle status of the previous frame and the current frame as an initial status without a vehicle.

[0108] S32, after the current frame undergoes S1 target detection and S2 coordinate mapping steps, the position information of the vehicle under different viewing angles is obtained in a unified coordinate system, and then the number of vehicles and the vehicle coordinate range of the current frame are updated, and the information of the current frame is recorded as the previous frame.

[0109] S33, receiving a new frame of information, and repeating step S32.

[0110] Finally, in step S4, the current position ranges of each vehicle are output to other recognition processes so that the final recognition results can be matched to the vehicles in the corresponding ranges.

[0111] Figures 3 to 9 FIG. 1 is a schematic diagram of another implementation process of the vehicle identification method of the present invention using multi-camera information fusion. Figures 3 to 8 As shown, another implementation process of the present invention mainly includes:

[0112] like Figure 3 As shown, image acquisition devices are respectively arranged at both ends of a passage 10 based on a wharf or a road to collect forward images (images in the direction of the front of the vehicle) and backward images (images in the direction of the rear of the vehicle) of vehicles 21 and 22 passing through the passage 10. The passage 10 is a one-way lane, and an image acquisition device 11 (e.g., a road monitoring camera) arranged at the outgoing direction 10A of the passage 10 collects forward images of vehicles passing through the passage 10, and an image acquisition device 12 (e.g., a road monitoring camera) arranged at the incoming direction 10B of the passage 10 collects backward images of vehicles passing through the passage 10. The image acquisition device 11 and the image acquisition device 12 respectively send the captured videos to the server 3.

[0113] like Figures 4 to 7 As shown, a first image coordinate system is established based on the forward image 3. Figure 5 As shown, a second image coordinate system is established based on the backward image 4. The first image coordinate system and the second image coordinate system are established respectively with the backward image, and the spatial stereo coordinate system of the channel 10 is established based on the channel 10, and the mapping relationship between each pixel in the first image coordinate system and the second image coordinate system and the spatial stereo coordinate is established. For example: the first image coordinate system and the second image coordinate system are established based on the forward image and the backward image respectively; before a vehicle passes, four common feature points of the channel 10 that are not on the same straight line (which can be marks on the ground or roadside lamp posts, railings, etc.) are first selected, and the coordinates of each common feature point in the first image coordinate system and the second image coordinate system are recorded respectively; the perspective transformation matrix of each image acquisition device is obtained, so as to pre-establish the mapping relationship between each pixel in the first image coordinate system and the second image coordinate system and the spatial stereo coordinate before implementing subsequent recognition. In the present invention, the existing multi-camera camera based on ground markers can also be calibrated to obtain the position relationship between the pixel position in each camera and the real three-dimensional space, but it is not limited to this.

[0114] Continue to see Figure 4 and 6, the vehicle image recognition is performed on the forward image through the trained vehicle recognition neural network, and the coordinate ranges 33 and 34 of the front area image in the pixel areas 31 and 32 corresponding to the two vehicle parts are identified, and the first text information contained in the coordinate ranges 31 and 32 is identified through the text neural network that recognizes characters. Among them, the coordinate range 31 includes the license plate number: Tianjin AH6139 and the container number front box: OOLU 9974030 45G1. The coordinate range 32 includes the license plate number: Shanghai A123456. In addition, the correspondence between the front area 33 and the pixel area 31, and the front area 34 and the pixel area 32 is established. The corresponding vehicle model is also identified according to the front area image, and the preset length range of the vehicle is obtained based on the vehicle model. The vehicle model corresponding to the coordinate range 33 is a container truck, and the preset length range is 16 meters to 19 meters. The vehicle model corresponding to the coordinate range 34 is a family car, and the preset length range is 3.5 meters to 5 meters.

[0115] Continue to see Figure 5 and 7 , the rear image is subjected to vehicle image recognition, and the coordinate ranges of the vehicle rear region images 41 and 42 in the pixel region corresponding to the vehicle portion are identified, and the second text information contained in the coordinate range 41 and the coordinate range 44 are identified by a text neural network for character recognition, which will not be described in detail here. In addition, the corresponding relationship between the vehicle rear region 43 and the pixel region 41, and the vehicle rear region 44 and the pixel region 42 is established.

[0116] Continue to see Figure 8 , according to the perspective transformation matrix of the image acquisition device for acquiring the forward image, the image area occupied by the front area 33 and 34 of the vehicle is mapped to the spatial three-dimensional coordinate system of the channel 10, the first front area 33 corresponds to the first vertical area 331 in the spatial three-dimensional coordinate system, and the second front area 34 corresponds to the second vertical area 341 in the spatial three-dimensional coordinate system. And a set of front area images is established based on the image areas occupied by at least two mutually separated front areas 33 and 34.

[0117] According to the perspective transformation matrix of the image acquisition device that captures the rearward image, the image areas respectively occupied by the rear areas 43 and 44 of the vehicle are mapped to the spatial coordinate system of the channel 10, the third vertical area 431 in the spatial coordinate system corresponding to the first rear area 43, the fourth vertical area 441 in the spatial coordinate system corresponding to the second rear area 44, and a set of rear area images is established based on the image areas occupied by at least two separated rear areas 43 and 44.

[0118] A method of pairing the front and rear of a vehicle of the present invention is:

[0119] Successively take out a front-end area image from the set of front-end area images, and take out a rear-end area image with the smallest distance from the oncoming direction 10B to the front end in the channel 10 spatial three-dimensional coordinate system from the set of rear-end area images to form a complete vehicle image set; add the first text information and the second text information to the complete vehicle image set containing the corresponding front-end area image and rear-end area image to generate vehicle recognition information.

[0120] For example, first take out the first vertical area 331 corresponding to the first front-end area 33 from the set of front-end area images, and take out the third vertical area 431 corresponding to a rear-end area image with the smallest distance from the oncoming direction 10B to the first vertical area 331 in the channel 10 spatial three-dimensional coordinate system from the set of rear-end area images (in the oncoming direction 10B, the third vertical area 431 is closer to the first vertical area 331 than the fourth vertical area 441). Since the third vertical area 431 corresponds to the first rear-end area 43, then form a complete vehicle image set from the coordinate range 31 corresponding to the first front-end area 33 and the coordinate range 41 corresponding to the first rear-end area 43, and add the first text information "License plate number: Jin AH6139 and container number of the front container: OOLU 9974030 45G1" to the complete vehicle image set to generate vehicle recognition information for vehicle 21.

[0121] Then take out the second vertical area 341 corresponding to the remaining second front-end area 34 from the set of front-end area images, and take out the fourth vertical area 441 corresponding to a rear-end area image with the smallest distance from the oncoming direction 10B to the second vertical area 341 in the channel 10 spatial three-dimensional coordinate system from the set of rear-end area images. Since the fourth vertical area 441 corresponds to the second rear-end area 44, then form a complete vehicle image set from the coordinate range 32 corresponding to the second front-end area 34 and the coordinate range 42 corresponding to the second rear-end area 44, and add the first text information "License plate number: Hu A123456" included in the coordinate range 32 to the complete vehicle image set to generate vehicle recognition information for vehicle 22. Thus, the vehicle separation recognition detection for multiple vehicles passing through the channel simultaneously is realized, and moreover, it can be used for the recognition scenarios of three vehicles or four vehicles passing through simultaneously.

[0122] Continue to refer to Fig. 9 , another way of pairing the front end and the rear end of the present invention is:

[0123] Using the previously obtained information on the preset length range of the vehicle based on the vehicle type, the vehicle type corresponding to coordinate range 33 is a container truck, and the preset length range is from 16 meters to 19 meters. The vehicle type corresponding to coordinate range 34 is a sedan, and the preset length range is from 3.5 meters to 5 meters. Match a rear vehicle area pattern in the corresponding preset length range of this range for each front vehicle area image along the incoming direction 10B of channel 10 in the three-dimensional coordinate system of channel 10 to form a set of whole vehicle images; add the first text information and the second text information to the set of whole vehicle images including the corresponding front vehicle area images and rear vehicle area images to generate vehicle identification information.

[0124] For example: First, take out the first vertical area 331 corresponding to the first front vehicle area 33 from the set of front vehicle area images, extend a range of 16 meters to 19 meters from the position of the first vertical area 331 along the incoming direction 10B, and take out a third vertical area 431 located within this range from the set of rear vehicle area images (in this embodiment, the distance D1 between the center of the third vertical area 431 and the center of the first vertical area 331 is 17.5 meters, meeting the requirement that the front vehicle image in coordinate range 33 corresponds to the preset vehicle length range of 16 meters to 19 meters). Since the first rear vehicle area 43 corresponds to the third vertical area 431, then form a set of whole vehicle images from the coordinate range 31 corresponding to the first front vehicle area 33 and the coordinate range 41 corresponding to the first rear vehicle area 43, and add the first text information "License plate number: Jin AH6139 and container number of the front container: OOLU 9974030 45G1" to the set of whole vehicle images to generate vehicle identification information for vehicle 21.

[0125] Similarly, take out the third vertical area 341 corresponding to the second front vehicle area 34 from the set of front vehicle area images, extend a range of 16 meters to 19 meters from the position of the third vertical area 341 along the incoming direction, and take out a fourth vertical area 441 located within this range from the set of rear vehicle area images (in this embodiment, the distance D1 between the center of the fourth vertical area 441 and the center of the second vertical area 341 is 4.2 meters, meeting the requirement that the front vehicle image in coordinate range 34 corresponds to the preset vehicle length range of 3.5 meters to 5 meters). Since the second rear vehicle area 44 corresponds to the fourth vertical area 441, then form a set of whole vehicle images from the coordinate range 32 corresponding to the second front vehicle area 34 and the coordinate range 42 corresponding to the second rear vehicle area 44, and add the first text information "License plate number: Hu A123456" included in the coordinate range 32 to the set of whole vehicle images to generate vehicle identification information for vehicle 22. Details will not be elaborated hereafter.

[0126] The neural networks used in the present invention for identifying vehicle images, front images, and rear images are all image recognition neural networks trained with a large number of vehicle, front, and rear images, thereby improving recognition progress and calculation speed. The mapping relationship model based on the front of the vehicle and the vehicle model and length is a preset mapping model, which will not be described in detail here.

[0127] The image acquisition devices 11 and 12 used in the present invention are the most basic cameras, which do not require laser radar or binocular cameras, thereby greatly reducing the overall cost and being beneficial to intelligent upgrades based on existing equipment in scenarios such as docks and gates.

[0128] Fig.10 FIG. 1 is a schematic diagram of the structure of the vehicle identification system of the present invention with multi-camera information fusion. Fig.10 As shown, the vehicle identification system 5 of the present invention with multi-camera information fusion includes:

[0129] The image acquisition module 51 has image acquisition devices at both ends of the channel to acquire forward images and backward images.

[0130] The mapping relationship module 52 establishes a first image coordinate system and a second image coordinate system based on the forward image and the backward image, respectively, and establishes a channel space stereo coordinate system based on the channel, and establishes a mapping relationship between each pixel in the first image coordinate system and the second image coordinate system and the spatial stereo coordinates.

[0131] The image recognition module 53 performs vehicle image recognition on the front image and the rear image.

[0132] The partition recognition module 54 obtains the channel space three-dimensional coordinate system corresponding to the image area occupied by the front and rear of each vehicle in the forward image and the backward image, respectively, when at least two vehicles appear in the image.

[0133] The pattern matching module 55 matches the front pattern and the rear pattern of each vehicle based on the spatial position relationship between the front and rear of each vehicle.

[0134] In a preferred embodiment, the image acquisition module 51 is configured to make the channel a one-way lane, and the image acquisition device arranged in the outgoing direction of the channel collects forward images of vehicles passing through the channel, and the image acquisition device arranged in the incoming direction of the channel collects rearward images of vehicles passing through the channel.

[0135] In a preferred embodiment, the mapping relationship module 52 is configured to establish a first image coordinate system and a second image coordinate system based on the forward image and the backward image, respectively. Four common feature points of the channel that are not on the same straight line are selected, and the coordinates of each common feature point in the first image coordinate system and the second image coordinate system are recorded respectively. The perspective transformation matrix of each image acquisition device is obtained, and the mapping relationship between each pixel in the first image coordinate system and the second image coordinate system and the spatial stereo coordinate is established.

[0136] In a preferred embodiment, the image recognition module 53 is configured to perform vehicle image recognition on the forward image, and identify the coordinate range and first text information of the front region image in the pixel region corresponding to the vehicle portion. Perform vehicle image recognition on the backward image, and identify the coordinate range and second text information of the rear region image in the pixel region corresponding to the vehicle portion.

[0137] In a preferred embodiment, the partition identification module 54 is configured to map the image area occupied by the front area of ​​the vehicle into the channel space three-dimensional coordinate system according to the perspective transformation matrix of the image acquisition device that acquires the forward image, and establish a set of images of the front area based on the image areas occupied by at least two mutually separated front areas. Map the image area occupied by the rear area of ​​the vehicle into the channel space three-dimensional coordinate system according to the perspective transformation matrix of the image acquisition device that acquires the backward image, and establish a set of images of the rear area based on the image areas occupied by at least two mutually separated rear areas.

[0138] In a preferred embodiment, the pattern matching module 55 is configured to sequentially extract a vehicle front region image from the vehicle front region image set, and extract a vehicle rear region image with the smallest distance from the vehicle front in the channel space stereo coordinate system along the incoming direction from the vehicle rear region image set to form a vehicle image set. The first text information and the second text information are added to the vehicle image set containing the corresponding vehicle front region image and vehicle rear region image to generate vehicle identification information.

[0139] In a preferred embodiment, the image recognition module 53 is further configured to recognize the corresponding vehicle type according to the image of the front area of ​​the vehicle, and obtain a preset length range of the vehicle based on the vehicle type.

[0140] The pattern matching module 55 is configured to match a rear area pattern in a corresponding preset length range of the front area image along the direction of the channel based on the position of each front area image in the channel space three-dimensional coordinate system to form a whole vehicle image set. The first text information and the second text information are added to the whole vehicle image set containing the corresponding front area image and rear area image to generate vehicle identification information.

[0141] The multi-camera information fusion vehicle identification system of the present invention can automatically identify scenes of multiple vehicles passing by based on multiple cameras, thereby reducing overall costs, greatly improving identification accuracy and response speed, and achieving high robustness in vehicle classification effects.

[0142] The embodiment of the present invention further provides a vehicle identification device for multi-camera information fusion, comprising a processor and a memory, wherein executable instructions of the processor are stored. The processor is configured to execute the steps of the vehicle identification method for multi-camera information fusion by executing the executable instructions.

[0143] As described above, the vehicle identification device with multi-camera information fusion of the present invention can automatically identify scenes of multiple vehicles passing by based on multiple cameras, thereby reducing the overall cost, greatly improving the accuracy and reaction speed of identification, and achieving high robustness of vehicle classification effect.

[0144] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0145] Fig.11 This is a schematic diagram of the structure of the vehicle identification device of the present invention with multi-camera information fusion. Fig.11 The electronic device 600 according to this embodiment of the present invention is described. Fig.11 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0146] like Fig.11 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0147] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the electronic prescription circulation processing method section of this specification. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0148] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0149] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0150] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0151] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0152] The embodiment of the present invention also provides a computer-readable storage medium for storing a program, and when the program is executed, the steps of the vehicle identification method of multi-camera information fusion are implemented. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned electronic prescription circulation processing method section of this specification.

[0153] As shown above, when the program of the computer-readable storage medium of this embodiment is executed, it can automatically identify scenes of multiple vehicles passing by based on multiple cameras, thereby reducing the overall cost, greatly improving the accuracy and reaction speed of recognition, and achieving high robustness of vehicle division effect.

[0154] Fig.12 Schematic diagram of the structure of the computer-readable storage medium of the present invention. Fig.12 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0155] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0156] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0157] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0158] In summary, the vehicle identification method, system, device and storage medium of the multi-camera information fusion of the present invention can automatically identify the scene of multiple vehicles passing by based on multiple cameras, reduce the overall cost, greatly improve the accuracy and reaction speed of identification, and the vehicle division effect is highly robust.

[0159] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A vehicle identification method based on multi-camera information fusion, characterized in that: The steps include: Image acquisition devices are respectively arranged at both ends of the channel in the direction of the channel to acquire forward images and backward images; Establishing a first image coordinate system and a second image coordinate system based on the forward image and the backward image respectively; Select four common feature points of the channel that are not on the same straight line, and record the coordinates of each common feature point in the first image coordinate system and the second image coordinate system respectively; obtain the perspective transformation matrix of each image acquisition device, and establish a mapping relationship between each pixel and the spatial stereo coordinates in the first image coordinate system and the second image coordinate system; Performing vehicle image recognition on the forward image, and identifying the coordinate range and first text information of the vehicle front region image within the pixel region corresponding to the vehicle portion; performing vehicle image recognition on the backward image, and identifying the coordinate range and second text information of the vehicle rear region image within the pixel region corresponding to the vehicle portion; mapping the image region occupied by the vehicle front region into a channel space stereoscopic coordinate system according to the perspective transformation matrix of the image acquisition device that acquires the forward image, and establishing a set of vehicle front region images based on at least two mutually separated image regions occupied by the vehicle front region; Mapping the image area occupied by the rear area of ​​the vehicle into the channel space three-dimensional coordinate system according to the perspective transformation matrix of the image acquisition device for acquiring the rearward image, and establishing a set of rear area images based on at least two image areas occupied by the rear of the vehicle that are separated from each other; as well as The vehicle front area image is sequentially taken out from the set of vehicle front area images, and a vehicle rear area image which has the smallest distance from the vehicle front along the direction in the channel space three-dimensional coordinate system is taken out from the set of vehicle rear area images to form a whole vehicle image set; the first text information and the second text information are added to the whole vehicle image set containing the corresponding vehicle front area image and vehicle rear area image to generate vehicle identification information.

2. The vehicle identification method based on multi-camera information fusion according to claim 1, characterized in that: The image acquisition devices are respectively arranged at both ends of the channel to acquire the forward image and the backward image, including: The channel is a one-way lane. An image acquisition device arranged in the outbound direction of the channel acquires forward images of vehicles passing through the channel, and an image acquisition device arranged in the incoming direction of the channel acquires rearward images of vehicles passing through the channel.

3. The vehicle identification method based on multi-camera information fusion according to claim 1, characterized in that: The method further includes mapping the image area occupied by the front area of ​​the vehicle into the channel space three-dimensional coordinate system according to the perspective transformation matrix of the image acquisition device for acquiring the forward image, and establishing a set of front area images based on at least two image areas occupied by the front of the vehicle that are separated from each other, and further includes: Identifying a vehicle type corresponding to the vehicle head area image, and obtaining a preset length range of the vehicle based on the vehicle type; The pairing of the front pattern and the rear pattern of each vehicle based on the spatial position relationship between the front and rear of each vehicle includes: Based on the position of each of the vehicle front area images in the channel space three-dimensional coordinate system, a vehicle rear area pattern is matched in a preset length range corresponding to the range of the vehicle front area image along the direction of the channel to form a whole vehicle image set; The first text information and the second text information are added to a whole vehicle image set including the corresponding vehicle front area image and vehicle rear area image to generate vehicle identification information.

4. A vehicle identification system with multi-camera information fusion, characterized in that: The system comprises: An image acquisition module, wherein image acquisition devices are respectively arranged at both ends of the channel in the direction of the channel to acquire forward images and backward images; A mapping relationship module is provided, which establishes a first image coordinate system and a second image coordinate system based on the forward image and the backward image respectively; selects four common feature points of the channel that are not on the same straight line, and records the coordinates of each common feature point in the first image coordinate system and the second image coordinate system respectively; obtains the perspective transformation matrix of each image acquisition device, and establishes a mapping relationship between each pixel in the first image coordinate system and the second image coordinate system and the spatial stereo coordinates; An image recognition module performs vehicle image recognition on the forward image, and identifies a coordinate range and a first text message of a front region image within a pixel region corresponding to the vehicle portion; performs vehicle image recognition on the backward image, and identifies a coordinate range and a second text message of a rear region image within a pixel region corresponding to the vehicle portion; a partition recognition module, mapping the image area occupied by the front area of ​​the vehicle into the channel space three-dimensional coordinate system according to the perspective transformation matrix of the image acquisition device for acquiring the forward image, and establishing a set of front area images based on at least two image areas occupied by the front area separated from each other; mapping the image area occupied by the rear area of ​​the vehicle into the channel space three-dimensional coordinate system according to the perspective transformation matrix of the image acquisition device for acquiring the backward image, and establishing a set of rear area images based on at least two image areas occupied by the rear area separated from each other; and The pattern matching module sequentially takes out one of the vehicle front area images from the set of vehicle front area images, and takes out a vehicle rear area image from the set of vehicle rear area images that has the smallest distance from the vehicle front along the direction in the channel space three-dimensional coordinate system, to form a whole vehicle image set; and adds the first text information and the second text information to the whole vehicle image set containing the corresponding vehicle front area image and vehicle rear area image to generate vehicle identification information.

5. A vehicle identification device with multi-camera information fusion, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to execute the steps of the vehicle identification method for multi-camera information fusion according to any one of claims 1 to 3 by executing the executable instructions.

6. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the vehicle identification method using multi-camera information fusion as described in any one of claims 1 to 3 are implemented.

Images