Method and system for associating two or more images

Abstract

A computer-implemented method and system for associating objects across two or more images is provided. The method includes receiving a first image from a first image sensor and a second image from a second image sensor, identifying corresponding points between the first image and the second image, approximating a projection matrix between the first image and the second image based on the identified corresponding points, the projection matrix including at least one polynomial, and associating at least one identified object in the first image with a corresponding identified object in the second image. The system can be mounted on a vehicle, a building, or any infrastructure.

Description

[Technical Field]

[0001] The present invention relates generally to computer vision, and more particularly to a method and system for associating objects between two or more images captured by separate image sensors. [Background technology]

[0002] Many computer vision applications require tracking multiple objects across multiple cameras, a process known as multi-target multi-camera (MTMC) tracking. MTMC tracking is used in several applications, including driving, where multiple traffic participants, such as vehicles and pedestrians, as well as infrastructure, are tracked across multiple cameras. Many MTMC tracking methods favor wide-angle or ultra-wide-field-of-view cameras, also known as fisheye cameras, because they have a wider field of view compared to rectilinear cameras. Images captured by fisheye cameras, also known as fisheye images, therefore contain more information than rectilinear images. An essential requirement for MTMC tracking is a representation of the scene captured by the multiple cameras within a common coordinate system. Such a representation requires pixel-to-pixel mapping between the two images, which is particularly challenging for fisheye images due to the distortions present in such images.

[0003] Current approaches to solving problems related to pixel-to-pixel mapping in fisheye images use either image rectification or image registration, both of which have several drawbacks. Image rectification involves aligning the image planes of two fisheye cameras using camera calibration parameters, which are difficult to calculate, require human intervention for reliable detection of calibration points, and are undesirable because they must be recalculated for each new setup and periodically due to potential shifts in camera position or lens aberrations. Furthermore, a significant overlap area is required for efficient image rectification. Image registration, on the other hand, involves identifying corresponding points, also known as keypoints, between the two image planes after distortion correction and performing pixel-to-pixel mapping between the images using a transformation calculated between the identified corresponding points. Affine transformations are typically used to approximate the geometric transformation between two image planes, but because affine transformations approximate rigid transformations, they cannot be applied to non-rigid transformations between the image planes of a rectilinear camera and a fisheye camera, or between the image planes of two fisheye cameras. In addition, a significant overlapping region is required for efficient image registration, and keypoint matching between two images is difficult due to different camera views of the same object. Summary of the Invention

[0004] Embodiments of the present invention improve image mapping by associating two or more images using a projection matrix for the transformation between two image planes or images. In some embodiments, the projection matrix can use polynomials and coefficients to approximate the non-rigid transformation between the two. The projection matrix can be used for pixel-to-pixel mapping between images to facilitate object association between images, allowing for tracking multiple objects across two or more cameras, which can be used for subsequent applications such as driving, computer vision applications, and security or surveillance applications.

[0005] In order to solve the above technical problem, the present invention provides a vehicle equipped for associating objects across two or more images, the vehicle system including at least a first image sensor and a second image sensor, one or more processors, and executable instructions for execution by the one or more processors, the executable instructions including: receiving a first image from the first image sensor and a second image from the second image sensor; Identifying corresponding points between the first image and the second image; approximating a projection matrix between the first image and the second image based on the identified corresponding points; associating at least one identified object in the first image with a corresponding identified object in the second image based at least on the approximated projection matrix; and a memory storing executable instructions including instructions for performing a computer-implemented method including:

[0006] The vehicle system of the present invention executes a computer-implemented method that is fully automated and requires no human intervention. Object association across two or more images by identifying corresponding points between images and approximating a projection matrix can be triggered automatically, thus avoiding the need for manual triggering or recalculation. The computer-implemented method has several advantages over previous solutions. Unlike image rectification, the computer-implemented method of the present invention does not rely on calibration parameters, which are labor-intensive and prone to errors when performed automatically. The computer-implemented method of the present invention is more computationally efficient because it does not require distortion correction. Furthermore, the computer-implemented method of the present invention is also advantageous because it does not require significant overlap between the image sensor's field of view and / or the images captured by the image sensor.

[0007] A preferred vehicle of the present invention is the vehicle described above, comprising one or more processors and executable instructions for execution by the one or more processors, receiving a first image from a first image sensor and a second image from a second image sensor; Identifying corresponding points between the first image and the second image; approximating a projection matrix between the first image and the second image based on the identified corresponding points; A memory storing executable instructions including instructions for performing a computer-implemented method including associating at least one identified object in the first image with a corresponding identified object in the second image based at least on the approximated projection matrix; and plotting positions of the identified objects on one common image plane, the one common image plane preferably covering a 360° surround view of the objects around the vehicle.

[0008] The above-described aspects of the present invention have the advantage that a single common image plane containing the positions of identified objects can better represent the positions of the objects than an image taken by a single fisheye camera with a large field of view that distorts the objects and their positions.

[0009] The above-described advantageous aspects of the vehicle of the present invention also apply to all aspects of the computer-implemented method of the present invention described below. All below-described advantageous aspects of the computer-implemented method of the present invention also apply to all aspects of the vehicle of the present invention described above.

[0010] The present invention also relates to a computer-implemented method for associating objects across two or more images, the method comprising: receiving a first image from a first image sensor and a second image from a second image sensor; Identifying corresponding points between the first image and the second image; approximating a projection matrix between the first image and the second image based on the identified corresponding points; and associating at least one identified object in the first image with a corresponding identified object in the second image based at least on the approximated projection matrix.

[0011] The computer-implemented method of the present invention is fully automated and does not require human intervention. The association of objects across two or more images by approximating a projection matrix can be triggered automatically, thus avoiding the need for manual triggering or recalculation. Furthermore, the computer-implemented method of the present invention has several advantages over previous solutions. Unlike image rectification, the computer-implemented method of the present invention does not rely on calibration parameters, which are labor-intensive and prone to errors when performed automatically. The computer-implemented method of the present invention is more computationally efficient because it does not require distortion correction. Furthermore, the computer-implemented method of the present invention is also advantageous because it does not require significant overlap between the image sensor's field of view and / or the images captured by the image sensor.

[0012] A preferred method of the present invention is the computer-implemented method described above, wherein the first image sensor and / or the second image sensor are fisheye cameras, and / or the first image sensor and the second image sensor are positioned such that there is an overlap between at least one first image of the scene captured by the first image sensor and at least one second image of the scene captured by the second image sensor.

[0013] The above-described aspects of the present invention have the advantage that fisheye cameras have a wider field of view. The wider the field of view, the greater the overlap area between the images generated by the cameras. This allows for a larger area to be covered using fewer cameras. For example, fewer cameras can be used to obtain a 360° view of the surroundings of the vehicle, which may be useful for driving applications.

[0014] A preferred method of the present invention is a computer-implemented method as described above or as preferred above, Corresponding points are points that correspond to distinct points, preferably vertices or intersections, and / or The corresponding points are key points.

[0015] The above-described aspects of the present invention have the advantage that not only keypoints but also distinct points, such as points corresponding to vertices or intersections, are distinctive points in an image and can be identified in the image regardless of orientation or distortion. Using points corresponding to vertices or intersections is advantageous because they are well-defined and easily detected, thus ensuring that the same points are accurately detected and selected in both the first and second images.

[0016] A preferred method of the present invention is a computer-implemented method as described above, or as preferred above, wherein the keypoints are selected using a neural network, preferably a neural network comprising a convolutional neural network backbone and a keypoint extractor, and wherein the convolutional neural network backbone preferably comprises constrained deformable convolution modules.

[0017] The above-described aspects of the present invention have the advantage that using neural networks to select keypoints allows for the learning and modeling of nonlinear and complex relationships, and subsequent application to new datasets or inputs. Furthermore, neural networks have the ability to learn on their own and generate outputs that are not limited to the inputs provided. A convolutional neural network (CNN) backbone is preferred because CNNs have high accuracy in image recognition and can automatically filter images and detect features without human supervision, while CNNs with limited deformable convolutions are preferred because they account for fisheye image distortion.

[0018] A preferred method of the invention is a computer-implemented method as described above or as preferred above, wherein the step of approximating a projection matrix comprises approximating at least one projection matrix comprising at least one polynomial, preferably wherein the at least one polynomial has a degree n and the number of identified corresponding points is between n+1 and n+7, where n is an integer greater than or equal to 2, and / or preferably wherein the at least one polynomial has a degree 2 and the number of identified corresponding points is between 3 and 9.

[0019] The above-described aspects of the present invention are advantageous because, unlike affine transformations in image registration, which are limited to rigid transformations, the use of polynomials can account for non-rigid transformations and / or projection approximations, as well as non-linear geometry or distortions present in fisheye images. Using a polynomial of degree n and having n+1 to n+7 corresponding points is advantageous because the approximated polynomial is accurate and may be sufficient for the purpose without overfitting. Using a polynomial of degree 2 and having 3 to 9 corresponding points is also advantageous because a polynomial of degree 2 may often be accurate enough for the purpose without overfitting.

[0020] A preferred method of the invention is a computer-implemented method as described above or as preferred above, wherein the projection matrix comprises two polynomials: a first polynomial for coordinates on the x-axis and a second polynomial for coordinates on the y-axis.

[0021] The above-described aspects of the present invention have the advantage that calculating different polynomials for the x-axis and y-axis increases the accuracy of the method, as the different polynomials for the x-axis and y-axis take into account variations in distortion along the x-axis and y-axis in images, particularly fisheye images.

[0022] A preferred method of the invention is a computer-implemented method as described above, or as described above as preferred, wherein associating at least one identified object in the first image with a corresponding identified object in the second image comprises: identifying at least one object in the first image and at least one object in the second image; generating at least one first bounding box in the first image and at least one second bounding box in the second image, each of the at least one first bounding box and the at least one second bounding box representing a spatial location of an identified object; and associating each first bounding box with a corresponding second bounding box based at least on a relationship between a projection of the first bounding box and the corresponding second bounding box, the projection being based on the approximated projection matrix.

[0023] The above-described aspects of the present invention have the advantage that the approximated projection matrix allows for accurate projection of the positions of multiple objects or bounding boxes from a first image onto a second image for comparison and association as a group or on a larger scale.

[0024] A preferred method of the invention is a computer-implemented method as described above, or as described above as preferred, wherein associating each first bounding box with a corresponding second bounding box comprises: projecting at least one first bounding box onto the second image based on the approximated projection matrix to generate at least one projected first bounding box on the second image, each projected first bounding box corresponding to one of the at least one first bounding box; determining a relationship between each projected first bounding box and each second bounding box; and associating each first bounding box with a corresponding second bounding box based on the determined relationship between the corresponding projected first bounding box and the corresponding second bounding box.

[0025] The above-described aspects of the present invention have the advantage that the association or matching between multiple objects or bounding boxes in a first image and multiple objects or bounding boxes in a second image is optimized as a group or on a global scale based on the positions of the multiple objects or bounding boxes.

[0026] A preferred method of the present invention is a computer-implemented method as described above or as described above as preferred, wherein the relationship between each projected first bounding box and each second bounding box is determined by the degree of overlap between each projected first bounding box and each second bounding box, and / or The distance between each projected first bounding box and each second bounding box includes the distance, preferably between the center of each projected first bounding box and the center of each second bounding box.

[0027] The above-described aspects of the present invention have the advantage that the positions of multiple objects or bounding boxes can be projected and matched simultaneously across two or more image sensors.

[0028] A preferred method of the invention is a computer-implemented method as described above, or as described above as preferred, wherein associating each first bounding box with a corresponding second bounding box comprises: identifying at least one feature of each identified object in the first image and the second image, the at least one feature preferably being an appearance feature vector; Further comprising comparing at least one characteristic of each identified object in the first image with at least one characteristic of each identified object in the second image.

[0029] The above-described aspects of the present invention have the advantage that in addition to the location of the objects or their bounding boxes, the visual similarity of the objects is also taken into account during association, so that the visual features or appearance of the objects are also compared, ensuring improved accuracy of the computer-implemented method, which is particularly advantageous in situations where there is little overlap between the first and second images.

[0030] A preferred method of the present invention is a computer-implemented method as described above or as preferred above, wherein the method further comprises plotting the positions of the identified objects on a common image plane, the common image plane preferably covering a 360° surround view of the objects around the first, second and further image sensors.

[0031] The above-described aspects of the present invention have the advantage that a plot of object positions on a common image plane provides an accurate representation of the spatial location of objects in a scene. A single common image plane containing the locations of identified objects can better represent the object locations than an image taken by a single fisheye camera with a large field of view, which distorts the objects and their locations.

[0032] A particularly preferred method of the present invention is a computer-implemented method as described above or as described above as preferred, the first image sensor and the second image sensor are fisheye cameras positioned such that there is an overlap between at least one first image of the scene captured by the first image sensor and at least one second image of the scene captured by the second image sensor; The corresponding points are selected using a neural network including a convolutional neural network backbone and a keypoint extractor, the neural network being trained on a scene classification dataset including images sorted into scene classes, each scene class including a series of images captured from a single camera capturing overlapping regions of a scene and a series of images captured over the same time period by another camera having an overlapping field of view with the single camera, the convolutional neural network backbone including a constrained deformable convolution module; the approximated projection matrix includes a first polynomial for coordinates on the x-axis and a second polynomial for coordinates on the y-axis, the first polynomial and the second polynomial each having a degree of 2, and the number of identified corresponding points is 3; Associating at least one identified object in the first image with a corresponding identified object in the second image includes: identifying at least one object in the first image and at least one object in the second image; generating at least one first bounding box in the first image and at least one second bounding box in the second image, each of the at least one first bounding box and the at least one second bounding box representing a spatial location of an identified object; projecting the at least one first bounding box onto the second image based on the approximated projection matrix to generate at least one projected first bounding box onto the second image, wherein each projected first bounding box corresponds to one of the at least one first bounding box; determining a degree of overlap between each projected first bounding box and each second bounding box; and associating each first bounding box with a corresponding second bounding box based on the determined degree of overlap between the corresponding projected first bounding box and the corresponding second bounding box.

[0033] The above-described advantageous aspects of the vehicle or computer-implemented method of the present invention also apply to all aspects of the training dataset of the present invention described below. All below-described advantageous aspects of the training dataset of the present invention also apply to all aspects of the vehicle or computer-implemented method of the present invention described above.

[0034] The invention also relates to a training data set for a neural network, in particular a neural network according to the invention, the training data set comprising images sorted into scene classes, each scene class having: a series of images captured from a first image sensor capturing an overlapping region of the scene; Optionally, a series of images captured during the same time period by another image sensor having an overlapping field of view with the first image sensor; and Optionally, the sequence of images captured from the first image sensor and / or an image modified from a sequence of images captured during the same period by another image sensor having a field of view that overlaps with the first image sensor.

[0035] The training data set of the present invention advantageously contains a large number of images for each scene class for training a neural network for keypoint identification. The training data set contains a wide variety of images for each scene class from limited images and limited image sensors.

[0036] The above-described advantageous aspects of the vehicle, computer-implemented method, or training dataset of the present invention also apply to all aspects of the system described below of the present invention. All below-described advantageous aspects of the system of the present invention also apply to all aspects of the vehicle, computer-implemented method, or training dataset of the present invention.

[0037] The present invention also relates to a system adapted for use in a vehicle comprising at least a first image sensor and a second image sensor, one or more processors, and a memory storing executable instructions for execution by the one or more processors, the executable instructions including instructions for performing a computer-implemented method according to the present invention.

[0038] The above-mentioned advantageous aspects of the vehicle, computer-implemented method, training data set or system of the present invention also apply to all aspects of the computer program, machine-readable medium or data signal described below of the present invention.All the above-mentioned advantageous aspects of the computer program, machine-readable medium or data signal of the present invention also apply to all aspects of the vehicle, computer-implemented method, training data set or system described below of the present invention.

[0039] The present invention also relates to a computer program product, machine-readable medium, or data carrier signal comprising instructions that, when executed on one or more processors, cause the one or more processors to perform a computer-implemented method according to the present invention. A machine-readable medium may include any medium and / or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). A machine-readable medium may be any medium, such as, for example, a read-only memory (ROM), a random-access memory (RAM), a universal serial bus (USB) stick, a compact disc (CD), a digital video disc (DVD), a data storage device, a hard disk, an electrical, acoustical, optical, or other form of propagated signal (e.g., a digital signal, a data carrier signal, a carrier wave), or any other medium capable of transmitting and / or storing program elements as described above.

[0040] As used in this Summary, the following description, the following claims, and the accompanying drawings, the term "scene" refers to a distinct physical environment that can be captured by one or more image sensors. A scene can include one or more objects that can be visually captured by one or more image sensors, whether such objects are stationary or moving.

[0041] As used in this Summary, the following description, the following claims, and the accompanying drawings, the term "fisheye camera" refers to an image sensor, camera, and / or video camera equipped with a fisheye or wide-angle lens with a field of view of 60 degrees or greater, and the term "fisheye image" refers to an image captured or generated by a fisheye camera. A fisheye image may also be characterized as a spherical or hemispherical image.

[0042] As used in this Summary, the following description, the following claims, and the accompanying drawings, the term "bounding box" refers to a bounding region of an object and may include a bounding box, a bounding circle, a bounding ellipse, or any other suitably shaped region representing the object. A bounding box associated with an object may have a rectangular, square, polygonal, blob-shaped, or any other suitable shape.

[0043] As used in this Summary, the following description, the following claims, and the accompanying drawings, the term "vehicle" refers to any mobile agent that is capable of movement, including cars, trucks, buses, agricultural machinery, forklifts, and robots, whether or not capable of carrying or transporting goods, animals, or humans.

[0044] As used in this Summary, the following description, the following claims, and the accompanying drawings, the term "keypoint" refers to a region of an image that is particularly distinct and identifies a unique feature. Keypoints are used to identify key regions of an object that are later used as a basis for matching and identifying it in another image.

[0045] As used in this Summary, the following description, the following claims, and the accompanying drawings, the term "keypoint descriptor" or "local descriptor" refers to an image patch around a keypoint as a high-dimensional point in feature space. A keypoint descriptor or local descriptor contains edge and / or color information that is invariant to small affine transformations and maintains spatial relationships. A keypoint descriptor or local descriptor may also contain shape, texture, and / or semantic information.

[0046] As used in this Summary, the following description, the claims below, and the accompanying drawings, the term "feature" refers to a variable, attribute, trait, or characteristic within a dataset, and the terms "feature map," "activation map," or "convolutional feature" can refer to the set of features output by a particular layer of a neural network after a filter (also known as a kernel or feature detector) containing a vector of weights and biases applied to the input dataset.

[0047] These and other features, aspects, and advantages will become better understood with regard to the following description, appended claims, and accompanying drawings. [Brief explanation of the drawings]

[0048] [Figure 1] 1 is a schematic diagram of a system for associating objects across two or more images according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a schematic diagram of a method for associating objects across two images according to an embodiment of the present disclosure. [Figure 3] 1 illustrates an example of corresponding first and second images according to an embodiment of the present disclosure. [Figure 4] 10 illustrates an example of identifying and matching corresponding points between an exemplary first image and a second image according to an embodiment of the present disclosure. [Figure 5]1 is a schematic diagram of an exemplary method for associating each of one or more identified objects in a first image with a corresponding object in a second image, according to an embodiment of the present disclosure. [Figure 6] 1 is a schematic diagram of a method for associating each first bounding box with a corresponding second bounding box according to an embodiment of the present disclosure. [Figure 7] 10 illustrates exemplary first and second images after projection, according to an embodiment of the present disclosure. [Figure 8] FIG. 1 is a schematic diagram of a top view of a vehicle equipped with an image sensor, according to an embodiment of the present disclosure. [Figure 9] FIG. 1 is a schematic diagram of the architecture of a trained neural network for identifying corresponding points, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0049] In the figures, like parts are given like reference numerals.

[0050] Those skilled in the art should understand that any block diagrams herein represent conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, any flowcharts, flow diagrams, state transition diagrams, pseudocode, etc. will be understood to represent various processes that may be substantially represented on a computer-readable medium and executed by a computer or processor, whether or not such a computer or processor is explicitly shown.

[0051] In the above summary, reference is made in the description, the claims that follow, and the accompanying drawings to specific features of the invention (including method steps). It is to be understood that the disclosure of the invention herein includes all possible combinations of such specific features. For example, if a particular feature is disclosed in a particular aspect or embodiment of the invention, or in the context of a particular claim, that feature can also be used, to the extent possible, in combination with and / or in the context of other specific aspects and embodiments of the invention, and in the invention generally.

[0052] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0053] While the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are described in detail below. It is to be understood, however, that it is not intended to limit the disclosure to the disclosed forms, but on the contrary, the disclosure is intended to cover all modifications, equivalents, and alternatives falling within the scope of the present disclosure.

[0054] 1 is a schematic diagram of a system for associating objects across two or more images according to an embodiment of the present disclosure. The system 100 for associating objects across two or more images may include a first image sensor 104, a second image sensor 108, one or more processors 112, and one or more displays 116. Although only the first image sensor 104 and the second image sensor 108 are shown, in some embodiments, the system 100 may include three or more image sensors.

[0055] According to some embodiments, the first image sensor 104 and the second image sensor 108 may be visible light sensors that capture information about the color of objects in a scene. In some embodiments, the first image sensor 104 and / or the second image sensor 108 may be cameras or video cameras. In some embodiments, the first image sensor 104 may be operable to provide at least one first image, and the second image sensor 108 may be operable to provide at least one second image. In some embodiments, the first image sensor 104 and / or the second image sensor 108 may be image sensors, cameras, and / or video cameras with standard lenses. Preferably, the first image sensor 104 and / or the second image sensor 108 may be fisheye cameras, which are image sensors, cameras, and / or video cameras with fisheye or wide-angle lenses having a field of view of 60 degrees or more.

[0056] According to some embodiments, the first image sensor 104 may be positioned to capture a scene from a first direction, and the second image sensor 108 may be positioned to capture the same scene from a second direction, with the first image sensor 104 and the second image sensor 108 positioned such that there is overlap between at least one first image of the scene captured by the first image sensor 104 and at least one second image of the scene captured by the second image sensor 108. Generally, the larger the overlap area, the more accurate the results, and the first image sensor 104 and the second image sensor 108 can be adjusted and customized based on the user's desired accuracy and desired scene coverage. For example, if the overlap exceeds 50% of the region of interest, distortion in the image plane can be mathematically modeled with the methods of the present disclosure at lower scene coverage and higher accuracy. For example, if the overlap is less than 50% of the region of interest, distortion can be mathematically modeled with the methods of the present disclosure at higher scene coverage and lower accuracy. In some embodiments, the scene may be a scene surrounding a vehicle, a scene along a hallway, a scene surrounding a building, a scene within a room, or any other scene where identifying and / or tracking objects, people, or participants may be useful. The first image sensor 104 and the second image sensor 108 may be mounted anywhere, at any position, and at any height, depending on the scene they are used to capture. In some embodiments, the first image sensor 104 and the second image sensor 108 may be mounted on a vehicle and positioned to capture a scene around the vehicle. In some embodiments, the first image sensor 104 and the second image sensor 108 may be mounted on the exterior of a building and positioned to capture a scene around the building. In some embodiments, the first image sensor 104 and the second image sensor 108 may be mounted along a hallway and positioned to capture a scene in the hallway.

[0057] According to some embodiments, one or more processors 112 may be coupled to the first image sensor 104 and the second image sensor 108 to receive at least one image captured by the first image sensor 104 and at least one second image captured by the second image sensor 108. The one or more processors 112 may be operable to identify and associate one or more objects found in both the at least one first image and the at least one second image. Methods of association are described in more detail below.

[0058] According to some embodiments, the one or more processors 112 may be coupled to one or more displays 116. In some embodiments, the one or more displays 116 may display at least one first image captured by the first image sensor 104 and / or at least one second image captured by the second image sensor 108 received by the one or more processors 112. In some embodiments, the one or more displays 116 may display markings, shading, or any other indicators generated by the one or more processors 112. Examples of such markings or indicators may include tracking device identification labels, bounding boxes, projected bounding boxes, keypoints, and points.

[0059] 2 is a schematic diagram of a method for associating objects across two images according to an embodiment of the present disclosure. While the present disclosure discusses the method in the context of two images, the method may be scaled to associate objects across three or more images, as long as there is an overlapping area between the images. The method 200 for associating objects across two images can be implemented by any architecture and / or computing system. For example, various architectures using, for example, multiple integrated circuit (IC) chips and / or packages, and / or various computing and / or consumer electronics (CE) devices, such as multifunction devices, tablets, smartphones, etc., can implement the techniques and / or arrangements described herein.

[0060] According to some embodiments, a method 200 for correlating objects across two images may begin at operation 204, where a first image is received from a first image sensor 104 and a second image is received from a second image sensor 108. Preferably, the first image and the second image are images taken over different views of the same scene. Preferably, there is overlap between the first image and the second image, with overlapping or common areas captured in both the first image and the second image. The overlapping area may be any percentage of the first image and / or the second image. Preferably, the overlapping area is greater than 50% of the first image and / or the second image, although the method has acceptable accuracy even with overlapping areas less than 50%. In some embodiments, the first image and the second image may be taken consecutively by the first image sensor 104 and the second image sensor 108. Preferably, the first image and the second image are taken simultaneously by the first image sensor 104 and the second image sensor 108. Preferably, the first image and the second image have the same timestamp.

[0061] 3 illustrates an example of a first image 304 and an example of a second image 308, according to an embodiment of the present disclosure. As illustrated, the first image 304 and the second image 308 comprise image content on a fisheye image plane. In other embodiments, the first image 304 and the second image 308 may comprise image content on a rectilinear image plane. As shown in FIG. 3, the first image 304 and the second image 308 cover different views of the same scene, and an overlap region 312 is captured in both the first image 304 and the second image 308.

[0062] Returning to FIG. 2 , method 200 may include operation 208, in which corresponding points between the first image and the second image are identified. Corresponding points are image points that exist or are found in both the first image captured by the first image sensor 104 and the second image captured by the second image sensor 108. In some embodiments, each identified corresponding point may include image point data, which may include any suitable data or data structure indicative of the identified image point, such as a location and / or a point vector for each identified point. A point vector may include any data structure, such as a vector of values ​​indicative of characteristics of a particular point (e.g., measurements of various parameters characteristic of the point). For example, a point location may be a pixel location.

[0063] According to some embodiments, operation 208 may include identifying image points in the first image and the second image and matching the image points in the first image and the second image to identify corresponding image points. In some embodiments, the identified image points may generally be geometrically invariant, i.e., invariant to image translation, rotation, and scale, and photometrically invariant, i.e., invariant to changes in brightness, contrast, and color, which may facilitate easier identification and matching of corresponding points in the first image and the second image. In some embodiments, the identified image points may be points corresponding to vertices or intersections, which may facilitate easier identification and matching of corresponding points in the first image and the second image. In some embodiments, the identified image points may be distinct points, also known as keypoints, or pixels of highly distinct or distinguishable visual features.

[0064] According to some embodiments, identifying image points can be performed manually or using any suitable technique(s) to detect or identify suitable image-based features, such as image-based features detected based on features extracted using image information such as pixel values. Examples of methods that can be used to identify image points include, but are not limited to, the method for extracting distinctive invariant features disclosed in "Distinctive Image Features from Scale-Invariant Keypoints" by David G. Lowe, SURF (Speeded Up Robust Features) disclosed by Bay et al., BRISK (Binary Robust Invariant Scalable Keypoints) disclosed by Leutenegger et al., BRISK (Binary Robust Invariant Scalable Keypoints) disclosed by Leutenegger et al., and ORB (Oriented FAST and Rotated BRIEF) disclosed by Rublee et al., "ORB: An Efficient Alternative to SIFT or SURF."

[0065] According to some embodiments, image points defined in a first image can be compared with image points identified in a second image to identify corresponding or matching points found in both the first and second images. Point comparison and matching can be performed using any known image analysis method for cross-matching purposes, such as similarity-based or template matching. Comparison and cross-matching can be repeated between each possible pair of points until all identified points have been processed. An example of an algorithm that can be used is the Random Sample Consensus (RANSAC) algorithm disclosed by Fischler, AM and Bolles, RC in "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography."

[0066] According to some embodiments, operation 208 of identifying corresponding points between the first and second images may be performed using a neural network trained to use features in the images to identify and match keypoints. An example of the architecture and training of a neural network to identify corresponding points is described in further detail in connection with FIG. 9.

[0067] 4 illustrates an example of identified and matched corresponding points between an exemplary first image 304 and a second image 308, according to an embodiment of the present disclosure. As illustrated, a plurality of image points 404 are identified in the first image 304, and a plurality of image points 408 are identified in the second image 308. In FIG. 4, each image point 404 or 408 is illustrated with a dot representing a pixel location in the first image 304 or the second image 308. Each image point 404 identified in the first image 304 may match a corresponding image point 408 in the second image 308. As illustrated, the image points identified in the first image 304 and the second image 308 are distinct points, such as corners of a sign, a corner of a building, a corner of a road marking, or a corner of a truck.

[0068] 2, method 200 may include act 212, in which a projection matrix is ​​approximated between the first image and the second image based on the corresponding points identified in act 208. The projection matrix represents a geometric and / or transformation relationship between points in the first image and corresponding points in the second image.

[0069] According to some embodiments, approximating the projection matrix preferably comprises approximating at least one polynomial, which is an algebraic equation with the general formula: P(x)=a n x n +a n-1 x n-1 +...+a2x 2 +a1x+a0(1) In the formula, a0, ..., a n are the coefficients of the polynomial, x is an indefinite value that can be replaced by any value, and the exponent n on the indefinite value x can be any integer representing the degree of the indefinite value x.

[0070] According to some embodiments, the first image and / or the second image may be a fisheye image. In such embodiments, the geometric relationship between the first image and the second image is nonlinear, and the polynomial may have a degree n, and the number of identified corresponding points may be between n+1 and n+7, where n is an integer greater than or equal to 2. In some embodiments, the polynomial may have a degree 2, and the number of identified corresponding points may be between 3 and 9 to reduce the required computational power while maintaining sufficient accuracy without overfitting. In some embodiments, the polynomial may have a degree independent of the number of corresponding points.

[0071] According to some embodiments, the number of corresponding points used to approximate the projection matrix in operation 212 may be less than the number of corresponding points identified in operation 208. In some embodiments, the corresponding points used to approximate the projection matrix in operation 212 may be selected in a distributed manner to cover the largest area within the overlap between the first image and the second image.

[0072] According to some embodiments, the projection matrix may include two polynomials: a first polynomial for coordinates on the x-axis and a second polynomial for coordinates on the y-axis. The position or location of each pixel or point on an image may be represented as two-dimensional (2D) coordinates that may be expressed as (x, y), where x represents the x-coordinate and y represents the y-coordinate. In some embodiments, the x-coordinate may be transformed based on the first polynomial, and the y-coordinate may be transformed based on the second polynomial.

[0073] For example, a projection matrix containing two polynomials is It can be expressed as TIFF2025533515000002.tif10170, where P(x) represents the first polynomial in the coordinates on the x-axis, P(y) represents the second polynomial in the coordinates on the y-axis, and a0,...,a n are the coefficients of the first polynomial, b0,...,b n are the coefficients of the second polynomial, x is an indefinite value of the first polynomial that can be replaced by an x ​​coordinate, y is an indefinite value of the second polynomial that can be replaced by a y coordinate, and the exponent n on the indefinite x or y can be any integer representing the degree of the indefinite x or y.

[0074] According to some embodiments, the coefficients a0,...,a0 of at least one polynomial n can be determined by any known curve fitting method that identifies a best-fit polynomial to a set of data points. One example of a known method is the 2D polynomial transformation function in the skimage library in Python.

[0075] According to some embodiments, method 200 may include operation 216, in which each of the at least one identified object in the first image is associated with a corresponding identified object in the second image based on the projection matrix approximated in operation 212. Each of the one or more identified objects in the first image may be associated with a corresponding identified object in the second image using any known object matching method. The association may be based on location using the approximated projection matrix, and may further be based on visual appearance. An example of a method for associating each of the one or more identified objects in the first image with a corresponding identified object in the second image is described in connection with FIG. 5.

[0076] According to some embodiments, method 200 may optionally include operation 220, in which the positions of identified objects in the first and second images are plotted on a common image plane. In some embodiments, one common image plane may cover a 360° surround view of the object around the first, second, and additional image sensors. This plotting of the positions of identified objects in the common image plane allows for accurate plotting of the object's position in the scene. The common image plane may be the image plane of the first image, the image plane of the second image, or a separate image plane. Preferably, the plotting is performed by projecting four corner points of each object and / or bounding box using approximate polynomials derived for the x- and y-axes. In some embodiments, any number of image sensors may be implemented, as long as pairs of adjacent image sensors have overlapping fields of view, and objects in adjacent image sensors may be associated with each other, tracked across multiple cameras, and tracked across a time sequence.

[0077] 5 is a schematic diagram of an exemplary method for associating each of at least one identified object in a first image with a corresponding object in a second image according to embodiments of the present disclosure. In some embodiments, method 500 for associating each of at least one identified object in a first image with a corresponding object in a second image may be used at operation 216 of method 200. Method 500 for associating each of at least one identified object in a first image with a corresponding object in a second image may begin at operation 504, where at least one object is identified in the first image and at least one object is identified in the second image. The at least one object may be identified using any known object detection or image instance segmentation method.

[0078] According to some embodiments, the method 500 may include operation 508, in which at least one first bounding box is generated in the first image and at least one second bounding box is generated in the second image. The bounding boxes represent the spatial location of the identified object. The object may be identified, and the bounding boxes may be defined using any known object detection method, such as a convolutional neural network. A CNN is a multi-layer feedforward neural network, built by stacking multiple hidden layers on top of each other in sequence. The sequential design may allow the CNN to learn hierarchical features. The hidden layers are typically convolutional layers followed by activation layers, some of which are followed by pooling layers. A CNN may be configured to identify patterns in the data. The convolutional layers may include convolutional kernels used to search for patterns across the input data. The convolutional kernel may return a large positive value for a portion of the input data that matches the kernel's pattern, or a smaller value for another portion of the input data that does not match the kernel's pattern. CNNs are preferred because they can extract useful features from training data without requiring manual processing of the training data. CNNs can produce accurate results when large unstructured data sets are involved, such as in image classification, speech recognition, and natural language processing. CNNs are also computationally efficient because they can assemble patterns of increasing complexity using relatively small kernels in each hidden layer. CNNs are also advantageous because they have high accuracy in image recognition and can automatically filter images and detect features without human supervision. An example of a convolutional neural network is YOLOv4. An example of YOLOv4 can be found in "YOLOv4: Optimal Speed ​​and Accuracy of Object Detection" by Bochkovskiy et al., where an example of the YOLOv4 architecture can be found at least in Section 3 and an example of training YOLOv4 can be found at least in Section 4.1.In some embodiments where the first image and / or the second image are fisheye images, the convolutional filter of the convolutional network may be a restricted deformable convolution (or RDC) module to effectively model the geometric transformations present in fisheye images, where the transformations are conditioned on input features in a local, dense, and adaptive manner. The shape of the RDC is learned to adapt to changes in the features, and the kernel shape adapts to unknown complex transformations in the input. In particular, the RDC module learns a sampling matrix with position offsets, where the offsets are learned from preceding feature maps via additional convolutional layers. More information regarding the RDC module can be found in "Restricted Deformable Convolution Based Road Scene Semantic Segmentation Using Surround View Cameras" by Deng et al.

[0079] According to some embodiments, each bounding box detected in an image can be assigned a first tracking device identifier (ID) using any known tracking device, such as the MOTDT tracking device disclosed in "Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-identification" by Chen et al. The first tracking device identifier can be a local tracking ID, which is a tracking ID associated with an object identified in an image captured by a respective image sensor.

[0080] According to some embodiments, method 500 may include operation 512, in which each first bounding box in the first image is associated with a corresponding second bounding box in the second image. In other words, each first bounding box in the first image is associated with a second bounding box in the second image, and the first bounding box and its corresponding second bounding box represent the spatial location of the same object seen in the first image captured by the first image sensor and the second image captured by the second image sensor. In some embodiments, each first bounding box in the first image may be assigned a second track identifier (ID), and its associated corresponding second bounding box in the second image may be assigned the same second track identifier (ID). The second track identifier may be a global track ID, which is a track ID associated with identified objects in the scene. This global track ID may be used to label the identified objects when their positions are subsequently plotted on a common image plane.

[0081] According to some embodiments, associating each first bounding box with a corresponding second bounding box may be based on a relationship between at least a projection of the first bounding box and the corresponding second bounding box, where the projection of the first bounding box is based on an approximated projection matrix, as further detailed in connection with FIG. 6. According to some embodiments, associating each first bounding box with a corresponding second bounding box may further include identifying at least one feature of each identified object in the first image and at least one feature of each identified object in the second image, and determining a visual similarity or dissimilarity between the objects contained within the bounding boxes by comparing the at least one feature of each identified object in the first image with the at least one feature of each identified object in the second image. This may be advantageous because it improves the accuracy of results in situations where there is little overlap between the first and second images.

[0082] 6 is a schematic diagram of a method for associating each first bounding box with a corresponding second bounding box according to an embodiment of the present disclosure. In some embodiments, method 600 for associating each first bounding box with a corresponding second bounding box may be used at operation 512 of method 500. Method 600 for associating each first bounding box with a corresponding second bounding box may begin at operation 604, in which at least one first bounding box from a first image is projected onto a second image to form at least one projected bounding box, each projected first bounding box corresponding to one of the at least one first bounding box. The projection may be based on the approximated projection matrix at operation 212 of method 200, and the position or coordinates of each first bounding box may be transformed using the approximated projection matrix and then projected onto the second image.

[0083] FIG. 7 illustrates exemplary first and second images 304 and 308 after projection according to operation 604 of method 600, in accordance with an embodiment of the present disclosure. As shown in FIG. 7, the first image 304 includes four first bounding boxes 704a-704d, and the second image 308 includes four second bounding boxes 708a-708d, each representing the spatial location of a person present in the overlap region 312 of the first and second images 304 and 308. The first and second bounding boxes 704a-704d and 708a-708d are each assigned a track ID. As shown in the first image 304 of FIG. 7, the first bounding box 704a is assigned track ID 5, the first bounding box 704b is assigned track ID 6, the first bounding box 704c is assigned track ID 7, and the first bounding box 704d is assigned track ID 8. As shown in second image 308 in Figure 7, second bounding box 708a is assigned track ID 1, second bounding box 708b is assigned track ID 2, second bounding box 708c is assigned track ID 3, and second bounding box 708d is assigned track ID 4. As shown in Figure 7, each of first bounding boxes 704a-704d is projected onto second image 308 as projected first bounding boxes 704a'-704d'. Each of the projected first bounding boxes 704a'-704d' corresponds to a first bounding box 704. The projected first bounding box 704a' corresponds to the first bounding box 704a, the projected first bounding box 704b' corresponds to the first bounding box 704b, the projected first bounding box 704c' corresponds to the first bounding box 704c, and the projected first bounding box 704d' corresponds to the first bounding box 704d. As shown in Figure 7, each of the projected first bounding boxes 704a'-704d' corresponds to a first bounding box 704a-704d, but the size and relative position of the bounding boxes may or may not be identical depending on the projection matrix.In the example shown in FIG. 7, because both the first image 304 and the second image 308 are fisheye images, the geometric relationship between the pixels of the first image 304 and the second image 308 is nonlinear, and therefore each projected first bounding box 704a′-704d′ may not be identical in size to the corresponding first bounding box 704a-704d.

[0084] Returning to FIG. 6, method 600 may include operation 608, in which a relationship between each projected first bounding box and each second bounding box is calculated. Each determined relationship is a cost for each projected first and second bounding box pair. In the example shown in FIG. 7, a separate relationship or cost may be determined between projected bounding box 704a' and each of second bounding boxes 708a-708d, as well as a separate relationship or cost may be determined between projected first bounding box 704b', 704c', and 704d' and each of second bounding boxes 708a-708d.

[0085] In some embodiments, the relationship between each projected first bounding box and each second bounding box may include the degree of overlap between each projected first bounding box and each second bounding box. In some embodiments, the degree of overlap between each projected first bounding box and each second bounding box may be determined based on an intersection-over-union (IOU) evaluation matrix, which is calculated by dividing the area of ​​overlap between the projected first bounding box and the second bounding box by the area of ​​union, i.e., the area encompassed by both the projected first bounding box and the second bounding box. The higher the IOU between the projected first bounding box and the second bounding box, the more likely it is that the first bounding boxes corresponding to the projected first bounding box and the second bounding box encompass the same object in the scene, and the lower the cost.

[0086] In some embodiments, the relationship between each projected first bounding box and each second bounding box may include a distance between each projected first bounding box and each second bounding box. Preferably, the distance is between the center of each projected first bounding box and the center of each second bounding box. In some embodiments, a Euclidean distance may be calculated between each projected first bounding box and each second bounding box. The shorter the Euclidean distance between the projected first bounding box and the second bounding box, the more likely it is that the first bounding boxes corresponding to the projected first bounding box and the second bounding box contain the same object in the scene, and the lower the cost.

[0087] According to some embodiments, method 600 may include operation 612, in which each first bounding box is associated with a corresponding second bounding box based on the determined relationship between the corresponding projected first bounding box and the corresponding second bounding box. In some embodiments, the determined relationship or cost may be arranged in a cost matrix that can be used for association in subsequent operations. For example, the cost matrix may be a two-dimensional matrix with one dimension being the projected first bounding box and the second dimension being the second bounding box. All projected first and second bounding box pairs include a relationship or cost included in the cost matrix. The best match between the projected first and second bounding boxes may be determined by identifying the projected first and second bounding box pair with the smallest cost in the matrix. In some embodiments, the association determined using the Hungarian algorithm, also known as the Kuhn-Munkres algorithm, is a combinatorial optimization algorithm that can optimize a global cost over all projected first bounding boxes with second bounding boxes to minimize the global cost. The combination of the projected first and second bounding boxes in the cost matrix that minimizes the global cost can be determined and used as the association. Any other method can also be used to associate each projected first bounding box with each second bounding box.

[0088] According to embodiments of the present disclosure, operation 612 of associating each first bounding box with a corresponding second bounding box may further include identifying at least one feature of each identified object in the first image and the second image and comparing at least one feature of each identified object in the first image with at least one feature of each identified object in the second image. The identification and comparison of features of the identified objects may enable improved accuracy of results because the visual similarity of the objects in the first bounding box and its associated second bounding box is also taken into account. In some embodiments, the identification and comparison of features may be performed separately using any known algorithm or machine learning method. For example, the identification or extraction of at least one feature of the identified objects in the first image and the second image may be performed using any known feature identification algorithm or machine learning method, such as a pre-trained convolutional neural network (CNN), and a cost may be calculated based on how similar the features appear. In some embodiments in which the first image and / or the second image are fisheye images, the convolution filter of the CNN may include the RDC module described above. In some embodiments, feature identification and comparison can be performed using any known generative model-based re-identification. In some embodiments, feature identification and comparison can be performed using a Siamese network, which is a class of neural networks that includes one or more identical networks. The identified object pairs are fed into one or more identical networks. The one or more identical networks calculate the features of one input object, and the similarity of the features is calculated using their difference or dot product.

[0089] 8 is a schematic diagram of an example of an arrangement of image sensors mounted on a vehicle, according to embodiments of the present disclosure. According to some embodiments, system 100 can be adapted for use in a vehicle including at least a first image sensor and a second image sensor, one or more processors, and a memory storing executable instructions for execution by the one or more processors, the executable instructions including instructions for performing method 200. In some embodiments, first image sensor 804 and second image sensor 808 may be mounted on vehicle 800 such that first image sensor 804 and second image sensor 808 capture images of a scene surrounding vehicle 800. For example, as shown in FIG. 8 , first image sensor 804 may be mounted in front of vehicle 800 such that first image sensor 804 captures images of a scene in front of vehicle 800 within a first field of view (FOV) 812. 8 , a second image sensor 808 may be mounted to the right of the vehicle 800 such that the second image sensor 808 captures images of a scene to the right of the vehicle 800 within a second field of view (FOV) 816. The first image sensor 804 and the second image sensor 808 are positioned such that there is an overlap region 820 between the first FOV 812 of the first image sensor 804 and the second FOV 816 of the second image sensor 808. This overlap region 820 allows association of objects in a first image received from the first image sensor 804 with corresponding objects in a second image received from the second image sensor 808.

[0090] According to some embodiments, system 100 and method 200 may be adapted to associate objects across three or more images by including more image sensors, as long as the fields of view of the additional image sensors overlap with at least one other image sensor. In some embodiments, method 200 may be adapted such that step 220 includes plotting the positions of the identified objects on a common image plane, which preferably covers a 360° surround view of the objects around the vehicle. For example, vehicle-mounted system 100 may further include a third image sensor 824 and a fourth image sensor 828. For example, as shown in FIG. 8 , third image sensor 824 may be mounted on the left of vehicle 800 such that third image sensor 824 captures images of a scene to the left of vehicle 800 within a third field of view (FOV) 832. Third image sensor 824 may be positioned such that there is an overlap region 836 between first FOV 812 of first image sensor 804 and third FOV 832 of third image sensor 824. 8 , a fourth image sensor 840 may be mounted to the rear of the vehicle 800 such that the fourth image sensor 840 captures images of the scene behind the vehicle 800 within a fourth field of view (FOV) 844. The fourth image sensor 840 may be positioned such that there is an overlap region 848 between a third FOV 832 of the third image sensor 824 and the fourth FOV 844 of the fourth image sensor 840, and such that there is an overlap region 852 between a second FOV 816 of the second image sensor 808 and the fourth FOV 844 of the fourth image sensor 840. In this example, method 200 may be applied between images received from the first image sensor 804 and the second image sensor 808, between images received from the first image sensor 804 and the third image sensor 816, between images received from the third image sensor 816 and the fourth image sensor 840, and between images received from the second image sensor 808 and the fourth image sensor. In this example, one common plane on which the positions of identified objects are plotted covers a 360° surround view of objects around vehicle 800.

[0091] 9 is a schematic diagram of a trained neural network architecture for identifying corresponding points, according to embodiments of the present disclosure. In some embodiments, the trained neural network 900 may be used in operation 208 of the method 200. It is emphasized that the architecture shown in FIG. 9 is an example, and other suitable architectures may be used depending on the user's application and requirements. In some embodiments, the trained neural network 900 may be used to identify keypoints in an input image 904. The neural network may include input nodes (i.e., layers), hidden nodes, and output nodes.

[0092] According to some embodiments, the neural network 900 can be trained on a scene classification dataset. In some embodiments, the scene classification dataset can include images of different scenes captured from different cameras mounted on the ego-vehicle. In some embodiments, the scene classification dataset can include images sorted into scene classes. For example, the scene classification dataset can include images sorted into at least 500 classes, with each class including at least 70 images. In some embodiments, the number of classes can be based on the potential environments in which the system 100 may be used. In some embodiments, a single class of scene representing the same scene content can include a series of images captured from image sensors capturing overlapping regions of the scene and can further include a series of images captured during the same period by other image sensors having a field of view that overlaps with the first-mentioned image sensor. In some embodiments, a single class of scene can further include images that have been modified from the aforementioned images using image scaling and random cropping to increase the number of images in the class, as long as the overlapping regions are preserved. In some embodiments, when training the neural network 900, pairs of images from the same class can be designated as matched pairs, and pairs of images from different classes can be designated as non-matched pairs.

[0093] According to some embodiments, the trained neural network 900 may include a convolutional neural network (CNN) backbone 908 for extracting features and a keypoint extractor 916 for identifying keypoints from the features extracted by the CNN backbone 908. In some embodiments, the trained neural network 900 may be trained with a batch size of 16 for 25 epochs by optimization using stochastic gradient descent (SGD) with a momentum of 0.9. In some embodiments, a decaying learning rate may be used until the rate reaches zero, and the learning rate may be initialized between 0.0003 and 0.01. In some embodiments, the training of the keypoint extractor 916 and the CNN backbone 908 may be performed independently to learn stronger local descriptors. In some embodiments, the CNN backbone 908 may be trained and refined first, and the keypoint extractor 916 may be trained thereafter, so that only the weights of the keypoint extractor 916 are learned without disturbing the trained and refined CNN backbone 908.

[0094] Examples of CNN backbone 908 include ResNet-50 and ResNet-101, which can be found in "Deep Residual Learning for Image Recognition," whose architectures can be found at least in Section 3, Table 1, and FIG. 5, and whose training can be found at least in Section 3.4. Example architectures of ResNet-50 and ResNet-101 are reproduced below in Table 1. In some embodiments in which the first image and / or the second image are fisheye images, the convolutional filter of CNN backbone 908 can include the RDC module described above. TIFF2025533515000003.tif107170

[0095] According to some embodiments, the CNN backbone 908 may include one or more shallow layers 924 and one or more deep layers 930. For example, if the CNN backbone 908 is ResNet-50 or ResNet-101, layers conv1, conv2_x, conv3_x, and conv4_x may be shallow layers 924, and layer conv5_x may be deep layer 930. In some embodiments, outputs from the shallow layers 924 may be input to a keypoint extractor 916 to identify a subset of features from the densely extracted features extracted by the shallow layers 924 of the CNN backbone 908 as keypoints. For example, if the CNN backbone 908 is ResNet-50 or ResNet-101, outputs from layer conv4_x may be input to the keypoint extractor 916 to identify keypoints. In some embodiments, the output from the shallow layers 924 can include 1024 channels and can be represented as 14×14×1024, meaning that each element in the 14×14 matrix has 1024 feature vectors. In some embodiments, the 196 elements in the 14×14 matrix can correspond to 196 overlapping patches (or local descriptors) from a 512×512 input image, such that each patch can have a 1024-size feature vector or local descriptor. The output from the shallow layers 924 is preferably used to extract keypoints because the feature map generated by the shallow layers 924 contains information about local features within the input image, compared to the output from the deep layers 930, which contains information about global features that describe the entire image. The deeper the layer, the more abstract and high-level the semantic features. The shallow layers 924 generate shallow feature maps that represent low-level local features including descriptors and geometric information about specific image regions such as objects, while the deep layers 930 generate deep feature maps that represent global features that summarize the content of the image but do not include information about the spatial arrangement of visual elements.

[0096] According to some embodiments, the CNN backbone 908 can be pre-trained on a labeled image classification and localization dataset, such as the ImageNet dataset available at https: / / image-net.org / download.php. Dense features can be extracted from the input image 904 by applying a fully convolutional network (FCN) obtained from ResNet-50 using the output of the conv4_x convolution block of ResNet-50. To handle scale changes, an image pyramid can be constructed, and FCN can be explicitly applied at each level. The resulting feature map can be viewed as a dense grid of local descriptors, and features can be localized based on their receptive fields, which can be calculated based on the configuration of the FCN's convolutional and pooling layers. In some embodiments, the pixel locations of features can be determined by taking the centers of their receptive fields. In some embodiments, to improve the discriminability of the local descriptors, the ResNet-50 model can be fine-tuned for scene classification by training it with the scene classification dataset described above and using cross-entropy loss, so that the local descriptors learn a better representation of the scene without using object-level and patch-level labels.

[0097] According to some embodiments, the keypoint extractor 916 may receive as input an output from the shallow layers 924 of the CNN backbone 908. In some embodiments, the keypoint extractor 916 may comprise an attention model 936 that selects a subset of features from the densely extracted features from the shallow layers 924 of the CNN backbone 908, and a dimension reducer 942 that reduces the dimensionality of the features. In some embodiments, the dimension reducer 942 may comprise a PCA (principal component analysis) or an autoencoder. In some embodiments, the dimension reducer 942 may generate local descriptors for the keypoints of the subset of features selected by the attention model 936.

[0098] According to some embodiments, the keypoint extractor 916 may comprise PCA and L2 normalization as the dimensionality reducer 942. In particular, the dimensions of the feature maps may be L2 normalized, where their dimensions are reduced from 1024 to 40 by PCA, followed by an additional L2 normalization.

[0099] According to some embodiments, the keypoint extractor 916 may comprise an autoencoder as the dimensionality reducer 942. In some embodiments, the autoencoder may reduce the dimensionality of the feature maps from 1024 to 128. In some embodiments, the autoencoder may be a convolutional autoencoder module that learns an appropriate low-dimensional representation of the feature maps. In some embodiments, the autoencoder uses a cross-entropy classification loss L with weight λ=10. r can be trained in.

[0100] According to some embodiments, the attention model 936 can be used to predict which densely extracted features are discriminative for objects in an image by learning the weighting of the local descriptors. In some embodiments, the attention model 936 can determine which features are discriminative based on keypoint detection scores. In some embodiments, the attention model 936 can remove redundant local descriptors based on the relevance of the local descriptors, such that the remaining local descriptors are keypoint descriptors 950, which correspond to keypoints identified in the image. In some embodiments, the attention model 936 can include two convolutional layers without striding, using convolutional filters of size 1×1, with ReLU as the activation function for the first convolutional layer and softplus as the activation function for the second convolutional layer. In some embodiments, the attention model can be trained using a weighted sum of the local descriptors, followed by a standard softmax cross-entropy loss with weight β=1. In some embodiments, the attention pool can be used with the keypoint descriptors 950 to determine the exact location of the identified keypoints in the image.

[0101] The illustrated steps are described to explain the illustrated exemplary embodiments, and ongoing technological developments should be expected to change the manner in which particular functions are performed. These examples are presented herein for purposes of illustration and not limitation. Moreover, functional component boundaries are arbitrarily defined herein for convenience of description. Alternative boundaries may be defined so long as the specified functions and relationships thereof are appropriately performed. Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent to those skilled in the art based on the teachings contained herein. Such alternatives are within the scope and spirit of the disclosed embodiments. The terms "comprises," "comprising," "includes," or any other variation thereof, are intended to cover non-exclusive inclusions, such that a configuration, device, or method that includes a list of components or steps does not include only those components or steps, but may also include other components or steps not expressly listed or inherent to such configuration, device, or method. In other words, one or more elements in a system or apparatus proceeded by "comprises...a" does not, without further constraints, exclude the presence of other or additional elements in the system or method. It should also be noted that as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise.

[0102] Finally, the language used herein has been chosen primarily for ease of reading and descriptive purposes, and may not be chosen to delineate or limit the subject matter of the invention. Accordingly, it is intended that the scope of the invention be limited not by this detailed description, but rather by any claims that issue in an application based thereon. Accordingly, the embodiments of the invention are intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.

Claims

1. 1. A vehicle comprising a vehicle system for associating objects across two or more images, the vehicle system comprising at least a first image sensor (104) and a second image sensor (108), one or more processors, and a memory storing executable instructions for execution by the one or more processors, the executable instructions comprising: receiving a first image from the first image sensor (104) and a second image from the second image sensor (108); identifying corresponding points between the first image and the second image; approximating a projection matrix between the first image and the second image based on the identified corresponding points; associating at least one identified object in the first image with a corresponding identified object in the second image based at least on the approximated projection matrix; comprising instructions for performing a computer-implemented method comprising: vehicle.

2. the one or more processors; and a memory storing executable instructions for execution by the one or more processors, the executable instructions comprising: receiving a first image from the first image sensor (104) and a second image from the second image sensor (108); identifying corresponding points between the first image and the second image; approximating a projection matrix between the first image and the second image based on the identified corresponding points; associating at least one identified object in the first image with a corresponding identified object in the second image based at least on the approximated projection matrix; - plotting the positions of the identified objects on one common image plane, said one common image plane preferably covering a 360° surround view of objects around said vehicle; 10. The vehicle of claim 1, comprising instructions for performing a computer-implemented method comprising:

3. 1. A computer-implemented method for associating objects across two or more images, comprising: receiving a first image from a first image sensor (104) and a second image from a second image sensor (108); identifying corresponding points between the first image and the second image; approximating a projection matrix between the first image and the second image based on the identified corresponding points; associating at least one identified object in the first image with a corresponding identified object in the second image based at least on the approximated projection matrix; 11. A computer-implemented method comprising:

4. 4. The computer-implemented method of claim 3, wherein the first image sensor (104) and / or the second image sensor (108) are fisheye cameras, and / or the first image sensor and the second image sensor are positioned such that there is an overlap between at least one first image of a scene captured by the first image sensor and at least one second image of a scene captured by the second image sensor.

5. said corresponding points are distinct points, preferably points corresponding to vertices or intersections; and / or The computer-implemented method of claim 3 or 4, wherein the corresponding points are keypoints.

6. 6. The computer-implemented method of any one of claims 3 to 5, wherein keypoints are selected using a neural network, preferably a neural network comprising a convolutional neural network backbone and a keypoint extractor, said convolutional neural network backbone preferably comprising a constrained deformable convolution module.

7. 7. The computer-implemented method of claim 3, wherein approximating a projection matrix comprises approximating at least one projection matrix comprising at least one polynomial, preferably wherein the at least one polynomial has a degree n and the number of identified corresponding points is between n+1 and n+7, where n is an integer greater than or equal to 2, and / or preferably wherein the at least one polynomial has a degree 2 and the number of identified corresponding points is between 3 and 9.

8. 8. The computer-implemented method of claim 3, wherein the projection matrix comprises two polynomials: a first polynomial for coordinates on the x-axis and a second polynomial for coordinates on the y-axis.

9. Associating at least one object in the first image with a corresponding object in the second image; identifying the at least one object in the first image and at least one object in the second image; generating at least one first bounding box in the first image and at least one second bounding box in the second image, each of the at least one first bounding box and the at least one second bounding box representing a spatial location of an identified object; associating each first bounding box with a corresponding second bounding box based at least on a relationship between a projection of the first bounding box and the corresponding second bounding box, the projection being based on the approximated projection matrix; The computer-implemented method of any one of claims 3 to 8, comprising:

10. Associating each first bounding box with the corresponding second bounding box; projecting the at least one first bounding box onto the second image based on the approximated projection matrix to generate at least one projected first bounding box on the second image, each projected first bounding box corresponding to one of the at least one first bounding box; determining a relationship between each projected first bounding box and each second bounding box; associating each first bounding box with a corresponding second bounding box based on the determined relationship between the corresponding projected first bounding box and the corresponding second bounding box; 10. The computer-implemented method of claim 9, comprising:

11. the relationship between each projected first bounding box and each second bounding box comprises a degree of overlap between each projected first bounding box and each second bounding box; and / or 11. The computer-implemented method of claim 10, including a distance between each projected first bounding box and each second bounding box, said distance preferably being between a center of each projected first bounding box and a center of each second bounding box.

12. Associating each first bounding box with the corresponding second bounding box; identifying at least one feature of each identified object in the first image and the second image, wherein the at least one feature is preferably an appearance feature vector; 12. The computer-implemented method of claim 9, further comprising: comparing the at least one feature of each identified object in the first image with the at least one feature of each identified object in the second image.

13. 13. A computer-implemented method according to any one of claims 3 to 12, further comprising plotting positions of identified objects on one common image plane, said one common image plane preferably covering a 360° surround view of objects around the first image sensor, the second image sensor and further image sensors.

14. the first image sensor and the second image sensor are fisheye cameras positioned such that there is an overlap between at least one first image of a scene captured by the first image sensor and at least one second image of a scene captured by the second image sensor; The corresponding points are selected using a neural network including a convolutional neural network backbone and a keypoint extractor, the neural network being trained on a scene classification dataset including images sorted into scene classes, each scene class including a series of images captured from a single camera capturing overlapping regions of a scene and a series of images captured during the same period by another camera having an overlapping field of view with the single camera, the convolutional neural network backbone including a constrained deformable convolution module; the approximated projection matrix includes a first polynomial for coordinates on the x-axis and a second polynomial for coordinates on the y-axis, the first polynomial and the second polynomial each having a degree of 2, and the number of identified corresponding points is 3; Associating at least one identified object in the first image with a corresponding identified object in the second image includes: identifying at least one object in the first image and at least one object in the second image; generating at least one first bounding box in the first image and at least one second bounding box in the second image, each of the at least one first bounding box and the at least one second bounding box representing a spatial location of an identified object; projecting the at least one first bounding box onto the second image based on the approximated projection matrix to generate at least one projected first bounding box on the second image, each projected first bounding box corresponding to one of the at least one first bounding box; determining a degree of overlap between each projected first bounding box and each second bounding box; associating each first bounding box with a corresponding second bounding box based on the determined degree of overlap between the corresponding projected first bounding box and the corresponding second bounding box; The computer-implemented method of any one of claims 3 to 13, comprising:

15. The training dataset contains images sorted into scene classes, and each scene class has a series of images captured from a first image sensor capturing an overlapping region of the scene; Optionally, a series of images captured during the same period by another image sensor having a field of view that overlaps with said first image sensor; and Optionally, an image modified from a series of images captured from said first image sensor and / or a series of images captured during the same period by another image sensor having a field of view that overlaps with said first image sensor; A training data set for a neural network, in particular a neural network according to claim 6 or 14, comprising:

16. 15. A system adapted for use in a vehicle comprising at least a first image sensor and a second image sensor, one or more processors, and a memory storing executable instructions for execution by said one or more processors, said executable instructions comprising instructions for performing a computer-implemented method according to any one of claims 3 to 14. system.

17. A computer program, machine-readable medium, or data signal comprising instructions that, when executed on one or more processors, cause the one or more processors to perform the steps of the computer-implemented method of any one of claims 3 to 14.

Images