Target detection method and camera

Abstract

The application provides a target detection method and a camera. The method comprises the following steps: when a target object enters an overlapping area between a long-focus sensor and a fisheye sensor, determining first target detection information corresponding to the target object under the long-focus sensor, determining second target detection information corresponding to the target object under the fisheye sensor, and determining that the first target detection information and the second target detection information belong to the same target object; if the target object enters the overlapping area from a collection area of the long-focus sensor, transmitting the first target detection information to the fisheye sensor, and associating the first target detection information with the second target detection information by the fisheye sensor; and if the target object enters the overlapping area from a collection area of the fisheye sensor, transmitting the second target detection information to the long-focus sensor, and associating the second target detection information with the first target detection information by the long-focus sensor. Through the technical scheme, full coverage of a target detection blind area is realized, and real-time positions are obtained.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a target detection method and a camera. Background Technology

[0002] Object detection, also known as object extraction, is an image segmentation technique based on the geometric and statistical features of target objects. It combines object segmentation and recognition, and accuracy and real-time performance are crucial capabilities of the system. With the development of computer technology and the widespread application of computer vision principles, there are increasingly more ways to perform real-time object detection using image processing techniques. Real-time object detection has broad application value in intelligent transportation systems, intelligent management systems, and other fields.

[0003] To achieve target detection, cameras are typically deployed in the target scene (i.e., the application scenario where target objects need to be detected). The cameras need to capture images of the target scene, and image processing technology can be used to analyze the real-time location of the target object based on the image, thereby enabling the management of the target object.

[0004] However, due to the limited field of view of the camera, when the target object is outside the camera's field of view, the camera cannot capture an image of the target object, making it impossible to analyze the real-time position of the target object. Summary of the Invention

[0005] This application provides a target detection method applied to a camera that achieves full coverage of the target detection blind zone. The camera includes a telephoto sensor and a fisheye sensor. The method includes:

[0006] When a target object enters the overlapping area between the telephoto sensor and the fisheye sensor, the first target detection information corresponding to the target object under the telephoto sensor is determined, the second target detection information corresponding to the target object under the fisheye sensor is determined, and the first target detection information and the second target detection information belong to the same target object.

[0007] If the target object enters the overlapping area from the acquisition area of ​​the telephoto sensor, the first target detection information is transmitted to the fisheye sensor so that the fisheye sensor associates the first target detection information with the second target detection information under the fisheye sensor.

[0008] If the target object enters the overlapping area from the acquisition area of ​​the fisheye sensor, the second target detection information is transmitted to the telephoto sensor so that the telephoto sensor associates the second target detection information with the first target detection information under the telephoto sensor.

[0009] This application provides a camera for achieving full coverage of target detection blind spots. The camera includes a telephoto sensor, a fisheye sensor, and a processor; wherein:

[0010] The telephoto sensor is used to acquire telephoto images and send the telephoto images to the processor;

[0011] The fisheye sensor is used to acquire fisheye images and send the fisheye images to the processor;

[0012] Based on the telephoto image and the fisheye image, the processor is configured to execute:

[0013] When a target object enters the overlapping area between the telephoto sensor and the fisheye sensor, first target detection information corresponding to the target object under the telephoto sensor is determined based on the telephoto image, and second target detection information corresponding to the target object under the fisheye sensor is determined based on the fisheye image. It is determined that the first target detection information and the second target detection information belong to the same target object. If the target object enters the overlapping area from the acquisition area of ​​the telephoto sensor, the first target detection information is transmitted to the fisheye sensor so that the fisheye sensor associates the first target detection information with the second target detection information under the fisheye sensor. If the target object enters the overlapping area from the acquisition area of ​​the fisheye sensor, the second target detection information is transmitted to the telephoto sensor so that the telephoto sensor associates the second target detection information with the first target detection information under the telephoto sensor.

[0014] As can be seen from the above technical solutions, in this embodiment, the camera may include a telephoto sensor and a fisheye sensor. The telephoto and fisheye sensors increase the camera's field of view, enabling it to capture images within a larger area and achieve full coverage of blind spots in target detection. This achieves blind-spot-free, full-scene target detection, avoiding situations where the real-time position of the target object cannot be obtained when it is in certain locations. When the target object enters the overlapping area between the telephoto and fisheye sensors, the target detection information corresponding to the target object under the telephoto sensor can be associated with the target detection information corresponding to the target object under the fisheye sensor, enabling real-time detection even when the target object is moving. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this application.

[0016] Figure 1 This is a flowchart illustrating a target detection method in one embodiment of this application;

[0017] Figure 2 This is a schematic diagram of the structure of a camera in one embodiment of this application;

[0018] Figures 3A-3D This is a schematic diagram of the preset unfolding center point position in one embodiment of this application;

[0019] Figure 4 This is a schematic diagram of the effective area in a fisheye image according to one embodiment of this application;

[0020] Figure 5A This is a schematic diagram illustrating the unfolding of a fisheye image in one embodiment of this application;

[0021] Figures 5B-5D This is a schematic diagram of a fisheye unfolded image in one embodiment of this application;

[0022] Figure 6A and Figure 6B This is the correspondence between the half field of view and the actual image height in one embodiment of this application;

[0023] Figure 6C This is the relationship between the initial ideal image height and the half field of view in one embodiment of this application;

[0024] Figure 6D This is the relationship between the initial ratio and the target ratio in one embodiment of this application;

[0025] Figure 6E This is the relationship between the height of the ideal target image and the half field of view in one embodiment of this application;

[0026] Figures 6F-6H This is a schematic diagram of a fisheye unfolded image in one embodiment of this application;

[0027] Figures 7A-7B This is a schematic diagram illustrating an application scenario of target detection in one embodiment of this application;

[0028] Figure 8 This is a schematic diagram of the target detection device in one embodiment of this application;

[0029] Figure 9 This is a hardware structure diagram of a camera according to one embodiment of this application. Detailed Implementation

[0030] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0031] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0032] This application proposes a target detection method for a camera that achieves full coverage of the target detection blind zone. The camera includes a telephoto sensor and a fisheye sensor. The number of telephoto sensors can be at least one, such as two or three, and the number of fisheye sensors can be at least one, such as one. There is no limitation on the number of either the telephoto or fisheye sensors. See also... Figure 1 The diagram shown is a flowchart of a target detection method, which may include:

[0033] Step 101: When the target object enters the overlapping area between the telephoto sensor and the fisheye sensor, determine the first target detection information corresponding to the target object under the telephoto sensor, determine the second target detection information corresponding to the target object under the fisheye sensor, and determine that the first target detection information and the second target detection information belong to the same target object, that is, the first target detection information and the second target detection information are the detection information of the same target object, that is, the detection information of the same target object under different sensors.

[0034] Step 102: If the target object enters the overlapping area from the acquisition area of ​​the telephoto sensor, the first target detection information is transmitted to the fisheye sensor so that the fisheye sensor associates the first target detection information under the telephoto sensor with the second target detection information under the fisheye sensor.

[0035] Step 103: If the target object enters the overlapping area from the acquisition area of ​​the fisheye sensor, the second target detection information is transmitted to the telephoto sensor so that the telephoto sensor associates the second target detection information under the fisheye sensor with the first target detection information under the telephoto sensor.

[0036] In one possible implementation, the first target detection information includes a first position of the target object under the telephoto sensor, and the second target detection information includes a second position of the target object under the fisheye sensor. Determining that the first target detection information and the second target detection information belong to the same target object may include, but is not limited to: if the first position and the second position correspond to the same physical location at the same time, then the first target detection information and the second target detection information belong to the same target object.

[0037] For example, a telephoto image can be acquired using a telephoto sensor, and a first position of the target object under the telephoto sensor can be determined based on the telephoto image; a fisheye image can be acquired using a fisheye sensor, and a second position of the target object under the fisheye sensor can be determined based on the fisheye image. Furthermore: based on the calibration relationship between the telephoto sensor and the fisheye sensor, the first position can be converted into a mapped position of the target object under the fisheye sensor. If this mapped position matches the second position, then the first and second positions are determined to correspond to the same physical location. Alternatively, based on the calibration relationship between the telephoto sensor and the fisheye sensor, the second position can be converted into a mapped position of the target object under the telephoto sensor. If this mapped position matches the first position, then the first and second positions are determined to correspond to the same physical location.

[0038] For example, determining the second position of a target object under a fisheye sensor based on a fisheye image may include, but is not limited to: dividing a first fisheye region of interest from the fisheye image based on a preset unfolding center point position, a preset unfolding width, and a preset unfolding height, and unfolding the first fisheye region of interest to obtain a first fisheye unfolded image; wherein the center pixel of the first fisheye unfolded image corresponds to the preset unfolding center point position, and the preset unfolding center point position is located within the acquisition area of ​​the telephoto sensor. Then, the second position of the target object can be determined based on the first fisheye unfolded image.

[0039] For example, expanding the first fisheye region of interest to obtain a first fisheye expanded image includes: for each initial pixel in the first fisheye expanded image, determining the coordinates of the projection point corresponding to the initial pixel on the hemisphere of the fisheye image based on perspective projection; determining the azimuth and incident angle corresponding to the initial pixel based on the projection point coordinates; determining the target pixel corresponding to the initial pixel in the first fisheye region of interest based on the equivalent focal length, azimuth, and incident angle; determining the pixel value of the initial pixel based on the pixel value of the target pixel; and generating the first fisheye expanded image based on the pixel values ​​of all initial pixels.

[0040] For example, the equivalent focal length may be determined in ways including but not limited to: determining the equivalent focal length based on the effective width of the first fisheye region of interest and the maximum field of view of the fisheye sensor.

[0041] In one possible implementation, if the target object enters the acquisition area of ​​the fisheye sensor, a fisheye image can be acquired using the fisheye sensor, and the third position of the target object under the fisheye sensor can be determined based on the fisheye image. This determination of the third position of the target object under the fisheye sensor based on the fisheye image may include, but is not limited to: dividing a second fisheye region of interest from the fisheye image and expanding the second fisheye region of interest to obtain a second expanded fisheye image; wherein the center pixel of the second expanded fisheye image corresponds to the center pixel of the second fisheye region of interest. Then, the third position of the target object can be determined based on the second expanded fisheye image.

[0042] For example, expanding the second fisheye region of interest to obtain a second fisheye expanded image may include, but is not limited to: for each initial pixel in the second fisheye region of interest, querying a configured mapping table based on the half field of view corresponding to the initial pixel to obtain the actual image height corresponding to the initial pixel; determining the target ideal image height corresponding to the initial pixel based on the half field of view; determining the target pixel in the second fisheye expanded image corresponding to the initial pixel based on the actual image height and the target ideal image height; wherein, the actual image height may represent the distance between the initial pixel and the center pixel of the second fisheye region of interest, and the target ideal image height may represent the distance between the target pixel and the center pixel of the second fisheye expanded image; determining the pixel value of the target pixel based on the pixel value of the initial pixel; and generating the second fisheye expanded image based on the pixel values ​​of all target pixels.

[0043] For example, determining the target ideal image height corresponding to the initial pixel based on the half field of view may include, but is not limited to: determining the initial ideal image height corresponding to the initial pixel based on the half field of view and the equivalent focal length; determining the initial ratio based on the initial ideal image height and the actual image height; modulating the initial ratio using a preset modulation ratio to obtain the target ratio; and determining the target ideal image height based on the target ratio and the actual image height.

[0044] As can be seen from the above technical solutions, in this embodiment, the camera may include a telephoto sensor and a fisheye sensor. The telephoto and fisheye sensors increase the camera's field of view, enabling it to capture images within a larger area and achieve full coverage of blind spots in target detection. This achieves blind-spot-free, full-scene target detection, avoiding situations where the real-time position of the target object cannot be obtained when it is in certain locations. When the target object enters the overlapping area between the telephoto and fisheye sensors, the target detection information corresponding to the target object under the telephoto sensor can be associated with the target detection information corresponding to the target object under the fisheye sensor, enabling real-time detection even when the target object is moving.

[0045] The technical solutions described above in the embodiments of this application will be explained below in conjunction with specific application scenarios.

[0046] This application proposes a camera with full coverage of target detection blind spots. Combining fisheye expansion technology and spatial matching technology, it achieves blind-spot-free full-scene target detection and enables cross-lens detection of target trajectories. The camera may include a telephoto sensor and a fisheye sensor. There can be one fisheye sensor and one or more telephoto sensors. The fisheye sensor faces directly below the camera, while the telephoto sensors face distant areas. For ease of description, this embodiment uses two telephoto sensors as an example, denoted as telephoto sensor 1 and telephoto sensor 2. Of course, in practical applications, the number of telephoto sensors can be more, such as three or four.

[0047] See Figure 2 The diagram shows the structure of a camera, which may include a fisheye sensor, a telephoto sensor 1, and a telephoto sensor 2. In this embodiment, the target scene can be divided into regions a1, a2, a3, a4, and a5. Target objects (such as target faces, target bodies, target vehicles, etc., without restriction on the type of target object) can be located in these regions.

[0048] In this diagram, region a1 is the acquisition area of ​​telephoto sensor 1. When the target object is located in region a1, telephoto sensor 1 can acquire a telephoto image of the target object (for ease of distinction, the image acquired by the telephoto sensor is referred to as a telephoto image). Region a2 is the overlapping area between telephoto sensor 1 and fisheye sensor. When the target object is located in region a2, both telephoto sensor 1 and fisheye sensor can acquire a fisheye image of the target object (for ease of distinction, the image acquired by the fisheye sensor is referred to as a fisheye image). Region a3 is the acquisition area of ​​fisheye sensor. When the target object is located in region a3, fisheye sensor can acquire a fisheye image of the target object. Region a4 is the overlapping area between telephoto sensor 2 and fisheye sensor. When the target object is located in region a4, both telephoto sensor 2 and fisheye sensor can acquire a fisheye image of the target object. Region a5 is the acquisition area of ​​telephoto sensor 2. When the target object is located in region a5, telephoto sensor 2 can acquire a telephoto image of the target object.

[0049] In the above application scenarios, the target detection method of this embodiment may involve the following process:

[0050] First, the installation methods of telephoto sensor 1 and telephoto sensor 2.

[0051] In this embodiment, it is necessary for the telephoto sensor 1 and the fisheye sensor to have an overlapping area (i.e., area a2). This means that when the target object is within the overlapping area, both the telephoto image and the fisheye image can detect the target object. To achieve this overlapping area, the angular relationship between the telephoto sensor 1 and the fisheye sensor needs to be constrained. However, this angular relationship is not restricted, as long as there is an overlapping area between the telephoto sensor 1 and the fisheye sensor. Similarly, it is necessary for the telephoto sensor 2 and the fisheye sensor to have an overlapping area (i.e., area a4). This means that when the target object is within the overlapping area, both the telephoto image and the fisheye image can detect the target object. To achieve this overlapping area, the angular relationship between the telephoto sensor 2 and the fisheye sensor needs to be constrained. However, this angular relationship is not restricted, as long as there is an overlapping area between the telephoto sensor 2 and the fisheye sensor.

[0052] Second, the processing method for the target object being located in the acquisition area (i.e., area a1) of the telephoto sensor 1.

[0053] When the target object is in region a1, the telephoto sensor 1 can acquire telephoto images (which can be multiple frames of telephoto images) of the target object. Based on these telephoto images, the target detection information b1 corresponding to the target object under the telephoto sensor 1 can be analyzed. The target detection information b1 can include, but is not limited to, at least one of the following: a unique identifier assigned to the target object by the telephoto sensor 1 (denoted as identifier w1), the target object's attribute information (e.g., if the target object is a face, it can be facial features; if the target object is a vehicle, it can be a license plate identifier; there are no restrictions on this attribute information), the target object's position, and the time the target object has been at that position. Of course, the above are just a few examples, and there are no restrictions on the target detection information b1.

[0054] Regarding the unique identifier of the target object, the telephoto sensor 1 can assign a unique identifier to the target object upon its first detection, without any restrictions, as long as it is different from the identifiers of other target objects. The unique identifier of the target object will not change during its movement.

[0055] Regarding the attribute information of the target object, the attribute information of the target object can be determined based on the telephoto image acquired by the telephoto sensor 1. For example, the telephoto image can be input into the neural network model, and the neural network model can output the attribute information of the target object. Alternatively, the attribute information of the target object can be obtained in other ways. This embodiment does not limit the process of obtaining this attribute information.

[0056] Regarding the location of the target object and the time it has been at that location, the location can be analyzed based on each long-focus image acquired by long-focus sensor 1. This process is unrestricted, and the acquisition time of the long-focus image represents the time the target object has been at that location. Clearly, multiple long-focus images correspond to multiple locations, and these locations can form a trajectory.

[0057] Third, the processing method for the target object being located in the overlapping area (i.e., area a2) between the telephoto sensor 1 and the fisheye sensor, that is, the target object moves from area a1 to area a2.

[0058] When the target object moves from region a1 to region a2, since the target object is always within the field of view of the telephoto sensor 1, the telephoto sensor 1 continuously acquires telephoto images of the target object and analyzes the target detection information b1 corresponding to the target object under the telephoto sensor 1 based on these telephoto images. The target detection information b1 may include the unique identifier w1 of the target object, the attribute information of the target object, the position and time of the target object in region a1, and the position and time of the target object in region a2.

[0059] When the target object moves from region a1 to region a2, it enters the field of view of the fisheye sensor. Therefore, the fisheye sensor can acquire a fisheye image of the target object and analyze it to determine the target detection information b2. The target detection information b2 may include, but is not limited to, at least one of the following: a unique identifier assigned to the target object by the fisheye sensor (denoted as identifier w2, this is optional), the target object's attribute information (this is optional), the target object's position, and the time the target object has been at that position. Of course, the above are just a few examples and are not intended to be restrictive.

[0060] Based on the location and time in target detection information b1, and the location and time in target detection information b2, if the location in target detection information b1 and the location in target detection information b2 correspond to the same physical location at the same time, then target detection information b1 and target detection information b2 are determined to belong to the same target object, that is, target detection information b1 and target detection information b2 are target detection information of the same target object.

[0061] For example, if target detection information b1 includes location P1 and time point T1, it means that the target object is at location P1 at time point T1. Target detection information b2 includes location P2 and time point T1, indicating that the target object is at location P2 at time point T1. Based on this, if location P1 and location P2 correspond to the same physical location, then target detection information b1 and target detection information b2 are determined to belong to the same target object.

[0062] For example, regarding how to determine whether positions P1 and P2 correspond to the same physical position, the following method can be used: Pre-calibrate the calibration relationship between the telephoto sensor 1 and the fisheye sensor. This calibration relationship represents the mapping relationship between the coordinate system of the telephoto sensor 1 and the coordinate system of the fisheye sensor, and also represents the pixel mapping relationship between the telephoto image (i.e., the image under the coordinate system of the telephoto sensor 1) and the fisheye image (i.e., the image under the coordinate system of the fisheye sensor). Each pixel in the telephoto image can be mapped to the fisheye image, and each pixel in the fisheye image can be mapped to the telephoto image.

[0063] To determine whether positions P1 and P2 correspond to the same physical location, the calibration relationship between the telephoto sensor 1 and the fisheye sensor can be used to convert position P1 (i.e., a pixel in the telephoto image) into a mapped position P3 (a pixel in the fisheye image) of the target object under the fisheye sensor. If mapped position P3 matches position P2 (i.e., a pixel in the fisheye image), then positions P1 and P2 correspond to the same physical location. If mapped position P3 does not match position P2, then positions P1 and P2 do not correspond to the same physical location. Alternatively, based on the calibration relationship between the telephoto sensor 1 and the fisheye sensor, position P2 can be converted into a mapped position P4 (a pixel in the fisheye image) of the target object under the telephoto sensor 1. If mapped position P4 matches position P1, then positions P1 and P2 correspond to the same physical location. If mapped position P4 does not match position P1, then positions P1 and P2 do not correspond to the same physical location.

[0064] For example, after determining that target detection information b1 and target detection information b2 belong to the same target object, target detection information b1 can be transmitted to the fisheye sensor, thereby associating target detection information b1 with target detection information b2, that is, fusing target detection information b1 with target detection information b2.

[0065] When fusing target detection information b1 and target detection information b2, the fisheye sensor can update the unique identifier of the target object to identifier w1 (i.e., the unique identifier assigned to the target object by telephoto sensor 1). Since the unique identifier of the target object is the same under different sensors, the target detection information under different sensors can be fused together, so that the target object has a unique identifier along the entire path.

[0066] When fusing target detection information b1 and target detection information b2, the fisheye sensor can update the target object attribute information in target detection information b1 to target detection information b2.

[0067] When fusing target detection information b1 and target detection information b2, the fisheye sensor can update the position and time in target detection information b1 to target detection information b2. In this way, target detection information b2 can include the trajectory of the target object when it is in region a1 and the trajectory of the target object when it is in region a2, thereby realizing the fusion of the trajectory of the target object in region a1 and region a2.

[0068] For example, although fisheye sensors (such as fisheye lenses) have a wide field of view (up to 180 degrees), they also suffer from significant distortion, resulting in large distortion of the fisheye image and making it difficult to accurately analyze the position of the target object. Therefore, to achieve target detection, position mapping, and visual effects, it is necessary to convert the fisheye image acquired by the fisheye sensor into a fisheye unfolded image. The position of the target object is then determined based on the fisheye unfolded image. In other words, when determining the target detection information b2 corresponding to the target object under the fisheye sensor, it can be based on the fisheye unfolded image. In summary, a mapping relationship between the fisheye image and the fisheye unfolded image can also be provided to convert the fisheye image into a fisheye unfolded image, and then the position of the target object under the fisheye sensor can be determined based on the fisheye unfolded image.

[0069] In one possible implementation, the following steps can be used to provide a mapping relationship between the fisheye image and the fisheye unfolded image. Of course, the following steps are just examples and are not intended to limit the scope of the implementation.

[0070] Step S11: Based on the preset unfolding center point position (corresponding to the telephoto sensor 1), preset unfolding width, and preset unfolding height, divide the fisheye region of interest from the fisheye image.

[0071] For example, the preset unfolding width and preset unfolding height can both be configured empirically and are not limited thereto. Furthermore, the preset unfolding center point position can be configured empirically, as long as it is located within the acquisition area of ​​the telephoto sensor 1. For instance, the installation direction of the fisheye sensor can be divided into longitudinal road installation and transverse road installation, see [reference needed]. Figure 3A The diagram shows a longitudinal installation along the road. The upper area is the overlapping area of ​​the fisheye sensor and the telephoto sensor 1. Therefore, a certain position in the upper area can be configured as the preset deployment center point position corresponding to the telephoto sensor 1. (See [reference]). Figure 3B The image shows an example of the preset deployment center point position corresponding to telephoto sensor 1, and also shows that a point in the lower region can be configured as the preset deployment center point position corresponding to telephoto sensor 2. See also... Figure 3C The diagram shows a horizontal installation along a road. The left-hand area is the overlapping area of ​​the fisheye sensor and telephoto sensor 1. Therefore, a certain position in the left-hand area can be configured as the preset deployment center point position corresponding to telephoto sensor 1. (See [reference]). Figure 3D The image shows an example of the preset unfolding center point position corresponding to telephoto sensor 1, and it also shows that a point in the right area can be configured as the preset unfolding center point position corresponding to telephoto sensor 2.

[0072] For example, based on the known preset expansion center point position, preset expansion width, and preset expansion height, the fisheye region of interest (i.e., a part of the fisheye image) can be divided from the fisheye image. In other words, the center point of the fisheye region of interest is the preset expansion center point position, the width of the fisheye region of interest is the preset expansion width, and the height of the fisheye region of interest is the preset expansion height.

[0073] For example, first extract the effective region from the fisheye image, see... Figure 4 As shown, the circular area in the middle is the valid region, while the black area at the edge is the invalid region. The valid region can be extracted from the fisheye image, and there are no restrictions on the extraction method. After obtaining the valid region, the fisheye region of interest is divided from the valid region based on the preset unfolding center point position, preset unfolding width, and preset unfolding height.

[0074] Step S12: For each initial pixel in the fisheye unfolded image, determine the coordinates of the projection point corresponding to that initial pixel on the hemisphere of the fisheye image based on perspective projection. Determine the azimuth and incident angle corresponding to that initial pixel based on the projection point coordinates. Determine the target pixel corresponding to that initial pixel in the region of interest of the fisheye image based on the equivalent focal length, azimuth, and incident angle. The equivalent focal length is determined based on the effective width of the region of interest of the fisheye image and the maximum field of view of the fisheye sensor.

[0075] For example, the mapping relationship between a fisheye image and a fisheye unfolded image is actually the mapping relationship between the pixels in the fisheye unfolded image and the pixels in the fisheye region of interest (i.e., the fisheye image). The pixels in the fisheye unfolded image are denoted as the initial pixels, and the pixels in the fisheye region of interest are denoted as the target pixels. The following describes the mapping relationship between the initial pixels in the fisheye unfolded image and the target pixels in the fisheye region of interest in conjunction with a specific application scenario.

[0076] See Figure 5A The diagram illustrates the unfolding of a fisheye image (such as the region of interest within the fisheye image) using methods like PTZ unfolding. It is based on an isometric projection fisheye imaging model. f oy f Let p(x) be the plane containing the fisheye image. f y f Let P be a point on the fisheye image, equidistantly projected from point P(x,y,z) on the hemisphere of the fisheye projection. The projection radius is the equivalent focal length f. Projecting point P through a perspective plane centered at O2 onto a plane tangent to the sphere at O1(0,0,f), intersecting at P1(x1,y1,z1), then point P1 is p(x...y...z1). f y fThe point p corresponds to on the perspective projection plane. Based on this, under equidistant projection, the distance r of point p relative to the center of the fisheye image and the angle of incidence are... It satisfies formula (4). Since the distance r from the projection point to the center reaches its maximum when the incident angle is maximum, the equivalent focal length f is calculated by formula (5). Wherein, W is the maximum effective width of the fisheye, which is twice the fisheye radius 2*R; FOV is the maximum field of view of the fisheye, which can be measured in advance.

[0077]

[0078]

[0079] Based on this, the reverse mapping establishes the positional correspondence between the fisheye unfolded image and the fisheye image as follows:

[0080] (1) For any point (m,n) on the fisheye unfolded image, the point (m,n) corresponds to P1(x1,y1,z1) on the projection plane tangent to point O1, and there is a correspondence as shown in formula (6).

[0081]

[0082] (2) P(x,y,z) satisfies x on the projected sphere. 2 +y 2 +z 2 =f 2 Since the similar triangles under perspective projection are related as ΔO2PQ~ΔO2P1Q1, the following relationship is satisfied. Since the projection point P1 is on the spherical tangent plane, z1 = f; in summary, the correspondence of formula (7) can be obtained.

[0083]

[0084] The coordinates (x, y, z) of point P on the sphere can be obtained from formula (7), as shown in formula (8):

[0085]

[0086] (3) Regarding the azimuth angle θ and incident angle of fisheye imaging Based on the trigonometric relationship of the coordinates of point P, the azimuth angle θ and the incident angle of the fisheye image can be obtained by formula (9).

[0087]

[0088] (4) p(x) on the fisheye image by equidistant projection f y fWhen the distance r of point p relative to the center of the fisheye image is given by formula (10), the coordinates of the projection point on the fisheye image are given by formula (11).

[0089]

[0090]

[0091] (5) The center O1 of the fisheye unfolded image corresponds to the center O of the fisheye image. If we want to use p(x) on the fisheye image... f y f If the point corresponds to the center of the fisheye unfolded image, then according to formulas (11), (10), and (9), for p(x) f y f After mapping the point in the positive direction to the corresponding three-dimensional point P(x,y,z) on the sphere surface, rotate counterclockwise along the Xc axis by ang_x and counterclockwise along the Yc axis by ang_y to O1(0,0,f). Then, calculate the rotation relationship according to formula (12) and perform the reverse rotation in step (2).

[0092]

[0093] Therefore, a relationship was established between any point (m,n) on the fisheye unfolded image and a point (x) on the fisheye image. f y f The one-to-one correspondence between the fisheye unfolded image and the fisheye image (region of interest) is established to complete the mapping between them.

[0094] In summary, for each initial pixel in the fisheye unfolded image, the coordinates of the projection point corresponding to that initial pixel on the hemisphere of the fisheye image can be determined based on perspective projection. See Equations (6) and (7), where (m,n) represents any initial pixel in the fisheye unfolded image, (x1,y1,z1) represents the coordinates of the projection point corresponding to that initial pixel on the hemisphere of the fisheye image, and f represents the equivalent focal length.

[0095] Furthermore, the azimuth and incident angle of the initial pixel (m,n) can be determined based on the coordinates of the projection point. Referring to formulas (8) and (9), the azimuth θ and incident angle of the initial pixel (m,n) can be obtained based on the projection point coordinates (x1,y1,z1) and the equivalent focal length f.

[0096] Furthermore, it can be based on the equivalent focal length f, azimuth angle θ, and incident angle. Determine the target pixel (x) in the fisheye region of interest corresponding to the initial pixel (m,n). f y fAs shown in equations (10) and (11), the equivalent focal length f and the incident angle can be used as a basis. Determine the distance r of point p relative to the center of the fisheye image. Then, based on the distance r and the azimuth angle θ, the target pixel (x) corresponding to the initial pixel (m,n) can be determined. f y f ).

[0097] Furthermore, the equivalent focal length f can be determined based on the effective width of the region of interest (ROI) of the fisheye and the maximum field of view (FOV) of the fisheye sensor. See formula (5), where R is the fisheye radius, 2R represents the effective width of the ROI, and FOV is the maximum field of view of the fisheye sensor.

[0098] In summary, referring to formulas (4)-(10), the mapping relationship between the initial pixels in the fisheye unfolded image and the target pixels in the fisheye region of interest can be obtained. Based on this mapping relationship, for each initial pixel in the fisheye unfolded image, the corresponding target pixel can be found in the fisheye region of interest. Furthermore, for each target pixel in the fisheye region of interest, the corresponding initial pixel can also be found in the fisheye unfolded image.

[0099] Step S13: For each initial pixel in the fisheye unfolded image, determine the target pixel in the fisheye region of interest corresponding to the initial pixel, and determine the pixel value of the initial pixel based on the pixel value of the target pixel (e.g., use the pixel value of the target pixel as the pixel value of the initial pixel). Based on this, the fisheye unfolded image can be generated based on the pixel values ​​of all initial pixels.

[0100] In one possible implementation, for each initial pixel in the fisheye unfolded image, step S12 can be used to determine the target pixel corresponding to that initial pixel in the fisheye region of interest. In another possible implementation, a mapping relationship can be established between the initial pixels in the fisheye unfolded image and the target pixels in the fisheye region of interest. The establishment method is described in step S12. Table 1 shows an example of this mapping relationship. Based on this, for each initial pixel in the fisheye unfolded image, the target pixel corresponding to that initial pixel can be obtained by looking up the mapping relationship shown in Table 1.

[0101] Table 1

[0102] P11 P21 P12 P22 … …

[0103] For each initial pixel in the fisheye unfolded image, after finding the target pixel corresponding to that initial pixel, the pixel value of the target pixel can be used as the pixel value of the initial pixel. The pixel values ​​of all the initial pixels then form the fisheye unfolded image, thus obtaining the fisheye unfolded image.

[0104] In summary, the region of interest (ROI) in a fisheye image can be expanded to obtain an expanded fisheye image. Referring to the above description, since the center point of the ROI is a preset expansion center point, after expanding the ROI into an expanded fisheye image, the center pixel of the expanded image also corresponds to the preset expansion center point. This preset expansion center point is located within the acquisition area of ​​the telephoto sensor 1, meaning it lies in the overlapping area between the telephoto sensor 1 and the fisheye sensor. Therefore, the center pixel of the expanded fisheye image also corresponds to this overlapping area.

[0105] For example, taking longitudinal installation along a road as an example, see... Figure 5B As shown, two preset expansion center point positions are illustrated. For the upper preset expansion center point position, after dividing the fisheye region of interest based on this preset expansion center point position, the corresponding fisheye expanded image can be found in [reference needed]. Figure 5C As shown, it is obvious that in Figure 5C In the image, the center pixel of the fisheye unfolded image corresponds to the preset unfolded center point position.

[0106] For the preset unfolding center point position on the lower side, after dividing the fisheye region of interest based on this preset unfolding center point position, the fisheye unfolded image corresponding to the fisheye region of interest can be found in [reference needed]. Figure 5D As shown, it is obvious that in Figure 5D In the image, the center pixel of the fisheye unfolded image corresponds to the preset unfolded center point position.

[0107] See Figure 5C and Figure 5D As shown, compared to fisheye images, fisheye unfolded images better remove the distortion of fisheye images and establish the positional mapping relationship between each initial pixel in the fisheye unfolded image and the target pixel in the fisheye image. Target detection can be performed using fisheye unfolded images.

[0108] Fourth, the processing method for the target object being located in the acquisition area of ​​the fisheye sensor (i.e., area a3).

[0109] When the target object moves from region a2 to region a3, since the target object remains within the field of view of the fisheye sensor, the fisheye sensor continuously acquires fisheye images of the target object and analyzes these images to derive target detection information b2. Target detection information b2 may include the target object's unique identifier w1, the target object's attribute information, the target object's position and time in region a2, and the target object's position and time in region a3. Since target detection information b1 and target detection information b2 have already been fused when the target object is in region a2, target detection information b2 may also include the target object's position and time in region a1.

[0110] In summary, the target detection information b2 can include the trajectory of the target object when it is in region a1, the trajectory of the target object when it is in region a2, and the trajectory of the target object when it is in region a3, thereby realizing the fusion of the trajectory of the target object in regions a1, a2, and a3. These trajectories correspond to the same unique identifier w1.

[0111] For example, after the target object moves to region a3, a fisheye image can be acquired using a fisheye sensor. Based on the fisheye image, the target object's position under the fisheye sensor can be determined, i.e., the target object is located in region a1. Further, to achieve target detection, the fisheye image acquired by the fisheye sensor needs to be converted into a fisheye unfolded image. The target object's position under the fisheye sensor is then determined based on the unfolded image. Therefore, a mapping relationship between the fisheye image and the fisheye unfolded image can be provided. The fisheye image is converted into a fisheye unfolded image, and then the target object's position under the fisheye sensor is determined based on the unfolded image. The mapping relationship between the fisheye image and the fisheye unfolded image can be implemented using the following steps; however, these steps are merely examples and are not intended to be limiting.

[0112] Step S21: Delineate the fisheye region of interest (i.e., the main image correction area) from the fisheye image.

[0113] For example, the region of interest (ROI) can be segmented from a fisheye image based on a preset unfolding center point position, a preset unfolding width, and a preset unfolding height. The preset unfolding width and height can be configured empirically and are not restricted. The preset unfolding center point position can also be configured empirically, as long as it is located within the acquisition area of ​​the fisheye sensor; for example, it can be located in the center of the fisheye sensor's acquisition area.

[0114] Given the preset unfolding center point position, preset unfolding width, and preset unfolding height, a region of interest (ROI) can be defined from the fisheye image. The center point of the ROI is the preset unfolding center point position, the width of the ROI is the preset unfolding width, and the height of the ROI is the preset unfolding height. Clearly, if the preset unfolding center point position is located in the center of the fisheye sensor's acquisition area, then the center point of the ROI can be the center point of the fisheye image.

[0115] Step S22: For each initial pixel in the fisheye region of interest, query the configured mapping table based on the half-field angle corresponding to the initial pixel to obtain the actual image height corresponding to the initial pixel; determine the target ideal image height corresponding to the initial pixel based on the half-field angle; and determine the target pixel in the fisheye unfolded image corresponding to the initial pixel based on the actual image height and the target ideal image height. Here, the actual image height can represent the distance between the initial pixel and the center pixel of the fisheye region of interest, and the target ideal image height can represent the distance between the target pixel and the center pixel of the fisheye unfolded image.

[0116] For example, determining the target ideal image height corresponding to the initial pixel based on the half field of view may include, but is not limited to: determining the initial ideal image height corresponding to the initial pixel based on the half field of view and the equivalent focal length; determining the initial ratio based on the initial ideal image height and the actual image height; modulating the initial ratio using a preset modulation ratio to obtain the target ratio; and determining the target ideal image height based on the target ratio and the actual image height.

[0117] For example, most imaging systems satisfy the object-image relationship h = f * tan(Y_angle) to ensure that a straight line remains a straight line after imaging. When the field of view is close to or exceeds 90 degrees, h = f * tan(Y_angle) becomes meaningless. Fisheye lenses based on equidistant projection are designed according to the object-image relationship h = f * Y_angle, resulting in significant distortion, especially as the image compression increases closer to the circumference. It is necessary to correct the distortion of the entire road image as much as possible, straightening the road while minimizing field of view loss. Based on the above principle, this embodiment employs an image height modulation correction method. Image height refers to the distance from the imaging point to the center of the image plane. Therefore, the process of correcting the distortion of the fisheye image to obtain the fisheye unfolded image of the main fisheye view can be carried out in the following way:

[0118] (1) Obtain the initial correspondence between the half field of view y_angle (unit: degrees) and the actual image height Real_height (unit: mm) from the optical measurement parameters of the fisheye sensor. See [link to relevant documentation]. Figure 6AThe image shown is an example of the initial correspondence. Furthermore, to improve accuracy, bilinear interpolation can be performed on the half-field angle with an accuracy of 0.2 degrees to obtain the corresponding actual image height, thus obtaining... Figure 6B The target correspondence is shown.

[0119] For example, the optical measurement parameters of the fisheye sensor may include an initial correspondence, which can be obtained from the optical measurement parameters. Then, bilinear interpolation is performed on the half-field angle to obtain the target correspondence. In subsequent embodiments, the initial correspondence or the target correspondence can be used for processing. For ease of description, the target correspondence is used as an example.

[0120] For example, for each initial pixel in the fisheye region of interest, the half-field of view corresponding to that initial pixel can be determined. The half-field of view corresponding to different initial pixels may be the same or different; this is not limited. After obtaining the half-field of view corresponding to the initial pixel, the target correspondence can be queried using that half-field of view. Figure 6B As shown in the figure, the actual image height corresponding to the initial pixel is obtained. The actual image height represents the distance between the initial pixel and the center pixel of the fisheye region of interest.

[0121] (2) Under the fisheye isometric projection model, the relationship between image height h and half field of view Y_angle can be expressed as follows: h = f * Y_angle, where f is the equivalent focal length. Under isometric projection, the closer the fisheye image is to the circumferential image, the greater the compression and distortion, presenting a bulging effect in the middle. For each initial pixel in the fisheye region of interest, the initial ideal image height h can be determined based on the half field of view Y_angle and the equivalent focal length f corresponding to that initial pixel.

[0122] In one possible implementation, see Figure 6C As shown, the relationship between the initial ideal image height after distortion correction and the half-field angle is illustrated. When the half-field angle Y_angle is less than 90 degrees, to obtain the initial ideal image height h, based on the half-field angle Y_angle corresponding to the initial pixel and the equivalent focal length f, the initial ideal image height h can be determined using the following formula: h = f * tna(Y_angle). Furthermore, when the half-field angle Y_angle is not less than 90 degrees, the initial ideal image height h can be determined using the following formula: h = f * tan(Y_angle1) + k(Y_angle - Y_angle1), where Y_angle1 is the value closest to 90 degrees in the half-field angle table, such as 89.8 degrees in this example; k is a preset increase ratio, such as 0.2, meaning that for Y_angle exceeding 90 degrees, increasing the image height by a certain ratio will yield the initial ideal image height h.

[0123] (3) For each initial pixel in the fisheye region of interest, determine the initial ratio based on the initial ideal image height and the actual image height corresponding to the initial pixel. See formula (13) for an example of determining the initial ratio, where the initial ratio represents the ratio of the actual image height to the initial ideal image height.

[0124]

[0125] scale1 represents the initial scale corresponding to the initial pixel, Real_height represents the actual image height corresponding to the initial pixel, and Ideal_height represents the initial ideal image height corresponding to the initial pixel.

[0126] (4) Modulate the initial ratio using a preset modulation ratio to obtain the target ratio.

[0127] For example, since the initial ideal image height tends to infinity as the half-field angle Y_angle approaches 90 degrees, the initial scale 1 drops to 0 when the half-field angle Y_angle approaches 90 degrees. Therefore, the initial scale 1 can be modulated by a preset modulation scale k to obtain the target scale 2. See formula (14) for an example of modulating the initial scale scAle1 to obtain the target scale 2.

[0128] scale2=scale1*k+(1-k)k∈[0,1] Formula (14)

[0129] See Figure 6D The diagram illustrates the relationship between the initial scale (scale1) and the target scale (scale2). The upper curve corresponds to the initial scale (scale1), and the lower curve corresponds to the target scale (scale2). Figure 6D The example below shows the relationship between the initial scale 1 and the target scale 2, with a preset modulation ratio k of 0.5.

[0130] The preset modulation ratio k can be configured based on experience. When the preset modulation ratio k is closer to 1, scale2 is closer to scale1, the correction intensity is greater, and the ideal image height of the correction stretch is greater.

[0131] (5) For each initial pixel in the fisheye region of interest, the target ideal image height corresponding to the initial pixel is determined based on the target ratio and the actual image height corresponding to the initial pixel. The target ideal image height can represent the distance between the target pixel and the center pixel of the fisheye unfolded image.

[0132] See Equation (14) for an example of determining the ideal image height of the target; no restrictions are imposed on this. In Equation (14), Ideal_height represents the ideal image height of the target corresponding to the initial pixel, Real_height represents the actual image height corresponding to the initial pixel, and scale2 represents the target scale.

[0133]

[0134] Obviously, based on the modulated target ratio, the modulated target ideal image height can be recalculated. When the preset modulation ratio k is 0.5, the modulated target ideal image height can be found by referring to... Figure 6E As shown, Figure 6E It represents the relationship between the corrected actual image height, the ideal image height of the target, and the half field of view Y_angle, where f*Y_angle represents the actual image height and Ideal_height represents the modulated ideal image height of the target.

[0135] (6) For each initial pixel in the fisheye region of interest, based on the actual image height and the target ideal image height corresponding to the initial pixel, determine the target pixel in the fisheye unfolded image that corresponds to the initial pixel.

[0136] For example, since the actual image height represents the distance (e.g., pixel distance) between the initial pixel and the center pixel of the region of interest (ROI) in the fisheye region of interest, and the target ideal image height represents the distance (e.g., pixel distance) between the target pixel and the center pixel of the expanded fisheye image, the initial pixel can be found in the ROI based on this distance. Similarly, the target pixel can be found in the expanded fisheye image based on this distance. Thus, a mapping relationship can be obtained between the target pixel in the expanded fisheye image and the initial pixel in the ROI. Based on this mapping relationship, for each target pixel in the expanded fisheye image, the corresponding initial pixel can be found in the ROI. Furthermore, for each initial pixel in the ROI, the corresponding target pixel can also be found in the expanded fisheye image.

[0137] Step S23: For each target pixel in the fisheye unfolded image, determine the initial pixel corresponding to the target pixel in the fisheye region of interest, and determine the pixel value of the target pixel based on the pixel value of the initial pixel (e.g., use the pixel value of the initial pixel as the pixel value of the target pixel). Based on this, a fisheye unfolded image can be generated based on the pixel values ​​of all target pixels.

[0138] In one possible implementation, for each target pixel in the fisheye unfolded image, step S22 can be used to determine the initial pixel corresponding to the target pixel in the fisheye region of interest. In another possible implementation, a mapping relationship can be established between the target pixels in the fisheye unfolded image and the initial pixels in the fisheye region of interest. For each target pixel in the fisheye unfolded image, the initial pixel corresponding to the target pixel can be obtained by querying this mapping relationship.

[0139] For each target pixel in the fisheye unfolded image, after finding the initial pixel corresponding to the target pixel, the pixel value of the initial pixel can be used as the pixel value of the target pixel. The pixel values ​​of all target pixels then form the fisheye unfolded image, thus obtaining the fisheye unfolded image.

[0140] In summary, the region of interest (ROI) in a fisheye image can be expanded to obtain an expanded fisheye image. Referring to the above description, since the center point of the ROI is a preset expansion center point, located in the center of the fisheye sensor's acquisition area, after expanding the ROI into an expanded fisheye image, the center pixel of the expanded image also corresponds to the preset expansion center point, i.e., it is located in the center of the fisheye sensor's acquisition area, and the center pixel of the expanded fisheye image corresponds to the center pixel of the ROI.

[0141] For example, see Figure 6F The image shown is a schematic diagram of a fisheye image. After unfolding the fisheye image, an unfolded fisheye image can be obtained. See [link / reference]. Figure 6G The image shown is a schematic diagram of the fisheye unfolded image after distortion correction of the main screen. Clearly, in Figure 6G In the image, the main road is straightened and corrected, and the field of view at both ends is well preserved. In practical applications, only the main part of the road can be displayed; see [link to relevant documentation]. Figure 6H As shown.

[0142] Fifth, the processing method for the target object being located in the overlapping area (i.e., area a4) between the telephoto sensor 2 and the fisheye sensor, that is, the target object moves from area a3 to area a4.

[0143] When the target object moves from region a3 to region a4, since the target object is always within the field of view of the fisheye sensor, the fisheye sensor continuously acquires fisheye images of the target object and analyzes the target detection information b2 corresponding to the target object under the fisheye sensor based on these fisheye images. The target detection information b2 may include the unique identifier w1 of the target object, the attribute information of the target object, the position and time of the target object in region a1, the position and time of the target object in region a2, the position and time of the target object in region a3, and the position and time of the target object in region a4.

[0144] When the target object moves from region a3 to region a4, since the target object enters the field of view of the telephoto sensor 2, the telephoto sensor 2 can acquire a telephoto image of the target object and analyze the target detection information b3 corresponding to the target object under the telephoto sensor 2 based on the telephoto image. The target detection information b3 may include, but is not limited to, at least one of the following: a unique identifier assigned to the target object by the telephoto sensor 2 (denoted as identifier w3, this item is optional), the attribute information of the target object (this item is optional), the position of the target object and the time the target object is in that position, without any restrictions.

[0145] Based on the location and time in target detection information b2, and the location and time in target detection information b3, if the location in target detection information b2 and the location in target detection information b3 correspond to the same physical location at the same time, then it is determined that target detection information b2 and target detection information b3 belong to the same target object.

[0146] For example, if target detection information b2 includes location P5 and time point T2, it means that the target object is at location P5 at time point T2. Target detection information b3 includes location P6 and time point T2, indicating that the target object is at location P6 at time point T2. Based on this, if location P5 and location P6 correspond to the same physical location, then target detection information b2 and target detection information b3 are determined to belong to the same target object.

[0147] For example, regarding how to determine whether positions P5 and P6 correspond to the same physical location, the following method can be used: Pre-calibrate the calibration relationship between the telephoto sensor 2 and the fisheye sensor. Based on this calibration relationship, position P5 can be converted into the mapped position of the target object under the fisheye sensor. If this mapped position matches position P6, then position P5 and position P6 correspond to the same physical location; if the mapped position does not match position P6, then position P5 and position P6 do not correspond to the same physical location. Alternatively, based on the calibration relationship between the telephoto sensor 2 and the fisheye sensor, position P6 can be converted into the mapped position of the target object under the telephoto sensor 2. If this mapped position matches position P5, then position P5 and position P6 correspond to the same physical location; if the mapped position does not match position P5, then position P5 and position P6 do not correspond to the same physical location.

[0148] For example, after determining that target detection information b2 and target detection information b3 belong to the same target object, target detection information b2 can be transmitted to telephoto sensor 2, thereby associating target detection information b2 with target detection information b3, that is, fusing target detection information b2 with target detection information b3.

[0149] When fusing target detection information b2 and target detection information b3, the telephoto sensor 2 can update the unique identifier of the target object to identifier w1, update the attribute information of the target object in target detection information b2 to target detection information b3, and update the position and time in target detection information b2 to target detection information b3. In this way, target detection information b3 can include the trajectory of the target object when it is in region a1, the trajectory of the target object when it is in region a2, the trajectory of the target object when it is in region a3, and the trajectory of the target object when it is in region a4, thereby realizing the trajectory fusion of the target object in each region.

[0150] For example, when the target object moves from region a3 to region a4, the fisheye image acquired by the fisheye sensor needs to be converted into a fisheye unfolded image. Based on the fisheye unfolded image, the position of the target object is determined. This process can be referred to in point three, except that when dividing the region of interest from the fisheye image, the preset unfolded center point position is the preset unfolded center point position corresponding to telephoto sensor 2, not the preset unfolded center point position corresponding to telephoto sensor 1. For example, see... Figure 3A As shown, a certain position in the upper region is configured as the preset unfolding center point position corresponding to telephoto sensor 1, and a certain position in the lower region is configured as the preset unfolding center point position corresponding to telephoto sensor 2. See [link to documentation]. Figure 3CAs shown, a certain position in the left area is configured as the preset unfolding center point position corresponding to the telephoto sensor 1, and a certain point in the right area is configured as the preset unfolding center point position corresponding to the telephoto sensor 2.

[0151] Sixth, the processing method for the target object being located in the acquisition area (i.e., area a5) of the telephoto sensor 2.

[0152] When the target object moves from region a4 to region a5, since the target object remains within the field of view of telephoto sensor 2, telephoto sensor 2 continuously acquires telephoto images of the target object. Based on these telephoto images, it analyzes the target detection information b3 corresponding to the target object under telephoto sensor 2. Target detection information b3 may include the target object's unique identifier w1, the target object's attribute information, the target object's position and time in region a1, region a2, region a3, region a4, and region a5. In summary, target detection information b3 may include the target object's trajectory in each region, thereby achieving trajectory fusion across regions. These trajectories correspond to the same unique identifier w1.

[0153] As can be seen from the above technical solutions, in this embodiment, the use of a telephoto sensor and a fisheye sensor can increase the camera's field of view, enabling the camera to acquire images within a larger field of view and achieve full coverage of target detection blind spots. This achieves blind-spot-free target detection across the entire road, avoiding situations where the real-time position of the target object cannot be obtained when it is in certain locations. When the target object enters the overlapping area between the telephoto sensor and the fisheye sensor, the target detection information corresponding to the target object under the telephoto sensor can be associated with the target detection information corresponding to the target object under the fisheye sensor, enabling real-time detection even when the target object is moving. Only by establishing a mapping relationship between the fisheye image and the expanded fisheye image can real-time position mapping be achieved. Cascading at T-junctions can also be considered to achieve omnidirectional, blind-spot-free target detection.

[0154] For example, the target detection method described in the embodiments of this application can be applied to the following scenarios.

[0155] 1. Acquisition of face and license plate information for electric vehicles. When the license plate of an electric vehicle is at the rear and the face is on the side of the vehicle, a single camera cannot acquire both the face and the license plate. Using the camera described in this application, the face of the target object can be acquired through telephoto sensor 1, and the license plate of the electric vehicle can be acquired through telephoto sensor 2. Furthermore, the overall association of the target object is achieved through telephoto sensor 1, a fisheye sensor, and telephoto sensor 2, thereby enabling the association between the electric vehicle license plate and the face. See [link to relevant documentation]. Figure 7AAs shown, an example of this application scenario is illustrated.

[0156] 2. Parking Space Management. Long-focus sensor 1 and / or long-focus sensor 2 are used to detect parking areas at a distance and to recognize vehicle license plates, while the fisheye sensor is used to detect parking areas below and to recognize vehicle license plates. By associating the fisheye sensor with the target objects of each long-focus sensor, the license plate information of the vehicles parked below can also be obtained. See [link to documentation]. Figure 7B As shown, an example of this application scenario is illustrated.

[0157] 3. Target Video Relay. Telephoto sensor 1 and / or telephoto sensor 2 are used for target object detection and feature recognition. Once the set target information is detected, it can upload the information to the platform. After the target object is detected, relay detection of the target object is performed between the telephoto sensor and the fisheye sensor, while continuously uploading information to the platform. The uploaded information may include, but is not limited to, the following: the identifier of the preset target, the ID of the detected target, the channel of the detected target, and the position of the detected target in the frame. Furthermore, the platform can display and dynamically switch the corresponding channel's image based on the target information.

[0158] Based on the same concept as the above method, this application proposes a target detection device for use with a camera that achieves full coverage of the target detection blind zone. The camera includes a telephoto sensor and a fisheye sensor. See [link to relevant documentation]. Figure 8 The diagram shown is a structural schematic of the device, which may include:

[0159] The determination module 81 is used to determine, when a target object enters the overlapping area between the telephoto sensor and the fisheye sensor, the first target detection information corresponding to the target object under the telephoto sensor, the second target detection information corresponding to the target object under the fisheye sensor, and determine that the first target detection information and the second target detection information belong to the same target object;

[0160] The processing module 82 is configured to, if the target object enters the overlapping area from the acquisition area of ​​the telephoto sensor, transmit the first target detection information to the fisheye sensor so that the fisheye sensor associates the first target detection information with the second target detection information under the fisheye sensor; and if the target object enters the overlapping area from the acquisition area of ​​the fisheye sensor, transmit the second target detection information to the telephoto sensor so that the telephoto sensor associates the second target detection information with the first target detection information under the telephoto sensor.

[0161] In one possible implementation, the first target detection information includes a first position of the target object under the telephoto sensor, and the second target detection information includes a second position of the target object under the fisheye sensor. When the determining module 81 determines that the first target detection information and the second target detection information belong to the same target object, it is specifically used for:

[0162] If the first position and the second position correspond to the same physical location at the same time, then the first target detection information and the second target detection information are determined to belong to the same target object.

[0163] In one possible implementation, the determining module 81 is further configured to:

[0164] The telephoto sensor acquires a telephoto image, and the target object is determined at a first position corresponding to the telephoto sensor based on the telephoto image; the fisheye sensor acquires a fisheye image, and the target object is determined at a second position corresponding to the fisheye sensor based on the fisheye image.

[0165] Based on the calibration relationship between the telephoto sensor and the fisheye sensor, the first position is converted into the corresponding mapped position of the target object under the fisheye sensor. If the mapped position matches the second position, it is determined that the first position and the second position correspond to the same physical position.

[0166] In one possible implementation, when the determining module 81 determines the second position of the target object under the fisheye sensor based on the fisheye image, it is specifically used for:

[0167] Based on the preset unfolding center point position, preset unfolding width, and preset unfolding height, a first fisheye region of interest is divided from the fisheye image, and the first fisheye region of interest is unfolded to obtain a first fisheye unfolded image; wherein, the center pixel of the first fisheye unfolded image corresponds to the preset unfolding center point position, and the preset unfolding center point position is located within the acquisition area of ​​the telephoto sensor.

[0168] The second position corresponding to the target object is determined based on the first fisheye unfolded image.

[0169] In one possible implementation, the determining module 81 expands the first fisheye region of interest to obtain the expanded first fisheye image, specifically for:

[0170] For each initial pixel in the first fisheye unfolded image, the coordinates of the projection point corresponding to the initial pixel on the fisheye image hemisphere are determined based on the equivalent focal length; the azimuth and incident angle corresponding to the initial pixel are determined based on the projection point coordinates; and the target pixel corresponding to the initial pixel in the first fisheye region of interest is determined based on the equivalent focal length, the azimuth, and the incident angle.

[0171] The pixel value of the initial pixel is determined based on the pixel value of the target pixel.

[0172] The first fisheye unfolded image is generated based on the pixel values ​​of all initial pixels.

[0173] Specifically, when determining the equivalent focal length, the determining module 81 is used to: determine the equivalent focal length based on the effective width of the first fisheye region of interest and the maximum field of view of the fisheye sensor.

[0174] In one possible implementation, the determining module 81 is further configured to: if the target object enters the acquisition area of ​​the fisheye sensor, acquire a fisheye image through the fisheye sensor, and determine the third position of the target object under the fisheye sensor based on the fisheye image; when determining the third position of the target object under the fisheye sensor based on the fisheye image, the module is configured to:

[0175] A second fisheye region of interest is defined from the fisheye image, and the second fisheye region of interest is expanded to obtain a second fisheye expanded image; wherein, the center pixel of the second fisheye expanded image corresponds to the center pixel of the second fisheye region of interest.

[0176] The third position corresponding to the target object is determined based on the second fisheye unfolded image.

[0177] In one possible implementation, the determining module 81 expands the second fisheye region of interest to obtain the expanded second fisheye image, specifically for the following purposes:

[0178] For each initial pixel in the second fisheye region of interest, the actual image height corresponding to the initial pixel is obtained by querying the configured mapping table based on the half field of view corresponding to the initial pixel.

[0179] The target ideal image height corresponding to the initial pixel is determined based on the half field of view.

[0180] Based on the actual image height and the target ideal image height, a target pixel point corresponding to the initial pixel point in the second fisheye unfolded image is determined; wherein, the actual image height represents the distance between the initial pixel point and the center pixel point of the second fisheye region of interest, and the target ideal image height represents the distance between the target pixel point and the center pixel point of the second fisheye unfolded image;

[0181] The pixel value of the target pixel is determined based on the pixel value of the initial pixel.

[0182] The second fisheye unfolded image is generated based on the pixel values ​​of all target pixels.

[0183] In one possible implementation, when the determining module 81 determines the target ideal image height corresponding to the initial pixel based on the half field of view, it is specifically used to: determine the initial ideal image height corresponding to the initial pixel based on the half field of view and the equivalent focal length; determine an initial ratio based on the initial ideal image height and the actual image height; modulate the initial ratio using a preset modulation ratio to obtain a target ratio; and determine the target ideal image height based on the target ratio and the actual image height.

[0184] Based on the same concept as the above method, this application proposes a camera for achieving full coverage of target detection blind spots. The camera includes a telephoto sensor, a fisheye sensor, and a processor; wherein: the telephoto sensor is used to acquire telephoto images and send the telephoto images to the processor; the fisheye sensor is used to acquire fisheye images and send the fisheye images to the processor; based on the telephoto image and the fisheye image, the processor is used to execute:

[0185] When a target object enters the overlapping area between the telephoto sensor and the fisheye sensor, first target detection information corresponding to the target object under the telephoto sensor is determined based on the telephoto image, and second target detection information corresponding to the target object under the fisheye sensor is determined based on the fisheye image. It is determined that the first target detection information and the second target detection information belong to the same target object. If the target object enters the overlapping area from the acquisition area of ​​the telephoto sensor, the first target detection information is transmitted to the fisheye sensor so that the fisheye sensor associates the first target detection information with the second target detection information under the fisheye sensor. If the target object enters the overlapping area from the acquisition area of ​​the fisheye sensor, the second target detection information is transmitted to the telephoto sensor so that the telephoto sensor associates the second target detection information with the first target detection information under the telephoto sensor.

[0186] Based on the same concept as the method described above, this application proposes a camera, see [link to relevant documentation]. Figure 9 As shown, the camera includes a processor 91 and a machine-readable storage medium 92, the machine-readable storage medium 92 storing machine-executable instructions that can be executed by the processor 91; the processor 91 is used to execute the machine-executable instructions to implement the target detection method disclosed in the above example of this application.

[0187] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the target detection method disclosed in the above examples of this application.

[0188] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0189] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0190] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0191] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0192] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0193] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0194] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0195] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A target detection method, characterized in that, A method for using a camera to achieve full coverage of blind spots in target detection, the camera including a telephoto sensor and a fisheye sensor, the method comprising: When a target object enters the overlapping area between the telephoto sensor and the fisheye sensor, the first target detection information corresponding to the target object under the telephoto sensor is determined, the second target detection information corresponding to the target object under the fisheye sensor is determined, and the first target detection information and the second target detection information belong to the same target object. If the target object enters the overlapping area from the acquisition area of ​​the telephoto sensor, the first target detection information is transmitted to the fisheye sensor so that the fisheye sensor associates the first target detection information with the second target detection information under the fisheye sensor. If the target object enters the overlapping area from the acquisition area of ​​the fisheye sensor, the second target detection information is transmitted to the telephoto sensor so that the telephoto sensor associates the second target detection information with the first target detection information under the telephoto sensor. If the target object enters the acquisition area of ​​the fisheye sensor, a fisheye image is acquired by the fisheye sensor, a second fisheye region of interest is delineated from the fisheye image, the second fisheye region of interest is expanded to obtain a second expanded fisheye image; and a third position of the target object under the fisheye sensor is determined based on the second expanded fisheye image. Specifically, for each initial pixel in the second fisheye region of interest, the actual image height corresponding to the initial pixel is obtained by querying a configured mapping table based on the half field of view corresponding to the initial pixel; the target ideal image height corresponding to the initial pixel is determined based on the half field of view; the target pixel in the second fisheye unfolded image corresponding to the initial pixel is determined based on the actual image height and the target ideal image height; wherein, the actual image height represents the distance between the initial pixel and the center pixel of the second fisheye region of interest, and the target ideal image height represents the distance between the target pixel and the center pixel of the second fisheye unfolded image; the pixel value of the target pixel is determined based on the pixel value of the initial pixel; and the second fisheye unfolded image is generated based on the pixel values ​​of all target pixels.

2. The method according to claim 1, characterized in that, The first target detection information includes a first position of the target object under the telephoto sensor, and the second target detection information includes a second position of the target object under the fisheye sensor. Determining that the first target detection information and the second target detection information belong to the same target object includes: If the first position and the second position correspond to the same physical location at the same time, then the first target detection information and the second target detection information are determined to belong to the same target object.

3. The method according to claim 2, characterized in that, The method further includes: The telephoto sensor acquires a telephoto image, and the target object is determined at a first position corresponding to the telephoto sensor based on the telephoto image; the fisheye sensor acquires a fisheye image, and the target object is determined at a second position corresponding to the fisheye sensor based on the fisheye image. Based on the calibration relationship between the telephoto sensor and the fisheye sensor, the first position is converted into the corresponding mapped position of the target object under the fisheye sensor. If the mapped position matches the second position, it is determined that the first position and the second position correspond to the same physical position.

4. The method according to claim 3, characterized in that, Determining the second position of the target object under the fisheye sensor based on the fisheye image includes: Based on the preset unfolding center point position, preset unfolding width, and preset unfolding height, a first fisheye region of interest is divided from the fisheye image, and the first fisheye region of interest is unfolded to obtain a first fisheye unfolded image; wherein, the center pixel of the first fisheye unfolded image corresponds to the preset unfolding center point position, and the preset unfolding center point position is located within the acquisition area of ​​the telephoto sensor. The second position corresponding to the target object is determined based on the first fisheye unfolded image.

5. The method according to claim 4, characterized in that, The step of expanding the first fisheye region of interest to obtain the expanded first fisheye image includes: For each initial pixel in the first fisheye unfolded image, the coordinates of the projection point corresponding to the initial pixel on the fisheye image hemisphere are determined based on perspective projection; the azimuth and incident angle corresponding to the initial pixel are determined based on the projection point coordinates; and the target pixel corresponding to the initial pixel in the first fisheye region of interest is determined based on the equivalent focal length, the azimuth, and the incident angle. The pixel value of the initial pixel is determined based on the pixel value of the target pixel. The first fisheye unfolded image is generated based on the pixel values ​​of all initial pixels.

6. The method according to claim 5, characterized in that, The method for determining the equivalent focal length includes: determining the equivalent focal length based on the effective width of the first fisheye region of interest and the maximum field of view of the fisheye sensor.

7. The method according to claim 1, characterized in that, Determining the target ideal image height corresponding to the initial pixel based on the half field of view includes: The initial ideal image height corresponding to the initial pixel is determined based on the half field of view and the equivalent focal length. The initial scale is determined based on the initial ideal image height and the actual image height; The initial ratio is modulated using a preset modulation ratio to obtain the target ratio; The target ideal image height is determined based on the target scale and the actual image height.

8. A camera, characterized in that, The camera is used to achieve full coverage of target detection blind spots, and the camera includes a telephoto sensor, a fisheye sensor, and a processor; wherein: The telephoto sensor is used to acquire telephoto images and send the telephoto images to the processor; The fisheye sensor is used to acquire fisheye images and send the fisheye images to the processor; Based on the telephoto image and the fisheye image, the processor is configured to execute: When a target object enters the overlapping area between the telephoto sensor and the fisheye sensor, first target detection information corresponding to the target object under the telephoto sensor is determined based on the telephoto image, and second target detection information corresponding to the target object under the fisheye sensor is determined based on the fisheye image. It is determined that the first target detection information and the second target detection information belong to the same target object. If the target object enters the overlapping area from the acquisition area of ​​the telephoto sensor, the first target detection information is transmitted to the fisheye sensor so that the fisheye sensor associates the first target detection information with the second target detection information under the fisheye sensor. If the target object enters the overlapping area from the acquisition area of ​​the fisheye sensor, the second target detection information is transmitted to the telephoto sensor so that the telephoto sensor associates the second target detection information with the first target detection information under the telephoto sensor. If the target object enters the acquisition area of ​​the fisheye sensor, a fisheye image is acquired by the fisheye sensor, a second fisheye region of interest is delineated from the fisheye image, the second fisheye region of interest is expanded to obtain a second expanded fisheye image; and a third position of the target object under the fisheye sensor is determined based on the second expanded fisheye image. Specifically, for each initial pixel in the second fisheye region of interest, the actual image height corresponding to the initial pixel is obtained by querying a configured mapping table based on the half field of view corresponding to the initial pixel; the target ideal image height corresponding to the initial pixel is determined based on the half field of view; the target pixel in the second fisheye unfolded image corresponding to the initial pixel is determined based on the actual image height and the target ideal image height; wherein, the actual image height represents the distance between the initial pixel and the center pixel of the second fisheye region of interest, and the target ideal image height represents the distance between the target pixel and the center pixel of the second fisheye unfolded image; the pixel value of the target pixel is determined based on the pixel value of the initial pixel; and the second fisheye unfolded image is generated based on the pixel values ​​of all target pixels.

Images