Target detection method for tail end of visual field of fisheye camera and related equipment

Abstract

The invention provides a method for detecting a target at the tail end of the view of a fisheye camera and related equipment. The method comprises the following steps: acquiring an image acquired by the fisheye camera; obtaining a bounding box of at least one target object completely located in a fisheye view field of the fisheye camera from the image; obtaining and storing the aspect ratios of the bounding boxes of different target objects; and when only part of the target object is within the fisheye view field, obtaining a prediction frame of the target object at the moment according to the aspect ratio of the stored bounding box of the target object. According to the method, when the target object leaves the tail end of the view of the fisheye camera, the target bounding box is predicted according to the pixel height-width ratio of the original bounding box, and finally, a more real prediction box of pixel plane coordinates can be identified, so that the target detection positioning error generated by the target object at the edge of the image is eliminated, and the detection accuracy is improved. The positioning precision of the fisheye camera is improved; meanwhile, the sensing range of the fisheye camera is expanded, and the matching success rate of multi-sensor fusion is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for detecting an object at the end of a fisheye camera's field of view and related equipment. Background Art

[0002] The roadside perception system uses a variety of sensors such as visual sensors, millimeter-wave radars, and lidars, combined with edge computing devices, to monitor the current road condition data in real time. In addition, the roadside perception system can provide drivers with real-time information about the road environment and early warning information related to travel safety based on the current road condition data, and provide monitoring and prediction of the road traffic environment for relevant transportation departments.

[0003] As a key link in the integration of vehicle, road and cloud, the roadside perception system has gradually diversified its system solutions. As a key solution for intersection perception, fisheye cameras are used for blind spot filling. The blind spot filling method for a single direction is: camera facing the intersection → fisheye camera blind spot filling → camera facing the road section. This introduces the fisheye camera calibration solution, which can be performed according to the traditional calibration solution. However, the fisheye camera needs to have a certain overlap area with the intersection camera and road section camera set up in front and behind to ensure that it can be successfully integrated with the fisheye camera. This requires deployment with extremely high requirements for camera angles. In the case of large-scale deployment, it is difficult to meet the deployment requirements, and the focal length of many cameras cannot overlap with the field of view of the fisheye camera, which affects the image fusion between the fisheye camera and other cameras, resulting in a larger positioning error of the fisheye camera for the detection target in the terminal field of view. Summary of the invention

[0004] The present invention provides a camera attitude adaptive calibration method based on feature matching and related equipment, aiming to solve the problem of detection and positioning deviation of target objects at the edge of images in the prior art.

[0005] In order to achieve the above purpose and other advantages, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a method for detecting an object at the end of a fisheye camera's field of view, comprising:

[0007] Get the image captured by the fisheye camera;

[0008] Acquire a bounding box of at least one target object completely within the fisheye field of view of the fisheye camera from the image;

[0009] Obtaining and storing different aspect ratios of the bounding boxes of the target object;

[0010] When only a portion of the target object is within the fisheye field of view, the predicted frame of the target object at this time is obtained according to the aspect ratio of the stored bounding box of the target object.

[0011] According to a method for detecting a target at the end of a fisheye camera's field of view provided by the present invention, the step of obtaining and storing different aspect ratios of the bounding boxes of the target objects comprises:

[0012] Obtaining the pixel size of the image;

[0013] Acquire, according to the pixel size, a first pixel width and a first pixel height of a bounding box of different target objects in the image;

[0014] Obtaining an aspect ratio of a bounding box of the target object according to the first pixel width and the first pixel height;

[0015] The aspect ratio is associated with the target object and then stored.

[0016] According to a method for detecting an object at the end of a fisheye camera's field of view provided by the present invention, the step of obtaining a predicted frame of the target object at this time according to the aspect ratio of the stored bounding box of the target object comprises:

[0017] When only a portion of the target object is within the fisheye field of view, obtaining a second pixel width of the edge of the image occupied by the target object;

[0018] Acquire, according to the second pixel width and the aspect ratio, a predicted pixel height of a prediction box for drawing the target object;

[0019] A prediction box for marking the target object is drawn on the image according to the second pixel width and the predicted pixel height, wherein at least a portion of the prediction box is outside the fisheye field of view.

[0020] According to the present invention, a method for detecting an object at the end of the field of view of a fisheye camera further includes: drawing a center point of a prediction box for marking the target object on the image according to the second pixel width and the predicted pixel height, specifically including:

[0021] Taking the upper left corner of the image as the origin of pixel coordinates, obtaining the pixel coordinates of the corner points of the prediction frame within the fisheye field of view;

[0022] According to the corner point pixel coordinates, the second pixel width and the predicted pixel height, the pixel coordinates of the center point of the predicted frame are calculated and marked.

[0023] According to a method for detecting a target at the end of a fisheye camera's field of view provided by the present invention, the corner point pixel coordinates include: pixel coordinates of a lower left corner point or a lower right corner point of the prediction frame within the fisheye field of view.

[0024] According to a method for detecting an object at the end of the field of view of a fisheye camera provided by the present invention, after the step of acquiring an image captured by the fisheye camera, the method further comprises:

[0025] Obtaining a distortion correction coefficient of the fisheye camera;

[0026] Based on the distortion correction coefficient, distortion correction is performed on the image.

[0027] A method for detecting a target at the end of a fisheye camera's field of view provided by the present invention further includes: tracking and detecting the target object based on the prediction frame.

[0028] In a second aspect, the present invention provides an electronic device, the electronic device comprising:

[0029] one or more processors;

[0030] and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the method for detecting an object at the end of the field of view of a fisheye camera as described above.

[0031] In a third aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of detecting an object at the end of the field of view of a fisheye camera as described in any one of the above.

[0032] In a fourth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of detecting an object at the end of the field of view of a fisheye camera as described in any one of the above.

[0033] The present invention provides a method and related equipment for detecting a target at the end of the field of view of a fisheye camera, which obtains an image captured by a fisheye camera; obtains a bounding box of at least one target object completely within the field of view of the fisheye camera from the image; obtains and stores the aspect ratios of the bounding boxes of different target objects; and when only a portion of the target object is within the field of view of the fisheye, obtains the prediction box of the target object at this time according to the aspect ratio of the bounding box stored by the target object. When the target object leaves the end of the field of view of the fisheye camera and the identified target object is not a complete object, the present invention predicts the target bounding box according to the pixel aspect ratio of the original bounding box, and finally identifies a prediction box of more realistic pixel plane coordinates, thereby eliminating the detection and positioning error generated at the edge of the image due to the limitation of the camera field of view, and improving the positioning accuracy of the fisheye camera; at the same time, the perception range of the fisheye camera is expanded, the perception overlap area with surrounding cameras, radars and other sensors is increased, and the matching success rate of the fusion with surrounding multi-sensors is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other implementation methods can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 It is a schematic diagram of using a fisheye camera to fill in the blind area of ​​the visual field in the prior art;

[0036] Figure 2a to 2c It is a schematic diagram of the boundary box and positioning coordinate points of the vehicle identified in the prior art when the vehicle gradually leaves the fisheye field of view according to the vehicle's driving direction within the fisheye field of view;

[0037] Figure 3 is a schematic flow chart of a method for detecting an object at the end of a fisheye camera's field of view provided by an embodiment of the present invention;

[0038] Figure 4a-4b It is an effect diagram of a bounding box identified by predicting a target vehicle according to a previously stored aspect ratio provided by an embodiment of the present invention;

[0039] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0040] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows.

[0041] It should be noted that it is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present invention can be combined with other embodiments without conflict. Unless otherwise defined, the technical terms or scientific terms involved in the present invention should be the usual meanings understood by people with ordinary skills in the technical field to which the present invention belongs. The words "one", "a", "a", "the" and the like involved in the present invention do not indicate a quantitative limitation and may represent the singular or plural. The terms "including", "comprising", "having" and any of their variations involved in the present invention are intended to cover non-exclusive inclusions; the terms "first", "second", "third", etc. involved in the present invention are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0042] A fisheye camera is a camera equipped with a fisheye lens. The diameter of the front lens of this lens is very short and it bulges out in a parabolic shape toward the front of the lens, similar in shape to a fisheye. Since the focal length of the lens is extremely short, the shooting angle is close to or equal to 180°, which enables ultra-wide-angle shooting compared to ordinary cameras. An omnidirectional camera or a spherical camera is also a type of fisheye camera. The fisheye camera can be set to look down at the detection object area from above the detection object area. The optical axis of the fisheye camera can be set vertically downward or tilted relative to the vertical direction.

[0043] The road section under the vehicle-road collaboration is equipped with a camera facing the road section and a camera facing the intersection. Due to the camera's orientation and angle restrictions, a blind spot is formed between the two cameras. The fisheye camera is set between the two cameras. The fisheye camera is mainly used to supplement the blind spot of the two cameras to provide comprehensive monitoring coverage, such as Figure 1 shown.

[0044] The images taken by the fisheye camera are uploaded to the integrated perception system of the vehicle, road and cloud. By accurately identifying the position and size of the object, more accurate object tracking, behavior analysis and other functions can be achieved, providing an important basis for subsequent decision-making and control. Therefore, marking and locating the target object in the image with a two-dimensional bounding box (2D box) is a key step in computer vision tasks. The bounding box is a closed figure or frame line used to enclose the target object, generally a rectangular box. The rectangular bounding box is represented by determining the coordinates of the upper left corner and lower right corner of the rectangular box (x1, y1, x2, y2) or by the coordinates of the center point and the width and height (x, y, w, h) of the rectangular box. For the detection of target objects in the image, a deep learning model (such as a convolutional neural network CNN) can be used to identify the objects in the image through model training and learning to accurately predict the bounding box of the object.

[0045] Taking the identification of vehicles on the road as an example, the calibration of the fisheye camera needs to rely on the pixel plane coordinates of the identified vehicle boundary box. Generally, the pixel coordinates of the center point of the boundary box are defaulted as the positioning coordinate point of the vehicle.

[0046] Referring to Figure 2, a group of schematic diagrams showing a vehicle in an image that is within the fisheye field of view and gradually leaves the fisheye field of view in the direction of vehicle travel are used as an example for explanation. The applicant found that when the vehicle gradually leaves the fisheye field of view, the identified bounding box will become gradually narrower due to the limitation of the camera field of view, resulting in the actual deviation of the positioning coordinate points obtained, which reduces the positioning accuracy of the fisheye camera and is difficult to apply in the field of vehicle-road-cloud integrated collaboration. Therefore, there is an urgent need for a target detection method and related equipment at the end of the fisheye camera field of view, which aims to ultimately predict a target bounding box with more realistic pixel plane coordinates, so as to eliminate the detection and positioning errors at the edge of the image due to the limitation of the camera field of view, and provide accurate data support for subsequent decision-making and control.

[0047] Reference Figure 3 As shown, an embodiment of the present invention provides a method for detecting a target at the end of a fisheye camera's field of view, comprising:

[0048] Step S1: Acquire an image captured by a fisheye camera.

[0049] Among them, the fisheye camera can provide a wider camera field of view than a standard camera, usually with a 180° field of view, and fewer cameras can be used to achieve coverage of larger scenes. Before the fisheye camera is installed, it is necessary to calibrate the fisheye camera to obtain the distortion correction coefficient. The calibration method can be calibrated by chessboard calibration, lateral expansion method, and longitude and latitude method. However, since the scene captured by the fisheye camera is very different from the scene in the real world, the closer the target object in the image is to the edge of the image, the greater the deformation, so the captured image needs to be distorted. In this embodiment, after step S1, it also includes:

[0050] Obtain the distortion correction coefficient of the fisheye camera; based on the distortion correction coefficient, perform distortion correction on the image. This effectively reduces the deformation of the target object at the edge of the image. In other embodiments, a dedistortion algorithm based on the Brownian camera model can also be used to accurately correct the image according to the parameters of the camera.

[0051] Step S2: obtaining a bounding box of at least one target object completely within the fisheye field of view of the fisheye camera from the image.

[0052] It should be noted that in some complex scenes, when multiple target objects need to be processed in real time, the essence of target detection is the positioning and recognition of multiple targets, that is, to locate multiple target objects in the image, including classification and positioning. For example, when performing target detection on an image, there are several target objects in the image, and each target object has a corresponding rectangular bounding box.

[0053] Exemplarily, the target object is an object in an actual scene, which may be a vehicle, a pedestrian, or an animal, etc. In this embodiment, a vehicle is used as the target object for detection.

[0054] Step S3: Obtain and store the aspect ratios of the bounding boxes of different target objects.

[0055] In this embodiment, step S3 specifically includes:

[0056] Step S301: obtaining the pixel size of the image;

[0057] Step S302: acquiring first pixel widths and first pixel heights of bounding boxes of different target objects in the image according to the pixel size;

[0058] Step S303: Obtaining the aspect ratio of the bounding box of the target object according to the first pixel width and the first pixel height;

[0059] Step S304: associate the aspect ratio with the target object and store them.

[0060] Specifically, the size of the image captured within the fisheye field of view is set to 1280*1280 pixels, that is, a square image, the width and height of the image are equal, both 1280 pixels. In this embodiment, the pixel coordinate system of the image is set with the upper left corner of the image as the origin, the right as the positive direction of the X axis, and the downward as the positive direction of the Y axis. When the body of the vehicle is completely within the fisheye field of view and is no longer at the edge of the image, the bounding box of the vehicle in the image is identified, and the first pixel width X1 and the first pixel height Y1 of the bounding box are obtained according to the set pixel size of the image, and then the first pixel width X1 and the first pixel height Y1 are divided to calculate the aspect ratio Z=X1 / Y1 of the vehicle bounding box, and store it. In other embodiments, other corner points of the image can be used as the origin, and the positive directions of the X axis and the Y axis in the pixel coordinate system are set according to actual conditions, which are not specifically limited here.

[0061] It should be noted that different target objects, or several categories of the same target object (such as vehicles including small cars, large cars, light trucks, heavy trucks, etc.), have different sizes and their corresponding bounding boxes have different sizes. Therefore, for vehicles of the same category with similar or identical sizes, only the aspect ratio of one of the bounding boxes can be obtained and associated with the vehicles of the same category. For vehicles of different categories with large size differences, different aspect ratios are obtained respectively, and the different aspect ratios are associated with the corresponding categories of vehicles, and the associated aspect ratios are stored.

[0062] Step S4: When only a portion of the target object is within the fisheye field of view, the predicted frame of the target object at this time is obtained according to the aspect ratio of the stored bounding box of the target object.

[0063] In this embodiment, step S4 specifically includes:

[0064] Step S401: when only a portion of the target object is within the fisheye field of view, obtaining a second pixel width of the edge of the image occupied by the target object;

[0065] Step S402: acquiring a predicted pixel height of a prediction frame for drawing a target object according to the second pixel width and the aspect ratio;

[0066] Step S403: drawing a prediction box for marking the target object on the image according to the second pixel width and the predicted pixel height, wherein at least a portion of the prediction box is outside the fisheye field of view.

[0067] Specifically, when the vehicle starts to leave the camera field of view and only part of the vehicle body is within the fisheye field of view, the second pixel width X2 of the edge of the image occupied by the vehicle is obtained. According to the second pixel width X2, the previously stored aspect ratio is used to calculate the predicted pixel height Y2 of the bounding box to be predicted according to the current second pixel width X2, and the calculation formula is: Y2 = X2 / Z. According to the second pixel width X2 and the predicted pixel height Y2, a predicted box for marking the vehicle can be drawn on the image, such as Figure 4a and 4b shown.

[0068] It should be noted that although the image is corrected in the above steps, it is still impossible to overcome the problem that the target object does not deform at the edge of the image. Therefore, when the vehicle is at the edge of the image, in order to calculate accurately, it is necessary to obtain the second pixel width X2 of the image edge occupied by the vehicle. However, in some scenarios where accurate calculation is not required, the first pixel width X1 can also be used as the second pixel width X2 for calculation, without the need to re-obtain the second pixel width X2 of the image edge occupied by the vehicle.

[0069] As we all know, the angle adjustment for fisheye cameras is relatively sensitive, and a slight angle adjustment can easily cause a large change in the camera's viewing angle. In order to debug the overlapping area between the fisheye camera and the surrounding multi-sensors, the installer needs to spend a lot of time to debug. The prediction frame finally drawn using the embodiment of the present invention has a part of the prediction frame outside the fisheye field of view. There is no need to deliberately adjust the camera angle. The camera only needs to be leveled to achieve a fixed overlapping area with the surrounding multi-sensors, thereby reducing the labor cost of large-scale deployment.

[0070] Moreover, at least part of the prediction frame is outside the fisheye field of view, which expands the camera's perception range. For example, the length of a general vehicle is 4 to 6 meters, and its perception range is also expanded by 4 to 6 meters on the original basis. In this way, the overlap area with the camera device facing the intersection or the road section is increased. The larger the overlap area, the easier it is to successfully match, which improves the matching success rate with the surrounding multi-sensor fusion.

[0071] In this embodiment, the method further includes: drawing a center point of a prediction box for marking the target object on the image according to the second pixel width and the predicted pixel height, specifically including:

[0072] Taking the upper left corner of the image as the origin of pixel coordinates, obtain the pixel coordinates of the corner points of the prediction frame within the fisheye field of view;

[0073] According to the pixel coordinates of the corner point, the second pixel width and the predicted pixel height, the pixel coordinates of the center point of the predicted box are calculated and marked.

[0074] Specifically, according to the origin of the pixel coordinate system set in the above steps and the positive directions of the X-axis and the Y-axis, the pixel coordinates of the lower left corner or the lower right corner of the prediction frame in the image within the fisheye field of view are obtained. For example, the pixel coordinates of the lower left corner are (A1, B1). According to the second pixel width X2 and the predicted pixel height Y2, the calculation formula of the pixel coordinates (Cx, Cy) of the center point of the prediction frame is: Cx = A1 + X2 / 2, Cy = B1-Y2 / 2.

[0075] If the pixel coordinates of the lower right corner point are (A2, B2) for explanation, based on the second pixel width X2 and the predicted pixel height Y2, the calculation formula for the pixel coordinates (Cx, Cy) of the center point of the predicted frame is: Cx=A2-X2 / 2, Cy=B2-Y2 / 2.

[0076] Mark the pixel coordinates of the center point of the calculated prediction box, such as Figure 4a and 4b As shown. It can be seen that as long as the prediction box of the identified vehicle does not completely leave the camera field of view, the size of the identified prediction box will basically not change because the originally stored aspect ratio remains unchanged, and its center point used for positioning will basically not be affected, and no positioning deviation will occur. Compared with the recognition and detection scheme using the prior art, which will make the boundary box gradually narrow due to the limitation of the camera field of view, the technical solution designed by the present invention significantly reduces the positioning error and increases the positioning range.

[0077] In this embodiment, the method further includes: tracking and detecting the target object based on the prediction frame. In the fields of intelligent transportation systems and intelligent monitoring systems that integrate vehicles, roads and clouds, the video sequence to be tracked uploaded by the fisheye camera is input into the target tracking algorithm, the target frame of the target object in the image frame is drawn, the features of each target frame are extracted, and each target frame is evaluated using a classifier. The parameters and strategies of the tracking algorithm are adjusted according to the actual situation to complete the judgment and tracking of the target, thereby realizing dynamic real-time tracking and positioning of the target object.

[0078] In summary, in order to cope with the problem that the recognition and detection scheme using the prior art will make the bounding box gradually narrow due to the limitation of the camera field of view, an embodiment of the present invention provides a method for detecting a target at the end of the field of view of a fisheye camera, by acquiring an image captured by the fisheye camera; acquiring the bounding box of at least one target object completely within the fisheye field of view of the fisheye camera from the image; acquiring and storing the aspect ratios of the bounding boxes of different target objects; when only part of the target object is within the field of view of the fisheye, the prediction box of the target object at this time is obtained according to the aspect ratio of the bounding box stored by the target object. When the target object leaves the end of the field of view of the fisheye camera and the identified target object is not a complete object, the target bounding box is predicted according to the pixel aspect ratio of the original bounding box, and finally a prediction box of a more realistic pixel plane coordinate can be identified, so as to eliminate the detection and positioning error generated at the edge of the image due to the limitation of the camera field of view, and improve the positioning accuracy of the fisheye camera; at the same time, the perception range of the fisheye camera is expanded, the perception overlap area with surrounding cameras, radars and other sensors is increased, and the matching success rate of the fusion with surrounding multi-sensors is improved.

[0079] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0080] The electronic device includes: one or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the method for detecting an object at the end of the field of view of a fisheye camera as provided in any one or more of the above embodiments. Figure 5 An exemplary structural diagram of the electronic device is disclosed. Figure 5As shown, the electronic device includes: one or more processors 1101, memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0081] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0082] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0083] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0084] In the embodiment of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the method for detecting an object at the end of the field of view of a fisheye camera provided in any one or more of the above embodiments are implemented. The computer-readable medium may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0085] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0086] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0087] It should be noted that more specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0088] Computer readable storage media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0089] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0090] In the above-described embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. For example, it can be implemented by using an application specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0091] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0092] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0093] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A method for detecting an object at the end of a fisheye camera's field of view, characterized in that: include: Get the image captured by the fisheye camera; Acquire a bounding box of at least one target object completely within the fisheye field of view of the fisheye camera from the image; Obtaining and storing different aspect ratios of the bounding boxes of the target object; When only a portion of the target object is within the fisheye field of view, the predicted frame of the target object at this time is obtained according to the aspect ratio of the stored bounding box of the target object.

2. The method for detecting an object at the end of the field of view of a fisheye camera according to claim 1, characterized in that: The step of obtaining and storing different aspect ratios of the bounding boxes of the target objects comprises: Obtaining the pixel size of the image; Acquire, according to the pixel size, a first pixel width and a first pixel height of a bounding box of different target objects in the image; Obtaining an aspect ratio of a bounding box of the target object according to the first pixel width and the first pixel height; The aspect ratio is associated with the target object and then stored.

3. The method for detecting an object at the end of the field of view of a fisheye camera according to claim 1, characterized in that: The step of obtaining the predicted frame of the target object at this time according to the aspect ratio of the stored bounding box of the target object comprises: When only a portion of the target object is within the fisheye field of view, obtaining a second pixel width of the edge of the image occupied by the target object; Acquire, according to the second pixel width and the aspect ratio, a predicted pixel height of a prediction box for drawing the target object; A prediction box for marking the target object is drawn on the image according to the second pixel width and the predicted pixel height, wherein at least a portion of the prediction box is outside the fisheye field of view.

4. The method for detecting an object at the end of the field of view of a fisheye camera according to claim 3, characterized in that: Also includes: Drawing a center point of a prediction box for marking the target object on the image according to the second pixel width and the predicted pixel height specifically includes: Taking the upper left corner of the image as the origin of pixel coordinates, obtaining the pixel coordinates of the corner points of the prediction frame within the fisheye field of view; According to the corner point pixel coordinates, the second pixel width and the predicted pixel height, the pixel coordinates of the center point of the predicted frame are calculated and marked.

5. The method for detecting an object at the end of the field of view of a fisheye camera according to claim 4, characterized in that: The corner point pixel coordinates include: the pixel coordinates of the lower left corner point or the lower right corner point of the prediction frame within the fisheye field of view.

6. The method for detecting an object at the end of the field of view of a fisheye camera according to claim 1, characterized in that: After the step of acquiring the image collected by the fisheye camera, it also includes: Obtaining a distortion correction coefficient of the fisheye camera; Based on the distortion correction coefficient, distortion correction is performed on the image.

7. The method for detecting an object at the end of the field of view of a fisheye camera according to claim 1, characterized in that: Also includes: Based on the prediction frame, the target object is tracked and detected.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the method for detecting an object at the end of the field of view of a fisheye camera as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method for detecting an object at the end of the field of view of a fisheye camera as described in any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the method for detecting an object at the end of the field of view of a fisheye camera as described in any one of claims 1 to 7 are implemented.

Images