Method and device for determining detection frame of fisheye image, equipment and medium

Abstract

The invention relates to a fisheye image detection frame determination method and device, equipment and a medium. After a to-be-processed target fisheye image is acquired, a preliminary detection frame of each target in the target fisheye image needs to be detected, and a first target located outside a normal detection range and a second target located in the normal detection range are determined; inputting the initial detection box of the first target into an instance segmentation network, and determining a final detection box of the first target according to an instance segmentation result; taking the preliminary detection frame of the second target as a final detection frame of the second target; therefore, in order to avoid the situation that the detection frame cannot accurately fit the target due to image distortion, the target is divided into the target outside the normal detection range and the target inside the normal detection range, and because the target outside the normal detection range has a serious distortion phenomenon, the detection frame of the target can be readjusted through the instance segmentation network; the adjusted detection frame can be closer to the shape of an actual object, and the detection effect is improved.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and in particular to a method, device, equipment and medium for determining a detection frame of a fisheye image. Background Art

[0002] A fisheye lens is a wide-angle lens with a short focal length and a viewing angle close to or equal to 180°. Fisheye lenses are widely used in monitoring, autonomous driving, and panoramic photography due to their unique ultra-wide-angle field of view. However, the image distortion produced by fisheye lenses is particularly obvious. The rectangular detection frame of the related technology often faces the problem of being unable to accurately fit the target object when processing fisheye images, resulting in a significant decrease in detection accuracy. In order to solve this problem, the effect of distortion can be reduced by image correction methods, but the new distortion and information loss introduced after image correction cannot be ignored. Therefore, how to accurately determine the rectangular detection frame in the fisheye image is a technical problem that needs to be solved by those skilled in the art. Summary of the invention

[0003] The present application provides a method, apparatus, device and medium for determining a detection frame of a fisheye image, so as to accurately determine a rectangular detection frame in a fisheye image.

[0004] In a first aspect, the present application provides a method for determining a detection frame of a fisheye image, comprising:

[0005] Acquire the target fisheye image to be processed;

[0006] Detecting a preliminary detection frame of each target in the target fisheye image;

[0007] Determine, according to the preliminary detection frames of the targets, a first target outside the normal detection range and a second target within the normal detection range;

[0008] Inputting the preliminary detection box of the first target into an instance segmentation network, and determining the final detection box of the first target according to the instance segmentation result;

[0009] The preliminary detection frame of the second target is used as the final detection frame of the second target.

[0010] Optionally, inputting the preliminary detection box of the first target into an instance segmentation network, and determining the final detection box of the first target according to the instance segmentation result includes:

[0011] Determine an image area covered by a preliminary detection frame of the first target;

[0012] Inputting the image region into an instance segmentation network, and identifying an object edge line of a first target in the image region through the instance segmentation network;

[0013] A final detection frame of the first target is determined according to the object edge line.

[0014] Optionally, identifying an object edge line of a first target in the image region by using the instance segmentation network includes:

[0015] Determining a downsampling multiple corresponding to the image area;

[0016] Determine at least one processing layer from the instance segmentation network according to the downsampling multiple and the image size information; each processing layer includes an upsampling layer and a convolution layer; the image size information is the size information of the target fisheye image;

[0017] The image region is processed by at least one processing layer, and an object edge line of a first target in the image region is determined according to the processing result.

[0018] Optionally, determining a final detection frame of the first target according to the object edge line includes:

[0019] Determine the target detection frame at each detection frame angle using the object edge line;

[0020] Determine the foreground ratio of each target detection frame; wherein the foreground ratio is the ratio of the number of pixels within the edge line of the object to the total number of pixels;

[0021] Select the target prospect ratio value with the largest value from each prospect ratio value;

[0022] The target detection frame corresponding to the target foreground ratio value is used as the final detection frame of the first target.

[0023] Optionally, determining, according to the preliminary detection frames of the targets, a first target outside a normal detection range and a second target within a normal detection range includes:

[0024] Determine the center point coordinates of each target; wherein the center point coordinates are the center point coordinates of the preliminary detection frame corresponding to each target;

[0025] Determine the position status according to the center point coordinates of each target and the normal detection range;

[0026] If the position state is: the center point coordinates are outside the normal detection range, the corresponding target is used as the first target; if the position state is: the center point coordinates are within the normal detection range, the corresponding target is used as the second target.

[0027] Optionally, determining the position state according to the center point coordinates of each target and the normal detection range includes:

[0028] Calculating normalized coordinate values ​​according to the center point coordinates and image size information of each target; the image size information is the size information of the target fisheye image;

[0029] The position state is determined using the normalized coordinate value and normal detection range of each target; wherein the normal detection range is the normalized range value.

[0030] Optionally, after determining the first target outside the normal detection range, the method further includes:

[0031] determining a target quantity of a first target located outside a normal detection range;

[0032] If the number of targets is less than the number threshold, the preliminary detection frame of each target is used as the final detection frame of each target;

[0033] If the number of targets is not less than the number threshold, continue to execute the step of inputting the preliminary detection box of the first target into the instance segmentation network.

[0034] In a second aspect, the present application provides a device for determining a detection frame of a fisheye image, comprising:

[0035] An acquisition module, used for acquiring a target fisheye image to be processed;

[0036] A detection module, used to detect a preliminary detection frame of each target in the target fisheye image;

[0037] A first determination module, used to determine a first target outside a normal detection range and a second target within a normal detection range according to a preliminary detection frame of each target;

[0038] A second determination module, configured to input a preliminary detection frame of the first target into an instance segmentation network, and determine a final detection frame of the first target according to an instance segmentation result;

[0039] The third determination module is used to use the preliminary detection frame of the second target as the final detection frame of the second target.

[0040] In a third aspect, the present application provides an electronic device, including:

[0041] A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the above-mentioned detection frame determination method of the present application through the computer program.

[0042] In a fourth aspect, the present application further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are used to execute the steps of the above-mentioned detection frame determination method of the present application.

[0043] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the present application provides a solution for determining a detection frame of a fisheye image; after obtaining the target fisheye image to be processed, the present application needs to detect the preliminary detection frame of each target in the target fisheye image, and determine the first target outside the normal detection range and the second target within the normal detection range according to the preliminary detection frame of each target; the preliminary detection frame of the first target is input into the instance segmentation network, and the final detection frame of the first target is determined according to the instance segmentation result; the preliminary detection frame of the second target is used as the final detection frame of the second target; it can be seen that in order to avoid the inability of the detection frame to accurately fit the target due to image distortion, the present application divides the target into a target outside the normal detection range and a target within the normal detection range. Since the target outside the normal detection range will be distorted, the detection frame of the target can be readjusted through the instance segmentation network at this time, so that the adjusted detection frame is closer to the actual object shape, thereby improving the detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0047] Figure 1 A schematic diagram of a flow chart of a method for determining a detection frame of a fisheye image provided in an embodiment of the present application;

[0048] Figure 2 A schematic flow chart of another method for determining a detection frame of a fisheye image provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of an example segmentation network structure provided in an embodiment of the present application;

[0050] Figure 4a A schematic diagram of a target detection frame provided in an embodiment of the present application;

[0051] Figure 4bA schematic diagram of another target detection frame provided in an embodiment of the present application;

[0052] Figure 4c A schematic diagram of another target detection frame provided in an embodiment of the present application;

[0053] Figure 5 A schematic flow chart of another method for determining a detection frame of a fisheye image provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the normal detection range setting provided in an embodiment of the present application;

[0055] Figure 7 A schematic flow chart of another method for determining a detection frame of a fisheye image provided in an embodiment of the present application;

[0056] Figure 8 A schematic diagram of a complete detection frame determination method flow provided in an embodiment of the present application;

[0057] Fig. 9 A schematic diagram of the structure of a device for determining a detection frame of a fisheye image provided in an embodiment of the present application;

[0058] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] Since the fisheye lens will produce radial distortion, the object will be distorted when it is located in the edge area of ​​the fisheye image. In the related art, it is difficult to accurately describe the outline of the object, resulting in the detection frame of the object containing a large amount of background or other objects, thereby affecting the detection accuracy. Therefore, in this embodiment, a method, device, equipment and medium for determining the detection frame of a fisheye image are provided. By combining target detection and instance segmentation technology, the application can accurately determine the rectangular detection frame in the fisheye image, so that the detection frame is more consistent with the actual outline of the object, thereby improving the detection accuracy.

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0061] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0062] See also Figure 1 , Figure 1 A flow chart of a method for determining a detection frame of a fisheye image provided in an embodiment of the present application is provided. The method specifically comprises the following steps:

[0063] S101, obtaining a target fisheye image to be processed;

[0064] In this embodiment, the fisheye image is a picture taken by a fisheye lens, and the target fisheye image is the image to be processed in this embodiment. There are multiple targets in the target fisheye image, such as pedestrians, vehicles, animals, etc., which are not specifically limited here.

[0065] S102, detecting a preliminary detection frame of each target in the target fisheye image;

[0066] After obtaining the target fisheye image to be processed, the present application first detects the preliminary detection frame of each target in the target fisheye image. When detecting the preliminary detection frame, the preliminary detection frame of each target in the target fisheye image can be detected by the target detection algorithm. The target detection algorithm can be a YOLOv (You Only Look Once) 5 network, a YOLOv6 network, etc., which is not specifically limited here. In this embodiment, only the YOLOv5 network is used as an example for explanation. Through S102, the preliminary detection frame of each target in the target fisheye image can be preliminarily detected. Due to the distortion of the image, the preliminary detection frame cannot accurately fit the actual contour of the object, so the present application also needs to adjust the preliminary detection frame through subsequent steps.

[0067] S103, determining a first target outside a normal detection range and a second target within a normal detection range according to a preliminary detection frame of each target;

[0068] In this embodiment, since the target fisheye image will produce severe distortion when the target is located in the edge area of ​​the fisheye image, the present application divides the target in the target fisheye image into two categories in consideration of the detection speed, and regards the target outside the normal detection range as the first category, referred to as the first target, and regards the target within the normal detection range as the second category, referred to as the second target. The normal detection range is a pre-set image range, and the target within the normal detection range produces less distortion, so there is no need to adjust the detection frame; while the area outside the normal detection range is an area where the radial distortion of the image is more serious, so the distortion of the target is larger, and the detection frame needs to be adjusted at this time.

[0069] In the present application, the normal detection range can be customized according to actual conditions, such as: taking the center point of the fisheye image as the reference point, dividing the fisheye image range up, down, left and right, and defining the central area of ​​the fisheye image as the normal detection range, specifically defined as no more than 0.5 up, down, left and right from the center point (based on a normalized scale of 0-1).

[0070] S104, inputting the preliminary detection frame of the first target into the instance segmentation network, and determining the final detection frame of the first target according to the instance segmentation result;

[0071] In this embodiment, since the first target outside the normal detection range is greatly distorted, the detection frame of the first target can be readjusted through the instance segmentation network. The present application refers to the detection frame re-determined by the instance segmentation network as the final detection frame. Among them, after the present application inputs the preliminary detection frame of the first target into the instance segmentation network, the instance segmentation network will segment the edge line of the first target to obtain an instance segmentation result. The instance segmentation result divides the pixels in the preliminary detection frame into two categories: one is the background, and the other is the edge of the object. The pixels contained in the edge line are the target, and the rest are the background. Therefore, after the present application re-identifies the edge of the first target through the instance segmentation network, it can re-determine the final detection frame that better fits the first target based on the edge of the first target, thereby improving the recognition accuracy.

[0072] S105: Using the preliminary detection frame of the second target as the final detection frame of the second target.

[0073] In this embodiment, since the second target located within the normal detection range produces less distortion, the preliminary detection frame of the second target can be directly used as the final detection frame.

[0074] In summary, it can be seen that the present application can ensure fast and accurate target detection within the normal detection range through this method. For targets outside the normal detection range, a final detection frame that is more in line with the actual contour of the object is generated by instance segmentation edge lines, effectively avoiding the problem of too much background or other objects in the traditional detection frame. At the same time, this method has the characteristics of high computational efficiency and simple operation, and is suitable for a variety of fisheye image processing scenarios.

[0075] See also Figure 2 , Figure 2 A schematic flow chart of another method for determining a detection frame of a fisheye image provided in an embodiment of the present application, the method specifically comprises the following steps:

[0076] S201, obtaining a target fisheye image to be processed;

[0077] S202, detecting a preliminary detection frame of each target in the target fisheye image;

[0078] S203, determining a first target outside a normal detection range and a second target within a normal detection range according to a preliminary detection frame of each target;

[0079] S204, determining an image area covered by a preliminary detection frame of the first target;

[0080] S205, inputting the image region into an instance segmentation network, and identifying an object edge line of the first target in the image region through the instance segmentation network;

[0081] S206, determining the target detection frame at each detection frame angle using the object edge line;

[0082] S207, determining a foreground ratio of each target detection frame; wherein the foreground ratio is the ratio of the number of pixels within the edge line of the object to the total number of pixels;

[0083] S208, selecting the target foreground proportion value with the largest value from each foreground proportion value;

[0084] S209, using the target detection frame corresponding to the target foreground ratio as the final detection frame of the first target;

[0085] S210: Using the preliminary detection frame of the second target as the final detection frame of the second target.

[0086] In this application, it is first necessary to train the target detection algorithm and instance segmentation network so that the detection frame can be accurately generated by the trained target detection algorithm and instance segmentation network. During the training process of the entity segmentation network, it is necessary to compare the output category with the label of the original image so that the entity segmentation network can better learn the foreground and obtain a more accurate detection frame.

[0087] Among them, this application uses the standard target detection algorithm YOLOv5 to directly detect the target fisheye image. The YOLOv5 network includes three parts: the backbone network backbone, the neck network neckhe and the head network head. The backbone network processes the input image through a series of convolutional layers and pooling layers, gradually reducing the size of the feature map while increasing the number of channels, so as to retain and extract important features in the image. The feature map extracted by the backbone network will be passed to the subsequent neck network and head network for processing. The neck network is used to further process and integrate the feature map extracted from the backbone network, including feature fusion, feature dimensionality reduction, feature enhancement, etc.; the head network is responsible for task-specific processing of the features obtained from the neck network, such as category classification and bounding box regression in target detection. In this application, the head network has 3 branches, corresponding to 3 neck networks, which detect large, medium and small objects respectively, and the image downsampling is 8, 16, and 32 respectively. The head network outputs object detection categories and object detection boxes.

[0088] After the application identifies the rectangular preliminary detection frame of each target through the target detection algorithm, if the target is within the normal detection range, the rectangular preliminary detection frame is directly used as the final detection frame to quickly and accurately detect objects within the normal detection range. When the detection frame of the detected target object is outside the normal detection range, the application will perform the instance segmentation step through the instance segmentation network, and first input the area covered by the preliminary detection frame of the second target into the instance segmentation network. Among them, when the application inputs the area covered by the preliminary detection frame into the instance segmentation network, it can extract features from the corresponding neck of the YOLOv5 network according to the position of the neck where the preliminary detection frame is located, and then input them into the instance segmentation network, and identify the object edge lines of each target through the instance segmentation network.

[0089] In some embodiments of the present application, when identifying the object edge line of each target through an instance segmentation network, the downsampling multiple corresponding to the image area can be first determined, and at least one processing layer is determined from the instance segmentation network based on the downsampling multiple and the image size information; each processing layer includes an upsampling layer and a convolution layer; the image size information is the size information of the target fisheye image; then the image area is processed through at least one processing layer, and the object edge line of the first target in the image area is determined based on the processing result.

[0090] Among them, the instance segmentation network has different processing layers, and each processing layer includes an upsampling layer Upsample and a convolution layer Conv. The instance segmentation network in this application sets different processing layers for upsampling according to the downsampling multiples of different necks. For example, if the downsampling multiples of the neck are 8 times, 16 times, 32 times, etc., then different processing layers need to be upsampled to 8 times, 16 times, 32 times, etc. Therefore, after extracting the features corresponding to the area covered by the preliminary detection frame from the neck, the application first determines the downsampling multiples of the features, and determines at least one processing layer from the instance segmentation network according to the downsampling multiples and the image size information. For example: if the downsampling multiple is 32 and the image size information is 126, then the features are upsampled from 32 to 64 through the first processing layer, and then upsampled to 126 through another processing layer. Upsample in the processing layer is used to enlarge the feature map to be consistent with the size of the original input image. Conv makes the extracted features more accurate through convolution operations. This application upsamples the features to be consistent with the size of the picture, which can obtain more accurate segmentation results.

[0091] See also Figure 3 , Figure 3 A schematic diagram of an example segmentation network structure provided in an embodiment of the present application, Figure 3 Only three processing layers (Upsample+Conv) and one output layer Output (Segmentation Msak, segmentation mask) are taken as examples for explanation. Figure 3 Upsample(2) in Conv(x, y) indicates two times upsampling, x in Conv(x, y) indicates the number of input channels, and y indicates the number of output channels, such as Conv(128, 64) has 128 input channels and 64 output channels. The instance segmentation result output by the output layer is the edge line of the object of the first target. The instance segmentation network divides the pixels in the area into two categories: one is the background, and the other is the edge of the object. The pixels contained in the edge line are the foreground of the object, and the rest are the background.

[0092] Furthermore, after identifying the object edge line of the first target in the image area through the instance segmentation network, the present application can extract the four outermost points of the target edge according to the segmented object edge line and different detection frame angles, corresponding to the four boundary points of the top, bottom, left and right, respectively, to obtain the target detection frame at each detection frame angle. Figure 4a , Figure 4b and Figure 4c , are schematic diagrams of target detection frames provided in the embodiments of the present application. The thin rectangle in the figure represents a vehicle, the origin of the coordinate system is the center point of the vehicle, and the inclination angle of the edge line obtained by detecting the vehicle frame is 15 degrees. Figure 4a The detection frame angle of the target detection frame is 0 degrees, and the target detection frame is a thick rectangle. Figure 4bThe detection frame angle of the target detection frame in is 15 degrees, and the target detection frame is a thick rectangle. Figure 4c The detection frame angle of the target detection frame in is 90 degrees, and the target detection frame is a thick rectangle.

[0093] After obtaining the target detection frames at different detection frame angles, the present application can calculate the foreground ratio of each target detection frame, and use the target detection frame with the largest foreground ratio as the final detection frame. The final detection frame is the rectangular detection frame that best fits the target, reducing the impact of background and other interferences. Among them, the foreground ratio is the ratio of the number of pixels within the edge line of the object to the total number of pixels. Based on the instance segmentation results, the present application can obtain: the pixel background outside the edge line, and the pixel foreground inside the edge line. At this time, the foreground ratio can be calculated according to the following formula: foreground ratio = pixel foreground / (pixel background + pixel foreground).

[0094] It can be seen that the present application inputs the image area covered by the preliminary detection frame into the entity segmentation network to re-identify the object edge line of the target through the entity segmentation network to improve the detection effect; the entity segmentation network can more accurately identify the edge line of the target through the upsampling layer and the convolution layer; when determining the final detection frame, the foreground proportion at each detection frame angle can be calculated according to the object edge line, so that the detection frame with the largest foreground proportion can be used as the final detection frame. In this way, the detection frame that best fits the target can be selected to reduce the influence of background and other interferences.

[0095] Compared with other related technologies, the present application brings significant beneficial effects. First of all, the present application greatly improves the accuracy of fisheye image object detection by segmenting based on edge lines and re-adjusting the detection frame, especially in the case of severe image distortion, effectively reducing the occurrence of false detection and missed detection. In addition, the scheme improves the accuracy of boundary segmentation, so that the detection frame can better fit the shape of the actual object, further improving the detection effect of the system. By avoiding complex image correction steps, this patented technology saves resources and time while still maintaining efficient real-time processing capabilities, and is suitable for a variety of fisheye image application scenarios. Through the above technical features, this patented solution not only excels in accuracy and compatibility, but also effectively reduces the overall cost of the system, and has extremely high application value.

[0096] See also Figure 5 , Figure 5 A schematic flow chart of another method for determining a detection frame of a fisheye image provided in an embodiment of the present application, the method specifically comprises the following steps:

[0097] S301, obtaining a target fisheye image to be processed;

[0098] S302, detecting a preliminary detection frame of each target in the target fisheye image;

[0099] S303, determining the center point coordinates of each target; wherein the center point coordinates are the center point coordinates of the preliminary detection frame corresponding to each target;

[0100] S304, determining the position state according to the center point coordinates of each target and the normal detection range;

[0101] S305, if the position state is: the center point coordinates are outside the normal detection range, the corresponding target is used as the first target; if the position state is: the center point coordinates are within the normal detection range, the corresponding target is used as the second target;

[0102] S306, inputting the preliminary detection frame of the first target into the instance segmentation network, and determining the final detection frame of the first target according to the instance segmentation result;

[0103] S307: Use the preliminary detection frame of the second target as the final detection frame of the second target.

[0104] In this application, the normal detection range needs to be predefined first. Figure 6 , Figure 6 A schematic diagram of the normal detection range setting provided in the embodiment of the present application is shown in FIG. Figure 6 As shown, the input image size information is 640*640, the image center point is set to o(320,320), the coordinate point is set to a at (160,160), and the range of up, down, left, and right is defined as 0 to 1 from the center point outward according to the actual ratio. The input image size in this article is (640,640), and the length of up, down, left, and right is 320, so the range of point a from the image center o is (x,y) = (320–160,320-160) / (320,320) = (0.5,0.5). In this application, the area around the image center o within the range of 0.5 up, down, left, and right is defined as the normal detection range.

[0105] After the application detects the preliminary detection frame of each target in the target fisheye image, the center point of each preliminary detection frame is used as a reference point to determine whether it is within the normal detection range. In some embodiments of the present application, when determining the position state according to the center point coordinates of each target and the normal detection range, the normalized coordinate value can be calculated according to the center point coordinates of each target and the image size information; the image size information is the size information of the target fisheye image; the position state is determined using the normalized coordinate value of each target and the normal detection range; wherein the normal detection range is the normalized range value. For example: the center point coordinates are (80,80), and the image size information is 640*640, so the normalized coordinate value is calculated to be (320–80,320-80) / (320,320)=(0.75,0.75). If the normalized range value is (0.5,0.5), it can be determined that the center point coordinates are outside the normal detection range, and the corresponding target is taken as the first target.

[0106] From the above, it can be seen that in order to accurately identify the first target and the second target, the present application compares the center point of the preliminary detection frame of each target with the normal detection range, so as to accurately identify the detection frame of the first target that needs to be adjusted; and, in order to determine the position state more quickly, the present application can calculate the normalized coordinate value of each target, compare the normalized coordinate value with the normalized range value, and quickly determine the position state.

[0107] See also Figure 7 , Figure 7 A schematic flow chart of another method for determining a detection frame of a fisheye image provided in an embodiment of the present application, the method specifically comprises the following steps:

[0108] S401, obtaining a target fisheye image to be processed;

[0109] S402, detecting a preliminary detection frame of each target in the target fisheye image;

[0110] S403, determining a first target outside a normal detection range and a second target within a normal detection range according to a preliminary detection frame of each target;

[0111] S404, determining the number of first targets outside the normal detection range;

[0112] S405, if the number of targets is less than the number threshold, the preliminary detection frame of each target is used as the final detection frame of each target;

[0113] S406: If the number of targets is not less than the number threshold, input the preliminary detection frame of the first target into the instance segmentation network, and determine the final detection frame of the first target according to the instance segmentation result;

[0114] S407: Use the preliminary detection frame of the second target as the final detection frame of the second target.

[0115] In the present application, after detecting all targets and targets inside and outside the normal detection range, the number of targets outside the normal detection range can be obtained. The present application sets a quantity threshold. If the number of targets outside the normal detection range is less than the predetermined threshold, it is a sparse scene, otherwise it is a congested scene. The present application can set the predetermined threshold to 6. Of course, it can also be set to other values ​​according to actual needs, and it is not specifically limited here.

[0116] In sparse scenes, the accuracy of objects is relatively less affected, and it is not necessary to distinguish whether they are in the normal detection range, and the preliminary detection frame of all targets is directly used as the final detection frame; in congested scenes, the accuracy of objects is relatively more affected, and it is necessary to distinguish whether each target is in the normal detection range. If the target is in the normal detection range, the preliminary detection frame is directly used as the final detection frame; if the target is outside the normal detection range, the area in the detection frame is input into the instance segmentation network for two types of segmentation: one is the background, and the other is the object edge. By obtaining the edge line of the object, the upper, lower, left and right edge points of the object are further calculated according to different angles, and a detection frame that better fits the shape of the object is redefined, thereby improving the detection accuracy in the fisheye distortion area.

[0117] See also Figure 8 , Figure 8 A schematic diagram of a complete detection frame determination method provided in an embodiment of the present application, through Figure 8 It can be seen that the method specifically includes the following four steps:

[0118] Step 1: Predefine; This step mainly predefines the normal detection range;

[0119] Step 2: Target detection and instance segmentation: In this step, the YOLOv5 target detection algorithm is used to detect the target of the input image, generate a preliminary rectangular detection frame, and determine whether each target is within the normal detection range. If the target is not within the normal detection range, the instance segmentation network is input. If the target is within the normal detection range, the detection frame of each target within the normal detection range is directly output in step 4.

[0120] Step 3: Re-detect the detection frame; in this step, it is necessary to re-extract the four outermost points of the edge according to the instance segmentation result and calculate the most fitting frame;

[0121] Step 4: Result output; in this step, if the target is not within the normal detection range, it is necessary to output the object edge frame and the redefined rectangular detection frame; if the target is within the normal detection range, the detection frame of each target within the normal detection range is output.

[0122] In order to improve the detection speed, the present application can decide whether to adjust the detection frame according to the number of first targets. If the number of first targets is large, it means that the recognition accuracy is greatly affected. At this time, the detection frame of the first target can be adjusted. Otherwise, the detection frame is not adjusted, thereby improving the processing speed. This solution can generate a rectangular detection frame more accurately, especially in the distorted area of ​​the fisheye image, and can better adapt to the actual shape of the target object.

[0123] In summary, the core technology of this application is: detection frame repositioning technology based on edge line segmentation, which recalculates and adjusts the detection frame by performing edge line segmentation on the objects detected in the fisheye image to more accurately adapt to the distorted object shape. In addition, this application avoids complex image correction steps by directly processing on the distorted image, thereby improving the real-time performance and computing efficiency of the system. This technology is not only suitable for object detection and segmentation of fisheye images, but can also be widely used in various scenarios such as monitoring and autonomous driving.

[0124] See also Fig. 9 , Fig. 9 A schematic diagram of a device for determining a detection frame of a fisheye image provided in an embodiment of the present application is provided. The device specifically comprises:

[0125] An acquisition module 11 is used to acquire a target fisheye image to be processed;

[0126] A detection module 12, used to detect a preliminary detection frame of each target in the target fisheye image;

[0127] A first determination module 13, configured to determine, based on the preliminary detection frames of the targets, a first target outside the normal detection range and a second target within the normal detection range;

[0128] A second determination module 14 is used to input the preliminary detection frame of the first target into an instance segmentation network, and determine the final detection frame of the first target according to the instance segmentation result;

[0129] The third determination module 15 is configured to use the preliminary detection frame of the second object as the final detection frame of the second object.

[0130] As an optional embodiment, the second determining module includes:

[0131] A first determining unit, configured to determine an image area covered by a preliminary detection frame of the first target;

[0132] A recognition unit, configured to input the image region into an instance segmentation network, and recognize an object edge line of a first target in the image region through the instance segmentation network;

[0133] The second determining unit is used to determine a final detection frame of the first target according to the edge line of the object.

[0134] As an optional embodiment, the identification unit includes:

[0135] A first determining subunit, used to determine a downsampling multiple corresponding to the image area;

[0136] A second determination subunit is used to determine at least one processing layer from the instance segmentation network according to the downsampling multiple and the image size information; each processing layer includes an upsampling layer and a convolution layer; the image size information is the size information of the target fisheye image;

[0137] The processing subunit is used to process the image area through at least one processing layer, and determine the object edge line of the first target in the image area according to the processing result.

[0138] As an optional embodiment, the second determining unit includes:

[0139] A third determination subunit is used to determine the target detection frame at each detection frame angle using the object edge line;

[0140] The fourth determination subunit is used to determine the foreground ratio of each target detection frame; wherein the foreground ratio is the ratio of the number of pixels within the edge line of the object to the total number of pixels;

[0141] A selection subunit is used to select a target foreground proportion value with the largest value from each foreground proportion value;

[0142] The fifth determination subunit is used to use the target detection frame corresponding to the target foreground ratio value as the final detection frame of the first target.

[0143] As an optional embodiment, the first determining module includes:

[0144] A third determination unit is used to determine the center point coordinates of each target; wherein the center point coordinates are the center point coordinates of the preliminary detection frame corresponding to each target;

[0145] The fourth determination unit is used to determine the position state according to the center point coordinates of each target and the normal detection range; if the position state is: the center point coordinates are outside the normal detection range, the corresponding target is used as the first target; if the position state is: the center point coordinates are within the normal detection range, the corresponding target is used as the second target.

[0146] As an optional embodiment, the fifth determination unit is specifically used to: calculate the normalized coordinate value based on the center point coordinates and image size information of each target; the image size information is the size information of the target fisheye image; determine the position state using the normalized coordinate value and normal detection range of each target; wherein the normal detection range is the normalized range value.

[0147] As an optional embodiment, the detection frame determining device further includes:

[0148] The fourth determination module is used to determine the number of targets of the first target located outside the normal detection range; if the number of targets is less than the number threshold, the preliminary detection frame of each target is used as the final detection frame of each target; if the number of targets is not less than the number threshold, the second determination module is triggered to input the preliminary detection frame of the first target into the instance segmentation network.

[0149] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0150] See also Fig.10 , Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device specifically includes:

[0151] The processor 21, the memory 22 and a computer program stored in the memory 22 and executable on the processor 21. The processor 21 executes the steps of the detection frame determination method described in any of the above method embodiments through the computer program.

[0152] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0153] The memory 22 may include one or more computer-readable storage media, which may be non-transitory. The memory 22 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 22 is at least used to store the following computer program 221, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps in the detection frame determination method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 22 may also include an operating system 222 and data 223, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 222 may include Windows, Unix, Linux, etc.

[0154] In some embodiments, the electronic device may further include a display screen 23 , an input / output interface 24 , a communication interface 25 , a sensor 26 , a power source 27 , and a communication bus 28 .

[0155] certainly, Fig.10 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of the present application. In actual applications, the electronic device may include Fig.10 More or fewer components than shown, or combinations of certain components.

[0156] In another exemplary embodiment, a computer storage medium is also provided, and when the program instructions are executed by a processor, the steps of the detection frame determination method described in any of the above method embodiments are implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0157] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0158] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0159] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for determining a detection frame of a fisheye image, characterized in that: include: Acquire the target fisheye image to be processed; Detecting a preliminary detection frame of each target in the target fisheye image; Determine, according to the preliminary detection frames of the targets, a first target outside the normal detection range and a second target within the normal detection range; Inputting the preliminary detection box of the first target into an instance segmentation network, and determining the final detection box of the first target according to the instance segmentation result; The preliminary detection frame of the second target is used as the final detection frame of the second target.

2. The detection frame determination method according to claim 1, characterized in that: Inputting the preliminary detection frame of the first target into the instance segmentation network, and determining the final detection frame of the first target according to the instance segmentation result includes: Determine an image area covered by a preliminary detection frame of the first target; Inputting the image region into an instance segmentation network, and identifying an object edge line of a first target in the image region through the instance segmentation network; A final detection frame of the first target is determined according to the object edge line.

3. The detection frame determination method according to claim 2, characterized in that: Identifying an object edge line of a first target in the image region by the instance segmentation network includes: Determining a downsampling multiple corresponding to the image area; Determine at least one processing layer from the instance segmentation network according to the downsampling multiple and the image size information; each processing layer includes an upsampling layer and a convolution layer; the image size information is the size information of the target fisheye image; The image region is processed by at least one processing layer, and an object edge line of a first target in the image region is determined according to the processing result.

4. The detection frame determination method according to claim 2, characterized in that: Determining a final detection frame of the first target according to the edge line of the object includes: Determine the target detection frame at each detection frame angle using the object edge line; Determine the foreground ratio of each target detection frame; wherein the foreground ratio is the ratio of the number of pixels within the edge line of the object to the total number of pixels; Select the target prospect ratio value with the largest value from each prospect ratio value; The target detection frame corresponding to the target foreground ratio value is used as the final detection frame of the first target.

5. The detection frame determination method according to claim 1, characterized in that: The step of determining, based on the preliminary detection frames of the respective targets, a first target outside the normal detection range and a second target within the normal detection range comprises: Determine the center point coordinates of each target; wherein the center point coordinates are the center point coordinates of the preliminary detection frame corresponding to each target; Determine the position status according to the center point coordinates of each target and the normal detection range; If the position state is: the center point coordinates are outside the normal detection range, the corresponding target is used as the first target; if the position state is: the center point coordinates are within the normal detection range, the corresponding target is used as the second target.

6. The detection frame determination method according to claim 5, characterized in that: Determining the position state according to the center point coordinates of each target and the normal detection range includes: Calculating normalized coordinate values ​​according to the center point coordinates and image size information of each target; the image size information is the size information of the target fisheye image; The position state is determined using the normalized coordinate value and normal detection range of each target; wherein the normal detection range is the normalized range value.

7. The detection frame determination method according to any one of claims 1 to 6, characterized in that: After determining the first target outside the normal detection range, the method further includes: determining a target quantity of a first target located outside a normal detection range; If the number of targets is less than the number threshold, the preliminary detection frame of each target is used as the final detection frame of each target; If the number of targets is not less than the number threshold, continue to execute the step of inputting the preliminary detection box of the first target into the instance segmentation network.

8. A device for determining a detection frame of a fisheye image, characterized in that: include: An acquisition module, used for acquiring a target fisheye image to be processed; A detection module, used to detect a preliminary detection frame of each target in the target fisheye image; A first determination module, used to determine a first target outside a normal detection range and a second target within a normal detection range according to a preliminary detection frame of each target; A second determination module, configured to input a preliminary detection frame of the first target into an instance segmentation network, and determine a final detection frame of the first target according to an instance segmentation result; The third determination module is used to use the preliminary detection frame of the second target as the final detection frame of the second target.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the detection frame determination method described in any one of claims 1 to 7 of the present application through the computer program.

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and the computer executable instructions are used to execute the steps of the detection frame determination method described in any one of claims 1 to 7 of the present application.

Images