Image labeling method and device, electronic equipment and computer readable storage medium

Abstract

An image labeling method, device and computer readable storage medium, the method comprising: acquiring a fisheye image (101) collected by a fisheye camera; processing the fisheye image through a rotation matrix parameter to obtain a target image (102); adding a labeling result of a target object in the target image (103); and mapping the labeling result in the target image to the fisheye image according to the rotation matrix parameter (104). The present application converts the fisheye image with picture distortion into a target image without distortion or with weak distortion through the rotation matrix parameter, and labels the target object in the target image, so that the influence of picture distortion in the labeling process is reduced. Finally, the labeling result in the target image is mapped to the fisheye image through the rotation matrix parameter, so that the target object in the fisheye image has a high-quality labeling result, the labeling effect is improved, and the entire labeling process can be independent of expensive 3D sensors, thereby reducing the cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image labeling, in particular to an image labeling method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] In the fields of automatic driving, assisted driving, automatic mapping, etc., tracking algorithms are used. The tracking algorithm usually has the labeling requirement of a target object (such as an obstacle), so as to achieve the purpose of tracking the target object under the condition of meeting the labeling requirement.

[0003] In the current scheme, the tracking algorithm can track a target object in a picture collected by a fisheye camera. In order to achieve the purpose of tracking, the target object needs to be labeled in the form of a pseudo 3D box, a 2D box, etc. One implementation manner is to directly label the target object in the picture collected by the fisheye camera. Another implementation manner is to use some 3D sensors (such as a laser radar) to collect data and model the environment, and label the target object therein.

[0004] However, the picture collected by the fisheye camera has a large distortion. Influenced by the distortion, the accuracy of labeling the target object is reduced, and the labeling quality is reduced. In addition, using the 3D sensor to assist labeling increases the cost and reduces the economy of labeling. SUMMARY

[0005] The present application provides an image labeling method, device and computer readable storage medium, which can solve the problem of reduced accuracy of labeling a target object and poor economy of labeling in the prior art.

[0006] In a first aspect, an image labeling method is provided, comprising:

[0007] obtaining a fisheye image collected by a fisheye camera;

[0008] processing the fisheye image through a rotation matrix parameter to obtain a target image, the rotation matrix parameter comprising a conversion parameter between a first imaging model of the fisheye image and a second imaging model of the target image;

[0009] adding a labeling result of a target object in the target image, the labeling result comprising position information of the target object in the target image;

[0010] mapping the labeling result in the target image to the fisheye image according to the rotation matrix parameter.

[0011] In a second aspect, an image labeling device is provided, comprising:

[0012] The memory and the processor;

[0013] The memory is configured to acquire a fisheye image collected by a fisheye camera.

[0014] The processor is configured to:

[0015] The fisheye image is processed by a rotation matrix parameter to obtain a target image, the rotation matrix parameter including a conversion parameter between a first imaging model of the fisheye image and a second imaging model of the target image.

[0016] A labeling result of a target object is added in the target image, the labeling result including position information of the target object in the target image.

[0017] The labeling result in the target image is mapped into the fisheye image according to the rotation matrix parameter.

[0018] In a third aspect, the present application provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, when the computer program is executed by the processor, the method in the above aspect is implemented.

[0019] In a fourth aspect, the present application provides a computer readable storage medium, including instructions, when the instructions are executed on a computer, the computer executes the method in the above aspect.

[0020] In a fifth aspect, the present application provides a computer program product, including instructions, when the instructions are executed on a computer, the computer executes the method in the above aspect.

[0021] In the embodiments of the present application, the fisheye image with picture distortion is converted into the target image without distortion or with weak distortion through the rotation matrix parameter between the first imaging model of the fisheye image and the second imaging model of the target image, and the target image is labeled with the target object, so that the influence of picture distortion in the labeling process is reduced, the labeling quality is improved, and finally the labeling result in the target image is mapped into the fisheye image through the rotation matrix parameter, so that the target object in the fisheye image also has high-quality labeling result, and the labeling effect is improved. In addition, the whole labeling process can not depend on expensive 3D sensors, and the cost of labeling is reduced, and the economy is improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of an image labeling method provided by the embodiments of the present application;

[0023] Figure 2is an fisheye image provided by an embodiment of the present application;

[0024] Figure 3 is a target image provided by an embodiment of the present application;

[0025] Figure 4 is another target image provided by an embodiment of the present application;

[0026] Figure 5 is another fisheye image provided by an embodiment of the present application;

[0027] Figure 6 is a specific step flow chart of an image labeling method provided by an embodiment of the present application;

[0028] Figure 7 is a block diagram of an image labeling device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present application more apparent, the following will describe the example embodiments according to the present application in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present application.

[0030] In the following description, a large number of specific details are given in order to provide a more thorough understanding of the present application. However, it is obvious to those skilled in the art that the present application can be implemented without one or more of these details. In other examples, some technical features known in the art are not described in order to avoid obscuring the present application.

[0031] It should be understood that the present application can be implemented in different forms, and should not be interpreted as being limited to the embodiments presented herein. On the contrary, these embodiments are provided to make the disclosure complete and full, and to fully convey the scope of the present application to those skilled in the art.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of associated items.

[0033] For a thorough understanding of the application, reference will be made to the following detailed description, in which reference will be made to the drawings. Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. There is no intention, therefore, of limiting the application as described herein to the precise embodiments and modifications described.

[0034] The image annotation method and device, system of the present application will be described in detail below in conjunction with the drawings. The features in the following embodiments and implementation manners can be combined with each other without conflict.

[0035] Figure 1 is a flowchart of an image annotation method provided by an embodiment of the present application, as shown in Figure 1 The method can include the following steps.

[0036] Step 101, acquiring a fisheye image collected by a fisheye camera.

[0037] In actual application, tracking algorithms are applied in automatic driving, surveying and mapping, etc. The tracking algorithm has the demand for perceiving and tracking the target object in the scene. The tracking algorithm can be implemented based on the fisheye image collected by the fisheye camera. Through accurate annotation of the target object in the fisheye image, the purpose of perceiving the target object is achieved, and high tracking accuracy is ensured in the subsequent tracking process. In addition, in the model training scene, when there is a functional demand for identifying the target object, there is also a demand for accurately annotating the target object in the fisheye image as training data, so as to improve the quality of the training data and improve the training efficiency of the model.

[0038] Specifically, since the lens of the fisheye camera is assembled by multiple arc-shaped lenses, and a certain assembly error will be generated when the lens is assembled, lens distortion will be generated in the shooting process. Lens distortion is actually a general term for the perspective distortion of optical lenses, that is, distortion caused by perspective. In a fisheye camera with a larger shooting range, the severity of this distortion is higher.

[0039] Referring to Figure 2 and Figure 3 , Figure 2a fisheye image photographed by a fisheye camera, Figure 3 a target image without distortion for the same scene, it can be seen that compared with Figure 3 , Figure 2 there is a large distortion in the picture of the fisheye image, especially a serious distortion deviation is generated at the four top corners of the fisheye image, so when labeling the target object in the fisheye image Figure 2 , the labeling quality will be reduced due to the influence of the picture distortion.

[0040] In this step, the fisheye camera can generate a fisheye image based on the collected original image data and the first imaging model of the corresponding fisheye camera. Specifically, since the lens of the fisheye camera is generally composed of several different lenses, the incident light is refracted to different degrees during imaging and projected onto the imaging plane with limited size, so the lens of the fisheye camera has a larger field of view compared with the lens of a general camera. The first imaging model can be approximated as a unit sphere projection model. Based on this characteristic, the imaging process of the fisheye camera can be divided into two steps: first, linearly project a three-dimensional space point to a virtual sphere, the center of the sphere coincides with the origin of the camera coordinate system; second, project the point on the virtual sphere to the image plane, which is a nonlinear process. The first imaging model includes the projection mapping relationship of the two steps.

[0041] Step 102, processing the fisheye image through a rotation matrix parameter to obtain a target image.

[0042] The rotation matrix parameter includes a conversion parameter between the first imaging model of the fisheye image and the second imaging model of the target image.

[0043] In the embodiments of the present application, since the labeling quality reduction caused by directly labeling the fisheye image is caused by the picture distortion in the fisheye image, in order to improve the labeling quality, it is necessary to weaken or even eliminate the picture distortion in the fisheye image. The embodiments of the present application can process the fisheye image based on the rotation matrix parameter to obtain a target image without picture distortion or with weak picture distortion, for example, change the fisheye image Figure 2 to the target image Figure 3 .

[0044] Specifically, the screen distortion in the target image is weak because the second imaging model corresponding to the target image is not affected or weakly affected by factors such as a lens during imaging. In an implementation, the second imaging model can be a pinhole imaging model, in which the spatial coordinates of an object and the image coordinates are in a linear mapping relationship, which directly reflects the process of mapping a three-dimensional point in a three-dimensional space to an imaging plane (a two-dimensional space). According to the pinhole imaging model, the target image obtained has no screen distortion or weak screen distortion. In addition, the second imaging model can also be other types of imaging models with weak screen distortion, which are not limited in the embodiments of the present application.

[0045] Further, in order to process the fisheye image to obtain the target image, the pixel points in the fisheye image need to be first mapped to the first camera coordinate system corresponding to the first imaging model, then the pixel points in the first camera coordinate system are mapped to the second camera coordinate system corresponding to the second imaging model, and finally the pixel points in the second camera coordinate system are mapped to the imaging plane to obtain the target image.

[0046] In the above process, the processes of mapping the pixel points in the fisheye image to the first camera coordinate system corresponding to the first imaging model and mapping the pixel points in the second camera coordinate system to the imaging plane can be implemented according to the mapping relationship contained in the first imaging model and the second imaging model. However, the process of mapping the pixel points in the first camera coordinate system to the second camera coordinate system corresponding to the second imaging model needs to be implemented through the rotation matrix parameter for pixel point conversion between the first imaging model and the second imaging model, that is, the product of the coordinates of the pixel points in the second camera coordinate system and the rotation matrix parameter to obtain the coordinates of the pixel points in the first camera coordinate system. The rotation matrix parameter represents the conversion relationship between the pixel points in the second camera coordinate system and the corresponding pixel points in the first camera coordinate system.

[0047] It should be noted that the coordinates of the pixel points in the fisheye image in the mapping process can be floating-point values. Therefore, when the target image is generated by sampling, the pixel values corresponding to the target image can be calculated by using a difference value calculation method such as a nearest neighbor difference value, a bilinear difference value, or a bicubic difference value, so as to generate the target image.

[0048] Step 103, adding a labeling result of the target object in the target image, the labeling result including position information of the target object in the target image.

[0049] In this step, after obtaining the target image as shown in FIG. 10, if the target object is the car 10, the car 10 can be labeled in the target image as shown in FIG. 11 to obtain the target image as shown in FIG. 12. Figure 3 Figure 3 In this step, after obtaining the target image as shown in FIG. 10, if the target object is the car 10, the car 10 can be labeled in the target image as shown in FIG. 11 to obtain the target image as shown in FIG. 12. Figure 4 ​the object frame 20 of the car 10 is accurately labeled, and the object frame 20 reflects the position information of the car 10 in the target image. Since the labeling process is performed in the target image without distortion, the labeling result is less affected by the first imaging model, so that the labeling result has high quality.

[0050] In step 104, the labeling result in the target image is mapped into the fisheye image according to the rotation matrix parameter.

[0051] In the embodiments of the present application, the ultimate goal is to achieve high-quality labeling results of target objects in the fisheye image. Therefore, after obtaining the labeling result in the target image, the labeling result in the target image can be mapped into the fisheye image by using the rotation matrix parameter, so as to obtain the fisheye image containing high-quality labeling results.

[0052] Specifically, the mapping process is to map the coordinates of the labeling result in the target image to the fisheye image, that is, the coordinates of the pixel points corresponding to the labeling result in the target image need to be mapped to the second camera coordinate system corresponding to the second imaging model first, then the pixel points corresponding to the labeling result in the second camera coordinate system are mapped to the first camera coordinate system corresponding to the first imaging model, and finally the pixel points corresponding to the labeling result in the first camera coordinate system are mapped to the fisheye image, so that the fisheye image has the labeling result.

[0053] In the above process, the process of mapping the coordinates of the pixel points corresponding to the labeling result in the target image to the second camera coordinate system corresponding to the second imaging model and mapping the pixel points corresponding to the labeling result in the first camera coordinate system to the fisheye image can be realized according to the mapping relationship contained in the first imaging model and the second imaging model; and for the process of mapping the pixel points corresponding to the labeling result in the second camera coordinate system to the first camera coordinate system corresponding to the first imaging model, the rotation matrix parameter for pixel point conversion between the first imaging model and the second imaging model is needed to realize.

[0054] For example, referring to Figure 4 After obtaining the target image with high-quality labeling frame 20 of the car 10, the labeling result in the target image is mapped into the fisheye image according to the rotation matrix parameter, and the fisheye image as shown in Figure 5 is obtained, in which the car 10 in the fisheye image also has a high-quality labeling frame 30.

[0055] Further, when tracking the target object in the fisheye image by using the tracking algorithm, the target object can be accurately tracked according to the labeling result reflecting the position information of the target object. In addition, when training the recognition model of the target object by using the fisheye image, the fisheye image with the high-quality labeling result as the training data can improve the training effect and efficiency.

[0056] In summary, the image labeling method provided by the embodiment of the present application can convert the fisheye image with the screen distortion into the target image without or with weak distortion through the rotation matrix parameter between the first imaging model of the fisheye image and the second imaging model of the target image, and label the target object in the target image, so that the influence of the screen distortion in the labeling process can be reduced, and the labeling quality can be improved. Finally, the labeling result in the target image is mapped to the fisheye image through the rotation matrix parameter, so that the target object in the fisheye image also has the high-quality labeling result, and the labeling effect is improved. In addition, the entire labeling process can not depend on the expensive 3D sensor, the cost of labeling is reduced, and the economy is improved.

[0057] Figure 6 is a specific flowchart of the image labeling method provided by the embodiment of the present application. The method can include:

[0058] Step 201: acquiring a fisheye image collected by a fisheye camera.

[0059] Specifically, step 201 can specifically refer to step 101 described above, and details are not described herein.

[0060] Step 202: processing the fisheye image through a rotation matrix parameter to obtain a target image.

[0061] The rotation matrix parameter includes a conversion parameter between a first imaging model of the fisheye image and a second imaging model of the target image.

[0062] Specifically, step 202 can specifically refer to step 102 described above, and details are not described herein.

[0063] Step 203: adjusting position information of the target object in the target image, so that the position information of the target object meets a preset labeling condition.

[0064] When the position information of the target object in the screen is poor, the labeling efficiency and quality will be reduced, for example, the position information of the target object cannot make it be completely displayed, the target object is at the edge of the screen, and the like. In the embodiment of the present application, the position information of the target object in the target image can be adjusted, so that the position information of the target object meets the optimal labeling condition, and the labeling quality and efficiency are improved in the labeling process.

[0065] Optionally, the labeling condition comprises: the target object is completely displayed in the target image, and the target object is at the center position information of the target image.

[0066] Specifically, in an implementation manner, the labeling condition capable of improving the labeling quality and the labeling efficiency can be that the target object is completely displayed in the target image and the target object is at the center position information of the target image. The target object being completely displayed in the target image is a basic requirement for labeling. If the target object cannot be completely displayed in the target image, labeling failure or extremely poor labeling quality will occur. The target object being at the center position information of the target image can improve the importance and the attention of the target object and reduce the probability of problems such as a decrease in clarity and content loss when the target object is at an edge position information.

[0067] Optionally, step 203 can specifically comprise:

[0068] Sub-step 2031: The position information of the target object in the target image is changed by adjusting the rotation matrix parameter and the internal parameter corresponding to the second imaging model, so that the position information of the target object meets the preset labeling condition.

[0069] Optionally, in the case where the second imaging model is a pinhole imaging model, the internal parameter corresponding to the second imaging model comprises: a focal length and a center position corresponding to the second imaging model.

[0070] In the embodiment of the application, the rotation matrix parameter can describe the rotation of the camera. In imaging, the affine transformation of the pixel points in the target image can be implemented by multiplying the rotation matrix parameter and the pixel points in the target image. The affine transformation comprises translation, rotation, scaling, shearing and the like. Further, the rotation of the XYZ three coordinate axes of the target image can be implemented by adjusting the rotation matrix parameter, and at most three degrees of freedom of rotation can be implemented, so as to change the position information of the target object in the target image, for example, the target object is moved from the edge of the picture to the center of the picture.

[0071] Further, in the case where the second imaging model is a pinhole imaging model, the internal parameter corresponding to the second imaging model is a parameter related to the characteristics of the camera (virtual camera) corresponding to the second imaging model, such as the center position and the focal length. Therefore, in imaging, the distance of the target object in the picture can be changed by adjusting the center position and the focal length, for example, the target object is zoomed in or zoomed out.

[0072] It should be noted that the adjustment of the position information of the target object can be realized through a visual interface. For example, the position information of the target object is changed by adjusting the rotation matrix parameter through a user's drag operation on the target object in the visual interface, and the size of the target object is changed by adjusting the corresponding internal parameter through a user's zoom-in or zoom-out operation on the target object in the visual interface.

[0073] Step 204: adding a labeling result of the target object in the target image, the labeling result including position information of the target object in the target image.

[0074] Specifically, step 204 can refer to step 103 described above, and details are not described herein.

[0075] Optionally, the labeling result includes a 2D labeling box and a pseudo 3D labeling box.

[0076] In the embodiments of the present application, the 2D labeling box is a labeling box reflecting the position information of the target object in a two-dimensional plane, and the pseudo 3D labeling box is a labeling box reflecting the projection of the three-dimensional image of the target object in a two-dimensional plane. The pseudo 3D labeling box can provide more three-dimensional spatial details of the target object.

[0077] Step 205: mapping the labeling result in the target image to the fisheye image according to a mapping relationship established by the rotation matrix parameter, the first imaging model and the second imaging model.

[0078] Optionally, the first imaging model includes a first internal parameter, the second imaging model includes a second internal parameter, and the mapping relationship includes:

[0079] a corresponding relationship between a coordinate of a first pixel point in the fisheye image established based on the first internal parameter and a coordinate of a second pixel point in a camera coordinate system corresponding to the first imaging model, the second pixel point corresponding to the first pixel point; and a corresponding relationship between a coordinate of a third pixel point in the target image established based on the second internal parameter and a coordinate of a fourth pixel point in the camera coordinate system corresponding to the second imaging model, the fourth pixel point corresponding to the third pixel point; wherein the coordinate of the second pixel point is a product of the coordinate of the fourth pixel point and the rotation matrix parameter.

[0080] In the embodiment of the present application, the mapping process of the annotation result in the target image to the fisheye image includes: mapping the coordinates of the pixel point corresponding to the annotation result in the target image to the second camera coordinate system corresponding to the second imaging model, then mapping the pixel point corresponding to the annotation result in the second camera coordinate system to the first camera coordinate system corresponding to the first imaging model, and finally mapping the pixel point corresponding to the annotation result in the first camera coordinate system to the fisheye image, so that the fisheye image has the annotation result.

[0081] Specifically, assuming that the rotation matrix parameter is R, the first imaging model is a nheta (flat field focusing mirror) projection model, the second imaging model is a pinhole imaging model, the coordinates of the third pixel point in the target image generated by the second imaging model are (u1, v1), the coordinates of the first pixel point in the fisheye image generated by the first imaging model are (u2, v2), the second intrinsic parameters of the second imaging model include optical centers (c x1 , c y1 ), and the focal length is f1, the first intrinsic parameters of the first imaging model include optical centers (c x2 , c y2 ), and the focal length is f2; the coordinates of the fourth pixel point in the camera coordinate system corresponding to the second imaging model are P1=(X1, Y1, Z1) T , and Z1=1, and the coordinates of the second pixel point in the camera coordinate system corresponding to the first imaging model are P2=(X2, Y2, Z2) T .

[0082] Therefore, according to the above definitions, the following mapping relationship is known:

[0083] / / Reflect the mapping relationship between the coordinates of the third pixel point and the coordinates of the fourth pixel point

[0084] / / Reflect the mapping relationship between the coordinates of the third pixel point and the coordinates of the fourth pixel point

[0085] Z1=1;

[0086] P2=R·P1; / / Reflect the mapping relationship between the coordinates of the fourth pixel point and the coordinates of the second pixel point

[0087] Further, define arc=arctan2(n, Z2), then:

[0088]

[0089]

[0090] Therefore, based on the above, it can be known that the embodiment of the application establishes a mapping relationship from the coordinate (u1, v1) of the third pixel point in the target image generated by the second imaging model to the coordinate (u2, v2) of the first pixel point in the fisheye image generated by the first imaging model according to the rotation matrix parameter, the first imaging model and the second imaging model. According to the mapping relationship, the annotation result in the target image can be mapped to the fisheye image, so as to obtain the fisheye image containing the high-quality annotation result.

[0091] It should be noted that according to the mapping relationship established by the rotation matrix parameter, the first imaging model and the second imaging model, the mapping table between the pixel point coordinates established according to the mapping relationship between the rotation matrix parameter, the first imaging model and the second imaging model can also be obtained. By looking up the coordinates of the pixel points of the target image, the coordinates of the corresponding pixel points in the fisheye image can be obtained. Similarly, by looking up the coordinates of the pixel points in the fisheye image, the coordinates of the corresponding pixel points in the target image can be obtained.

[0092] Optionally, after step 205, it can further include:

[0093] Step 206, calculating the motion parameter of the target object according to the preset tracking algorithm and the annotation result, the motion parameter including one or more of speed, position information coordinate, moving direction and acceleration.

[0094] In an application scenario of the embodiment of the application, the preset tracking algorithm and the annotation result of the target object can be used to track the target object in the fisheye image. Specifically, the motion state of the target object can be tracked according to the fisheye images continuously collected by the fisheye camera and the annotation result of the target object in the fisheye image, specifically tracking the speed, position information coordinate, moving direction and acceleration of the target object. When the target object has a high-quality annotation result in the fisheye image, the motion parameter determined according to the tracking algorithm is more accurate.

[0095] Optionally, after step 205, it can further include:

[0096] Step 207, taking the fisheye image with the annotation result as training data.

[0097] Step 208, training a target model through the training data to obtain the model parameter of the target model, the target model being used to identify the target object corresponding to the annotation result.

[0098] In another application scenario of the embodiment of the present application, when there is a functional requirement for identifying a target object, there is also a requirement for accurately labeling the target object in the fisheye image as training data. When the fisheye image is used to train a target model for identifying the target object, the fisheye image with a high-quality labeling result can be used as training data, so that the model parameters of the target model are obtained through training of the target model, and the training effect and efficiency can be improved when the fisheye image with a high-quality labeling result is used as training data.

[0099] In summary, the image labeling method provided by the embodiment of the present application can convert the fisheye image with picture distortion into the target image without distortion or with weak distortion through the rotation matrix parameter between the first imaging model of the fisheye image and the second imaging model of the target image, and label the target object in the target image, so that the influence of picture distortion on the labeling process can be reduced and the labeling quality can be improved. Finally, the labeling result in the target image is mapped into the fisheye image through the rotation matrix parameter, so that the target object in the fisheye image also has a high-quality labeling result, and the labeling effect is improved. In addition, the entire labeling process can not depend on expensive 3D sensors, the cost of labeling is reduced, and the economy is improved.

[0100] Figure 7 is a block diagram of an image labeling apparatus provided by the embodiment of the present application, as shown in Figure 7 the image labeling apparatus 300 can include a memory 301 and a processor 302;

[0101] The memory 301 is configured to acquire a fisheye image collected by a fisheye camera.

[0102] The processor 302 is configured to:

[0103] process the fisheye image through a rotation matrix parameter to obtain a target image, the rotation matrix parameter including a conversion parameter between a first imaging model of the fisheye image and a second imaging model of the target image;

[0104] add a labeling result of a target object in the target image, the labeling result including position information of the target object in the target image;

[0105] map the labeling result in the target image into the fisheye image according to the rotation matrix parameter.

[0106] Optionally, the processor 302 is specifically configured to:

[0107] adjust the position information of the target object in the target image, so that the position information of the target object meets a preset labeling condition.

[0108] Optionally, the processor 302 is specifically configured to:

[0109] by adjusting the rotation matrix parameter and the internal parameter corresponding to the second imaging model, thereby changing the position information of the target object in the target image, so that the position information of the target object meets the preset annotation condition.

[0110] Optionally, in the case where the second imaging model is a pinhole imaging model, the internal parameter corresponding to the second imaging model includes: a focal length and a center of light corresponding to the second imaging model.

[0111] Optionally, the annotation condition includes:

[0112] the target object is completely displayed in the target image, and the target object is at a central position in the target image.

[0113] Optionally, the processor 302 is specifically configured to:

[0114] According to the mapping relationship established by the rotation matrix parameter, the first imaging model and the second imaging model, the annotation result in the target image is mapped to the fisheye image.

[0115] Optionally, the first imaging model includes a first internal parameter, and the second imaging model includes a second internal parameter, and the mapping relationship includes:

[0116] a corresponding relationship between a first pixel point in the fisheye image established based on the first internal parameter and a second pixel point in a camera coordinate system corresponding to the first imaging model, the second pixel point corresponding to the first pixel point;

[0117] a corresponding relationship between a third pixel point in the target image established based on the second internal parameter and a fourth pixel point in a camera coordinate system corresponding to the second imaging model, the fourth pixel point corresponding to the third pixel point;

[0118] wherein the coordinate of the second pixel point is a product of the coordinate of the fourth pixel point and the rotation matrix parameter.

[0119] Optionally, the processor 302 is further configured to:

[0120] According to a preset tracking algorithm and the annotation result, a motion parameter of the target object is calculated, the motion parameter including one or more of: speed, position information coordinate, moving direction, and acceleration.

[0121] Optionally, the processor 302 is further configured to:

[0122] The fisheye image with the annotation result is taken as training data;

[0123] A target model is trained through the training data to obtain model parameters of the target model, and the target model is used to identify a target object corresponding to the annotation result.

[0124] Optionally, the annotation result includes a 2D annotation box and a pseudo 3D annotation box.

[0125] In summary, the image annotation device provided by the embodiment of the present application converts the fisheye image with picture distortion into the target image without distortion or with weak distortion through the rotation matrix parameter between the first imaging model of the fisheye image and the second imaging model of the target image, and performs annotation of the target object on the target image, so that the influence of picture distortion in the annotation process can be reduced, and the annotation quality can be improved. Finally, the annotation result in the target image is mapped to the fisheye image through the rotation matrix parameter, so that the target object in the fisheye image also has a high-quality annotation result, and the annotation effect is improved. In addition, the entire annotation process can not depend on expensive 3D sensors, and the cost of annotation is reduced, and the economy is improved.

[0126] The embodiment of the present application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement each process of the image annotation method embodiment, and the same technical effect can be achieved. To avoid repetition, details are not repeated here. The computer readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like.

[0127] The memory can be an interface for connecting the external control terminal and the image annotation device. For example, the external control terminal can include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a control terminal with an identification module, an audio input / output (I / O) port, a video I / O port, a headset port, and the like. The memory can be used to receive input (for example, data information, power, and the like) from the external control terminal and transmit the received input to one or more elements within the image annotation device, or can be used to transmit data between the image annotation device and the external control terminal.

[0128] For example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0129] The processor is a control center of the terminal, connects each part of the terminal by using various interfaces and lines, executes various functions of the terminal and processes data by running or executing software programs and / or modules stored in the memory and calling data stored in the memory, thereby performing overall monitoring on the terminal. The processor can include one or more processing units; preferably, the processor can integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces, and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor.

[0130] Each embodiment in the specification is described in a progressive manner, and each embodiment mainly explains the difference from other embodiments, and the same or similar parts between each embodiment can be referred to each other.

[0131] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a terminal, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0132] The present application is described with reference to flowcharts and / or block diagrams according to the method, terminal device (system), and computer program product of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal device produce a machine that implements the functions specified in the flowchart and / or block diagram. Figure 1 one or more flows and / or blocks Figure 1 a terminal that controls the functions specified in one or more blocks.

[0133] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product that includes an instruction terminal, and the instruction terminal implements the functions specified in the flowchart and / or block diagram. Figure 1 one or more flows and / or blocks Figure 1 a terminal that controls the functions specified in one or more blocks.

[0134] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operational steps are performed on the computer or other programmable terminal device to generate a computer implemented process, so that the instructions executed on the computer or other programmable terminal device provide a process for implementing the functions specified in the flowchart block(s) or block(s). Figure 1 These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operational steps are performed on the computer or other programmable terminal device to generate a computer implemented process, so that the instructions executed on the computer or other programmable terminal device provide a process for implementing the functions specified in the flowchart block(s) or block(s). Figure 1 Figure 1 These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operational steps are performed on the computer or other programmable terminal device to generate a computer implemented process, so that the instructions executed on the computer or other programmable terminal device provide a process for implementing the functions specified in the flowchart block(s) or block(s).

[0135] Although the preferred embodiments of the application have been described, those skilled in the art will be able to make additional changes and modifications to these embodiments once they have the basic inventive concept. Therefore, the appended claims are intended to cover all changes and modifications falling within the scope of the application.

[0136] Finally, it should be noted that the relational terms herein, such as first and second, and the like, are used solely to distinguish one from another entity or action, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or terminal device. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or terminal device that comprises the element.

[0137] The above has been a detailed description of the application, and the principles and implementation modes of the application have been described herein using specific examples; the above description of the embodiments is only to help understand the method of the application and its core idea; at the same time, for those skilled in the art, according to the idea of the application, there will be changes in the specific implementation mode and application range; in view of the above, the content of the specification should not be understood as a limitation of the application.

Claims

1. An image labeling method, characterized by, The method comprises: acquiring a fisheye image collected by a fisheye camera; processing the fisheye image through a rotation matrix parameter to obtain a target image, the rotation matrix parameter comprising a conversion parameter between a first imaging model of the fisheye image and a second imaging model of the target image; the rotation matrix parameter is used to map a pixel point in a first camera coordinate system corresponding to the first imaging model to a second camera coordinate system corresponding to the second imaging model; adjusting the rotation matrix parameter and an internal parameter corresponding to the second imaging model to change position information of a target object in the target image, so that the position information of the target object meets a preset labeling condition; adding a labeling result of a target object in the target image, the labeling result comprising position information of the target object in the target image; mapping the labeling result in the target image to the fisheye image according to the rotation matrix parameter.

2. The method of claim 1, wherein, In the case that the second imaging model is a pinhole imaging model, the internal parameter corresponding to the second imaging model comprises a light center position and a focal length corresponding to the second imaging model.

3. The method of claim 1, wherein, The labeling condition comprises: the target object is completely displayed in the target image, and the target object is at a central position in the target image.

4. The method of claim 1, wherein, The mapping of the labeling result in the target image to the fisheye image according to the rotation matrix parameter comprises: mapping the labeling result in the target image to the fisheye image according to a mapping relationship established by the rotation matrix parameter, the first imaging model and the second imaging model.

5. The method of claim 4, wherein, The first imaging model comprises first internal parameters, the second imaging model comprises second internal parameters, and the mapping relationship comprises: a corresponding relationship between a coordinate of a first pixel point in the fisheye image established based on the first internal parameters and a coordinate of a second pixel point in a camera coordinate system corresponding to the first imaging model, the second pixel point corresponding to the first pixel point; a corresponding relationship between a coordinate of a third pixel point in the target image established based on the second internal parameters and a coordinate of a fourth pixel point in a camera coordinate system corresponding to the second imaging model, the fourth pixel point corresponding to the third pixel point; wherein the coordinate of the second pixel point is a product of the coordinate of the fourth pixel point and the rotation matrix parameter.

6. The method of claim 1, wherein, After the mapping of the labeling result in the target image to the fisheye image, the method further comprises: calculating a motion parameter of the target object according to a preset tracking algorithm and the labeling result, the motion parameter comprising one or more of a speed, a position information coordinate, a moving direction and an acceleration.

7. The method of claim 1, wherein, After the mapping of the labeling result in the target image to the fisheye image, the method further comprises: taking the fisheye image with the labeling result as training data; training a target model through the training data to obtain a model parameter of the target model, the target model being used to identify a target object corresponding to the labeling result.

8. The method according to any one of claims 1 to 7, characterized in that, The labeling result comprises a 2D labeling box and a pseudo 3D labeling box.

9. An image labeling apparatus characterized by comprising: The device comprises a memory and a processor; The memory is configured to acquire a fisheye image collected by a fisheye camera; The processor is configured to: process the fisheye image through a rotation matrix parameter to obtain a target image, the rotation matrix parameter comprising a conversion parameter between a first imaging model of the fisheye image and a second imaging model of the target image; and map a pixel point in a first camera coordinate system corresponding to the first imaging model to a second camera coordinate system corresponding to the second imaging model through the rotation matrix parameter; add a labeling result of a target object in the target image, the labeling result comprising position information of the target object in the target image; map the labeling result in the target image to the fisheye image according to the rotation matrix parameter; The processor is specifically configured to: adjust the rotation matrix parameter and an internal parameter corresponding to the second imaging model to change the position information of the target object in the target image, so that the position information of the target object satisfies a preset labeling condition.

10. The apparatus of claim 9, wherein, In a case where the second imaging model is a pinhole imaging model, the internal parameter corresponding to the second imaging model comprises a principal point position and a focal length corresponding to the second imaging model.

11. The apparatus of claim 9, wherein, The labeling condition comprises: the target object is completely displayed in the target image, and the target object is at a central position in the target image.

12. The apparatus of claim 9, wherein, The processor is specifically configured to: map the labeling result in the target image to the fisheye image according to a mapping relationship established by the rotation matrix parameter, the first imaging model and the second imaging model.

13. The apparatus of claim 12, wherein, The first imaging model comprises first internal parameters, the second imaging model comprises second internal parameters, and the mapping relationship comprises: a corresponding relationship between a coordinate of a first pixel point in the fisheye image established based on the first internal parameters and a coordinate of a second pixel point in a camera coordinate system corresponding to the first imaging model, the second pixel point corresponding to the first pixel point; a corresponding relationship between a coordinate of a third pixel point in the target image established based on the second internal parameters and a coordinate of a fourth pixel point in a camera coordinate system corresponding to the second imaging model, the fourth pixel point corresponding to the third pixel point; wherein the coordinate of the second pixel point is a product of the coordinate of the fourth pixel point and the rotation matrix parameter.

14. The apparatus of claim 9, wherein, The processor is further configured to: calculate a motion parameter of the target object according to a preset tracking algorithm and the labeling result, the motion parameter comprising one or more of a speed, a position information coordinate, a moving direction and an acceleration.

15. The apparatus of claim 9, wherein The processor is further configured to: use the fisheye image with the labeling result as training data; train a target model through the training data to obtain a model parameter of the target model, the target model being used to identify a target object corresponding to the labeling result.

16. The apparatus of any one of claims 9 to 15, wherein, The labeling result comprises a 2D labeling box and a pseudo 3D labeling box.

17. An electronic device, comprising: An image annotation method as claimed in any one of claims 1 to 8, comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, which, when executed by the processor, implements the image annotation method.

18. A computer-readable storage medium, characterized in that, An image annotation method as claimed in any one of claims 1 to 8, comprising instructions which, when executed on a computer, cause the computer to perform the image annotation method.

Images