Image processing method and device and panoramic camera

Abstract

The invention is suitable for the technical field of image processing, and provides an image processing method and device and a panorama camera, the image processing method is applied to the panorama camera, the panorama camera can perform data transmission with specified sensor equipment, and the panorama camera obtains sensor attitude data corresponding to a specified time period; acquiring a head rotation angle according to the sensor attitude data; obtaining a human phase distance corresponding to an end point moment of a specified time period; calculating theta c according to the head rotation angle, the human phase distance and the phase object distance; and outputting an image picture corresponding to the theta c. That is, the panoramic camera can accurately and flexibly output the image expected to be seen by the user.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and panoramic camera. Background Art

[0002] With the continuous development of technology, more and more devices have sprung up, such as panoramic cameras, which can shoot based on a 360° shooting angle. However, panoramic cameras still have some defects that need to be solved urgently.

[0003] Currently, although panoramic cameras can shoot at a 360° shooting angle, they can only present planar image content within a small viewing angle range, which is generally a fixed viewing angle in the inertial coordinate system of the panoramic camera. Panoramic cameras cannot accurately and flexibly output the images that users expect to see. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device and panoramic camera, which can solve the problem that the current panoramic cameras cannot accurately and flexibly output the image pictures that the user expects to see.

[0005] In a first aspect, the present application provides an image processing method, which is applied to a panoramic camera, wherein the panoramic camera can transmit data with a designated sensor device, wherein the designated sensor device is a sensor worn on the user's head, and the image processing method comprises: obtaining sensor posture data corresponding to a designated time period, wherein the sensor posture data is used to represent the posture of the designated sensor device; obtaining a head rotation angle according to the sensor posture data, wherein the head rotation angle is the angle of rotation of the user's head within the designated time period; obtaining a person-to-person distance corresponding to an end moment of the designated time period, wherein the person-to-person distance refers to the distance between the panoramic camera and a body point of the user, wherein the body point of the user is a point in the user's body located on the median sagittal plane of the user's body; calculating θc according to the head rotation angle, the person-to-person distance and the object-to-object distance, wherein θc is the direction angle corresponding to the target object in the inertial coordinate system of the panoramic camera, and the object-to-object distance is the distance between the target object and the origin of the inertial coordinate system; and outputting an image corresponding to θc.

[0006] In a possible implementation manner of the first aspect, calculating θc according to the head rotation angle, the person-object distance, and the person-object distance includes:

[0007] Calculate θ according to the head rotation angle, the person-object distance, the object-object distance and the specified relationship. C , the specified relationship is Among them, the O CP is used to represent the phase-object distance, and the θ r Equal to 90 degrees and θ e The difference between the θ e Used to represent the head rotation angle, D T Used to indicate the distance between the persons.

[0008] In a possible implementation manner of the first aspect, in the step of calculating θ according to the head rotation angle, the person-object distance, the object-object distance, and a specified relationship. G Previously, including:

[0009] Calculate θ according to the head rotation angle, the distance between the person and the preset value B , the θ B equal Wherein, B is used to represent the preset value;

[0010] Determine the θ in the target panoramic image B an area of ​​interest within the corresponding image area, the target panoramic image being a panoramic image captured by the panoramic camera at the end moment of the specified time period, and the area of ​​interest being used to represent the target object;

[0011] A phase-out object distance is determined based on the region of interest.

[0012] In a possible implementation manner of the first aspect, the panoramic camera and the first end of the selfie stick are fixedly connected, and correspondingly, obtaining the human distance corresponding to the end moment of the specified time period includes: obtaining the pole distance corresponding to the end moment of the specified time period, the pole distance being the distance between the panoramic camera and the second end of the selfie stick; obtaining the human transmission distance corresponding to the end moment of the specified time period, the human transmission distance being the distance between the user's body point and the specified sensor device; and obtaining the human distance corresponding to the end moment of the specified time period according to the pole distance and the human transmission distance.

[0013] In a possible implementation of the first aspect, before obtaining the distance between people corresponding to the end moment of the specified time period, it includes: obtaining a user image, where the user image is an image obtained by the panoramic camera based on the shooting of the user at the end moment of the specified time period; determining key points of the human body based on the user image; correspondingly, obtaining the distance between people corresponding to the end moment of the specified time period includes: obtaining the distance between people corresponding to the end moment of the specified time period based on a first feature point position, where the first feature point position is a position of a pixel corresponding to the key point of the human body in the user image.

[0014] In a possible implementation of the first aspect, the lens in the panoramic camera is a fisheye lens, the user image is a fisheye image, and the method of obtaining the distance between people corresponding to the end moment of the specified time period based on the first feature point position includes: determining the second feature point position based on the first feature point position and a specified mapping relationship, the specified mapping relationship is used to represent the mapping relationship between the fisheye image and a specified spherical coordinate system, the second feature point position is used to represent the coordinates of the point corresponding to the key point of the human body mapped to the specified spherical coordinate system; obtaining the distance between people corresponding to the end moment of the specified time period based on the second feature point position.

[0015] In a possible implementation of the first aspect, the second feature point position includes a left eye feature point position and a right eye feature point position, and correspondingly, obtaining the distance between people corresponding to the end moment of the specified time period based on the second feature point position includes: obtaining a target field of view angle based on the second feature point position, the target field of view angle being the angle between a first straight line segment and a second straight line segment, the two endpoints of the first straight line segment including the optical center of the fisheye lens and the left eye of the user's head, and the two endpoints of the second straight line segment including the optical center of the fisheye lens and the right eye of the user's head; obtaining the distance between people corresponding to the end moment of the specified time period according to the target field of view angle and the user's pupil distance.

[0016] In a second aspect, an embodiment of the present application provides an image processing device, which is applied to a panoramic camera, and the panoramic camera can transmit data with a designated sensor device, and the designated sensor device is a sensor worn on the user's head. The image processing device includes: a first acquisition unit, which is used to acquire sensor posture data corresponding to a specified time period, and the sensor posture data is used to represent the posture of the designated sensor device; a second acquisition unit, which is used to acquire a head rotation angle according to the sensor posture data, and the head rotation angle is the angle of rotation of the user's head within the specified time period; a third acquisition unit, which is used to acquire a person-to-person distance corresponding to the end moment of the specified time period, and the person-to-person distance refers to the distance between the panoramic camera and a user's body point, wherein the user's body point is: a point in the user's body located on the median sagittal plane of the user's body; a calculation unit, which is used to calculate θc according to the head rotation angle, the person-to-person distance and the object-to-object distance, and the θc is: the direction angle corresponding to the target object in the inertial coordinate system of the panoramic camera, and the object-to-object distance is the distance between the target object and the origin of the inertial coordinate system; an output unit, which is used to output an image corresponding to θc.

[0017] In a third aspect, an embodiment of the present application provides a panoramic camera, comprising a display, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the panoramic camera is capable of transmitting data with a designated sensor device, wherein the designated sensor device is a sensor worn on the user's head, and when the processor executes the computer program, the method described in any one of the first aspects above is implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the first aspects above is implemented.

[0019] In an embodiment of the present application, the panoramic camera can transmit data with a designated sensor device, where the designated sensor device is a sensor worn on the user's head. The panoramic camera obtains sensor posture data corresponding to a designated time period, where the sensor posture data is used to represent the posture of the designated sensor device; the head rotation angle is obtained based on the sensor posture data, where the head rotation angle is the angle at which the user's head rotates within the designated time period; the person-to-person distance corresponding to the end moment of the designated time period is obtained, where the person-to-person distance refers to the distance between the panoramic camera and a body point of the user; θc is calculated based on the head rotation angle, the person-to-person distance, and the object-to-object distance, where θc is: the direction angle corresponding to the target object in the inertial coordinate system of the panoramic camera, where the object-to-object distance is the distance between the target object and the origin of the inertial coordinate system; and an image corresponding to θc is output. Among them, the sensor posture data can accurately reflect the user's head rotation, and the user's head rotation can reflect the user's viewing direction of the environment, and the user's viewing direction of the environment is generally the direction of the target object (the object of interest to the user). Based on the head rotation angle, the person-object distance and the object distance, the direction of the target object in the inertial coordinate system can be calculated more accurately (represented by θc), and then the panoramic camera outputs the image corresponding to θc, that is, it can accurately and flexibly output the image that the user expects to see. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present application;

[0022] Figure 2is a schematic diagram of an image processing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] It should be noted that the terms "comprises" and "has" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or equipment.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0026] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it may be directly on the other element or there may be an intermediate element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element at the same time.

[0027] In this article, the term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.

[0028] Embodiment 1

[0029] Figure 1The flowchart of an image processing method provided by an embodiment of the present application is shown. The image processing method is applied to a panoramic camera. The panoramic camera can transmit data with a designated sensor device. The designated sensor device is a sensor worn on the user's head. The image processing method includes steps S101, S102, S103, S104, and S105. The details are as follows:

[0030] Step S101: Acquire sensor posture data corresponding to a specified time period, wherein the sensor posture data is used to represent the posture of the specified sensor device.

[0031] As an example but not limitation, the user wears a hat and the designated sensor device is set on the hat; or the user wears glasses and the designated sensor device is set on the glasses. The designated sensor device is provided with a gyroscope, and the designated sensor device obtains sensor posture data corresponding to a designated time period through the gyroscope, and then the designated sensor device sends the sensor posture data to the panoramic camera, and the panoramic camera receives the sensor posture data.

[0032] Step S102: Acquire a head rotation angle according to the sensor posture data, where the head rotation angle is an angle at which the user's head rotates within the specified time period.

[0033] Specifically, the head rotation angle is the angle between a first head orientation and a second head orientation, the first head orientation is: the frontal orientation of the user's head at the start moment of a specified time period, the second head orientation is: the frontal orientation of the user's head at the end moment of the specified time period, and the frontal orientation at the same moment is consistent with the user's line of sight direction.

[0034] Optionally, before step S102, the panoramic camera establishes an eye coordinate system, which is a three-dimensional coordinate system. The eye coordinate system takes the eye midpoint as the origin, and the eye midpoint is the midpoint between the two eyes of the user's head. The direction of the first axis of the eye coordinate system is: the eye midpoint points to the direction of one of the eyes of the user's head. For example, the positive direction of the first axis of the eye coordinate system is: the eye midpoint points to the direction of the right eye of the user's head. The direction of the second axis of the eye coordinate system is the user's line of sight, and the third axis of the eye coordinate system is perpendicular to the plane formed by the first axis and the second axis of the eye coordinate system. The eye coordinate system in this embodiment rotates with the rotation of the head. Correspondingly, the first head direction is equal to: the positive direction of the first axis of the eye coordinate system at the start time of the specified time period, and the second head direction is equal to: the positive direction of the first axis of the eye coordinate system at the end time of the specified time period.

[0035] Step S103: obtaining the human distance corresponding to the end time of the specified time period, where the human distance refers to the distance between the panoramic camera and the user's body point.

[0036] The user body point is a point on the user's body located on the median sagittal plane of the user's body.

[0037] As an example and not limitation, the user body point may be an eye midpoint, which is the midpoint between the two eyes of the user's head, and is located on the median sagittal plane of the user's body; or, the user body point may be a shoulder midpoint, which is the midpoint between the left shoulder and the right shoulder of the user's body, and is located on the median sagittal plane of the user's body point.

[0038] Specifically, the human distance may be the horizontal distance between the panoramic camera and a body point of the user.

[0039] In some embodiments, the panoramic camera is on the same horizontal plane as the user's body point.

[0040] In some embodiments, the panoramic camera and the user's body point are on different horizontal planes. Correspondingly, the human distance specifically refers to: the straight-line distance between a first point and a second point on a specified horizontal plane, the first point is the point formed by the panoramic camera projected onto the specified horizontal plane, and the second point is the point formed by the user's body point projected onto the specified horizontal plane.

[0041] Optionally, the panoramic camera is fixedly connected to the first end of the selfie stick, and correspondingly, obtaining the distance between people corresponding to the end moment of the specified time period includes: obtaining the distance between the poles corresponding to the end moment of the specified time period, the distance between the poles is the distance between the panoramic camera and the second end of the selfie stick; obtaining the distance between people corresponding to the end moment of the specified time period, the distance between people is the distance between the user's body point and the specified sensor device; obtaining the distance between people corresponding to the end moment of the specified time period according to the distance between the poles and the distance between people. That is, this embodiment is applicable to the scenario where the panoramic camera is fixedly connected to the selfie stick, and this embodiment can accurately calculate the distance between people in this scenario.

[0042] Among them, the two ends of the selfie stick include a first end and a second end, the distance between the sticks can be specifically the horizontal distance between the panoramic camera and the second end of the selfie stick, and the human transmission distance can be specifically the horizontal distance between the user's body point and the designated sensor device.

[0043] In some embodiments, obtaining the rod distance corresponding to the end moment of the specified time period includes: obtaining the phase-rod angle at the end moment of the specified time period, the phase-rod angle being the angle between the direction of the main optical axis of the lens of the panoramic camera and the direction of the rod length of the selfie stick, and obtaining the rod distance corresponding to the end moment of the specified time period based on the phase-rod angle and the rod length of the selfie stick.

[0044] In some embodiments, obtaining the human distance corresponding to the end time of the specified time period based on the pole distance and the human transmission distance includes: determining the sum of the pole distance and the human transmission distance as the human distance corresponding to the end time of the specified time period.

[0045] Optionally, before obtaining the distance between people corresponding to the end time of the specified time period, it includes: obtaining a user image, the user image is: an image obtained by the panoramic camera based on the user at the end time of the specified time period; determining the key points of the human body based on the user image; correspondingly, obtaining the distance between people corresponding to the end time of the specified time period includes: obtaining the distance between people corresponding to the end time of the specified time period based on the position of the first feature point, the first feature point position is the position of the pixel corresponding to the key point of the human body in the user image. That is, this embodiment can accurately calculate the distance between people based on the user image.

[0046] As an example but not limitation, determining the human body key points based on the user image includes: determining the human body key points for the user image using a specified human body key point recognition algorithm, the specified human body key point recognition algorithm is an algorithm for identifying human body key points, the human body key points include eye feature points and / or shoulder key points, the eye feature points are key points used to represent eyes, the shoulder key points are key points used to represent shoulders, and the specified human body key point recognition algorithm may be any one of the following algorithms: OpennPose algorithm, AlphaPose algorithm, MobilePose algorithm.

[0047] Optionally, the lens in the panoramic camera is a fisheye lens, the user image is a fisheye image, and the method of obtaining the distance between people corresponding to the end moment of the specified time period based on the first feature point position includes: determining the second feature point position based on the first feature point position and a specified mapping relationship, the specified mapping relationship is used to represent the mapping relationship between the fisheye image and a specified spherical coordinate system, the second feature point position is used to represent the coordinates of the points corresponding to the key points of the human body mapped to the specified spherical coordinate system; obtaining the distance between people corresponding to the end moment of the specified time period based on the second feature point position.

[0048] Wherein, the fisheye image is a planar image.

[0049] As an example but not limitation, the human body key points are eye feature points, and the eye feature points include left eye feature points and right eye feature points. The left eye feature points are feature points used to represent the left eye (of the user's head), and the right eye feature points are feature points used to represent the right eye (of the user's head). Correspondingly, the second feature point positions include left eye feature point positions and right eye feature point positions. The left eye feature point positions are used to represent the coordinates of the points corresponding to the left eye feature points mapped to the specified spherical coordinate system, and the right eye feature point positions are used to represent the coordinates of the points corresponding to the right eye feature points mapped to the specified spherical coordinate system. The specified spherical coordinate system may be a unit spherical coordinate system.

[0050] For example, the position of the left eye feature point can be expressed as (α1, θ1), and the position of the right eye feature point can be expressed as (α2, θ2). A two-dimensional rectangular coordinate system can be established based on the fisheye image, wherein the direction of the first axis of the two-dimensional rectangular coordinate system is the horizontal direction, and the second axis of the two-dimensional rectangular coordinate system is perpendicular to the first axis of the two-dimensional rectangular coordinate system, wherein α1 is used to represent the angle between the line from the origin of the two-dimensional rectangular coordinate system to the left eye feature point and the first axis of the two-dimensional rectangular coordinate system, and θ1 is used to represent the angle between the first incident light and the main optical axis direction of the fisheye lens, and the first incident light is the incident light in the direction of the line connecting the point corresponding to the left eye feature point mapped to the specified spherical coordinate system and the optical center of the fisheye lens, α2 is used to represent the angle between the line connecting the origin of the two-dimensional rectangular coordinate system to the right eye feature point and the first axis of the two-dimensional rectangular coordinate system, θ2 is used to represent the angle between the second incident light and the main optical axis direction of the fisheye lens, the second incident light is the incident light in the direction of the line connecting the point corresponding to the right eye feature point mapped to the specified spherical coordinate system and the optical center of the fisheye lens, and the optical center of the fisheye lens may coincide with the center of a sphere in the specified spherical coordinate system.

[0051] Optionally, the second feature point position includes a left eye feature point position and a right eye feature point position, and correspondingly, obtaining the distance between people corresponding to the end moment of the specified time period based on the second feature point position includes: obtaining a target field of view angle based on the second feature point position, the target field of view angle being the angle between a first straight line segment and a second straight line segment, the two endpoints of the first straight line segment including the optical center of the fisheye lens and the left eye of the user's head, and the two endpoints of the second straight line segment including the optical center of the fisheye lens and the right eye of the user's head; obtaining the distance between people corresponding to the end moment of the specified time period according to the target field of view angle and the user's pupil distance.

[0052] As an example but not a limitation, the target field of view angle is obtained based on the second feature point position, Δθ=sin(θ1)sin(θ2)+cos(θ1)cos(θ2)cos(α2-α1), where Δθ is used to represent the target field of view angle. The distance between people corresponding to the end time of the specified time period is obtained based on Δθ and the user's pupil distance, where Δθ, the user's pupil distance, and the distance between people corresponding to the end time of the specified time period can satisfy the following relationship: D T =De / Δθ, where D T It is used to indicate the distance between people corresponding to the end time of the specified time period, and De is used to indicate the user's pupil distance.

[0053] Step S104, calculating θc according to the head rotation angle, the person-object distance and the object-object distance, wherein θc is the direction angle corresponding to the target object in the inertial coordinate system of the panoramic camera, and the object-object distance is the distance between the target object and the origin of the inertial coordinate system.

[0054] The target object is the object (object) that the user is interested in. θc is the direction angle corresponding to the target object in the inertial coordinate system of the panoramic camera, that is, θc can be used to indicate the direction of the target object. The inertial coordinate system does not rotate with the rotation of the user's body (part), for example, the inertial coordinate system does not rotate with the rotation of the user's head.

[0055] Since there is a certain distance between the panoramic camera and the user's eyes, for example, when the panoramic camera is connected to a selfie stick, there is a large distance between the panoramic camera and the user's eyes, the head rotation angle cannot be directly determined as θc. Instead, θc should be accurately calculated based on the head rotation angle, the distance between the person and the object.

[0056] Optionally, the step S104 includes: calculating θ according to the head rotation angle, the person-object distance, the object-object distance and a specified relationship. C , the specified relationship is Among them, the O C P is used to represent the phase-object distance, and the θ r Equal to 90 degrees and θ e The difference between the θ e Used to represent the head rotation angle, D T Used to indicate the distance between the persons.

[0057] Optionally, in the step of calculating θ according to the head rotation angle, the person-object distance, the object-object distance and a specified relationship G Before, it includes: calculating θ according to the head rotation angle, the distance between the person and the preset value B, where θ B equal Wherein, B is used to represent the preset value; determining the θ in the target panoramic image B The target panoramic image is a panoramic image obtained by the panoramic camera based on the end time of the specified time period, and the region of interest is used to represent the target object; the object distance is determined based on the region of interest.

[0058] For example, assuming that the preset value is 5 meters, when θ B Afterwards, the θ in the target panoramic image is determined by a visual saliency algorithm. B The (user's) region of interest in the corresponding image area, that is, the target object is determined, and then the depth (information) corresponding to the region of interest is extracted, and the depth is determined as the distance to the object. As an example but not limitation, as an example but not limitation, the visual saliency algorithm can be any of the following: Itti algorithm, SR algorithm, FT algorithm, HC algorithm.

[0059] Step S105: outputting the image frame corresponding to the θc.

[0060] Among them, the image picture corresponding to θc is a plane image, which can be used for image preview, video generation, live broadcast and other functions of the panoramic camera.

[0061] In an embodiment of the present application, the panoramic camera can transmit data with a designated sensor device, where the designated sensor device is a sensor worn on the user's head. The panoramic camera obtains sensor posture data corresponding to a designated time period, where the sensor posture data is used to represent the posture of the designated sensor device; the head rotation angle is obtained based on the sensor posture data, where the head rotation angle is the angle at which the user's head rotates within the designated time period; the person-to-person distance corresponding to the end moment of the designated time period is obtained, where the person-to-person distance refers to the distance between the panoramic camera and a body point of the user; θc is calculated based on the head rotation angle, the person-to-person distance, and the object-to-object distance, where θc is: the direction angle corresponding to the target object in the inertial coordinate system of the panoramic camera, where the object-to-object distance is the distance between the target object and the origin of the inertial coordinate system; and an image corresponding to θc is output. Among them, the sensor posture data can accurately reflect the user's head rotation, and the user's head rotation can reflect the user's viewing direction of the environment, and the user's viewing direction of the environment is generally the direction of the target object (the object of interest to the user). Based on the head rotation angle, the person-object distance and the object distance, the direction of the target object in the inertial coordinate system can be calculated more accurately (represented by θc), and then the panoramic camera outputs the image corresponding to θc, that is, it can accurately and flexibly output the image that the user expects to see.

[0062] Embodiment 2

[0063] Corresponding to the image processing method described in the above embodiment, an embodiment of the present application provides an image processing device, which is applied to a panoramic camera. The panoramic camera can transmit data with a designated sensor device, and the designated sensor device is a sensor worn on the user's head. Figure 2 A schematic diagram of an image processing device provided in an embodiment of the present application is shown. The image processing device comprises: a first acquisition unit 201 , a second acquisition unit 202 , a third acquisition unit 203 , a calculation unit 204 and an output unit 205 .

[0064] The first acquisition unit 201 is used to acquire sensor posture data corresponding to a specified time period, where the sensor posture data is used to represent the posture of the specified sensor device.

[0065] The second acquisition unit 202 is used to acquire a head rotation angle according to the sensor posture data, where the head rotation angle is an angle at which the user's head rotates within the specified time period.

[0066] Optionally, the image processing device further includes: a coordinate system establishing unit.

[0067] The coordinate system establishment unit is used to establish an eye coordinate system before the second acquisition unit 202 executes the acquisition of the head rotation angle according to the sensor posture data. The eye coordinate system is a three-dimensional coordinate system. The eye coordinate system takes the eye midpoint as the origin. The eye midpoint is the midpoint between the two eyes of the user's head. The direction of the first axis of the eye coordinate system is: the eye midpoint points to the direction of one of the eyes of the user's head. The direction of the second axis of the eye coordinate system is the user's line of sight. The third axis of the eye coordinate system is perpendicular to the plane formed by the first axis and the second axis of the eye coordinate system.

[0068] The third acquisition unit 203 is used to acquire the human distance corresponding to the end time of the specified time period, where the human distance refers to the distance between the panoramic camera and the user's body point, wherein the user's body point is: a point on the user's body located on the median sagittal plane of the user's body.

[0069] Optionally, the image processing device further includes: a key point determination unit.

[0070] The key point determination unit is used to: before the third acquisition unit 203 executes the acquisition of the human distance corresponding to the end time of the specified time period, acquire the user image, the user image being: the image obtained by the panoramic camera based on the user at the end time of the specified time period; determine the key points of the human body based on the user image; correspondingly, when the third acquisition unit 203 executes the acquisition of the human distance corresponding to the end time of the specified time period, acquire the human distance corresponding to the end time of the specified time period based on the first feature point position, the first feature point position being the position of the pixel corresponding to the human key point in the user image. That is, this embodiment can accurately calculate the human distance based on the user image.

[0071] The calculation unit 204 is used to calculate θc according to the head rotation angle, the person-object distance and the object-object distance, where θc is the direction angle corresponding to the target object in the inertial coordinate system of the panoramic camera, and the object-object distance is the distance between the target object and the origin of the inertial coordinate system.

[0072] Optionally, the calculation unit 204 is specifically configured to calculate θ according to the head rotation angle, the person-object distance, the object-object distance, and a specified relationship. C , the specified relationship is Among them, the O G P is used to represent the phase-object distance, and the θ r Equal to 90 degrees and θ e The difference between the θ e Used to represent the head rotation angle, D T Used to indicate the distance between the persons.

[0073] Optionally, the image processing device further includes: a distance determination unit.

[0074] The distance determination unit is used to: perform the calculation of θ according to the head rotation angle, the person-object distance, the object-object distance and the specified relationship in the calculation unit 204. C Before, θ is calculated according to the head rotation angle, the distance between the person and the preset value. B , where θ B equal Wherein, B is used to represent the preset value; determining the θ in the target panoramic image B The target panoramic image is a panoramic image obtained by the panoramic camera based on the end time of the specified time period, and the region of interest is used to represent the target object; the object distance is determined based on the region of interest.

[0075] The output unit 205 is used to output the image frame corresponding to the θc.

[0076] It should be noted that for technical details not fully described in this embodiment, reference may be made to the image processing methods provided in the various embodiments in the above-mentioned embodiment 1.

[0077] Embodiment 3

[0078] The panoramic camera in this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on at least one of the processors. The panoramic camera can transmit data with a designated sensor device, and the designated sensor device is a sensor worn on the user's head. When the processor executes the computer program, the steps in any of the above-mentioned image processing method embodiments are implemented.

[0079] When the processor executes the computer program, the steps in the above-mentioned various image processing method embodiments are implemented, for example Figure 1 Alternatively, when the processor executes the computer program, the functions of each unit in the above-mentioned device embodiments are implemented, for example, Figure 2 The functions of units 201 to 205 are shown.

[0080] Those skilled in the art will understand that this embodiment is merely an example of a panoramic camera and does not constitute a limitation on the panoramic camera. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0081] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0082] In some embodiments, the memory may be an internal storage unit of the panoramic camera, such as a hard disk or memory of the panoramic camera. In other embodiments, the memory may also be an external storage device of the panoramic camera, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the panoramic camera. Furthermore, the memory may also include both the internal storage unit of the panoramic camera and an external storage device. The memory is used to store an operating system, an application, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory may also be used to temporarily store data that has been output or is to be output.

[0083] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0084] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0085] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0086] An embodiment of the present application provides a computer program product. When the computer program product runs on a panoramic camera, the panoramic camera can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0087] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the panoramic camera, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0088] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0089] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0090] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0091] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0092] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The image processing method is applied to a panoramic camera, the panoramic camera can transmit data with a designated sensor device, the designated sensor device is a sensor worn on the user's head, and the image processing method includes: Acquire sensor posture data corresponding to a specified time period, wherein the sensor posture data is used to represent the posture of the specified sensor device; Acquire a head rotation angle according to the sensor posture data, where the head rotation angle is an angle at which the user's head rotates within the specified time period; Obtaining the person distance corresponding to the end time of the specified time period, wherein the person distance refers to the distance between the panoramic camera and a user's body point, wherein the user's body point is: a point on the user's body located on the median sagittal plane of the user's body; Calculating θc according to the head rotation angle, the person-object distance and the object-object distance, wherein θc is: the direction angle corresponding to the target object in the inertial coordinate system of the panoramic camera, and the object-object distance is the distance between the target object and the origin of the inertial coordinate system; Output the image picture corresponding to the θc.

2. The image processing method according to claim 1, characterized in that: The calculating θc according to the head rotation angle, the person-object distance and the object-object distance comprises: Calculate θ according to the head rotation angle, the person-object distance, the object-object distance and the specified relationship. C , the specified relationship is Among them, the O C P is used to represent the phase-object distance, and the θ r Equal to 90 degrees and θ e The difference between the θ e Used to represent the head rotation angle, D T Used to indicate the distance between the persons.

3. The image processing method according to claim 2, characterized in that: In the step of calculating θ according to the head rotation angle, the person-object distance, the object-object distance, and a specified relationship C Previously, including: Calculate θ according to the head rotation angle, the distance between the person and the preset value B , the θ B equal Wherein, B is used to represent the preset value; Determine the θ in the target panoramic image B an area of ​​interest within the corresponding image area, the target panoramic image being a panoramic image captured by the panoramic camera at the end moment of the specified time period, and the area of ​​interest being used to represent the target object; A phase-out object distance is determined based on the region of interest.

4. The image processing method according to claim 1, characterized in that: The panoramic camera is fixedly connected to the first end of the selfie stick, and correspondingly, obtaining the distance between people corresponding to the end time of the specified time period includes: Obtaining the pole distance corresponding to the end time of the specified time period, where the pole distance is the distance between the panoramic camera and the second end of the selfie stick; Obtaining a human transmission distance corresponding to the end time of the specified time period, where the human transmission distance is the distance between the user's body point and the specified sensor device; The human distance corresponding to the end point of the specified time period is obtained according to the pole distance and the human distance.

5. The image processing method according to claim 1, characterized in that: Before obtaining the distance between people corresponding to the end time of the specified time period, the method includes: Acquire a user image, where the user image is: an image obtained by the panoramic camera based on the end time of the user being photographed; Determine key points of a human body based on the user image; Correspondingly, obtaining the distance between people corresponding to the end time of the specified time period includes: The distance between people corresponding to the end time of the specified time period is obtained based on the first feature point position, where the first feature point position is the position of the pixel corresponding to the key point of the human body in the user image.

6. The image processing method according to claim 5, characterized in that: The lens of the panoramic camera is a fisheye lens, the user image is a fisheye image, and the obtaining of the distance between people corresponding to the end time of the specified time period based on the position of the first feature point includes: Determine a second feature point position based on the first feature point position and a specified mapping relationship, wherein the specified mapping relationship is used to represent a mapping relationship between the fisheye image and a specified spherical coordinate system, and the second feature point position is used to represent the coordinates of a point corresponding to the human body key point mapped to the specified spherical coordinate system; The distance between people corresponding to the end time of the specified time period is obtained based on the position of the second feature point.

7. The image processing method according to claim 6, characterized in that: The second feature point position includes a left eye feature point position and a right eye feature point position. Correspondingly, obtaining the person distance corresponding to the end time of the specified time period based on the second feature point position includes: Acquire a target field of view angle based on the position of the second feature point, the target field of view angle being the angle between a first straight line segment and a second straight line segment, two endpoints of the first straight line segment including the optical center of the fisheye lens and the left eye of the user's head, and two endpoints of the second straight line segment including the optical center of the fisheye lens and the right eye of the user's head; The distance between people corresponding to the end time of the specified time period is obtained according to the target field of view angle and the user's pupil distance.

8. An image processing device, characterized in that: The image processing device is applied to a panoramic camera, the panoramic camera can transmit data with a designated sensor device, the designated sensor device is a sensor worn on the user's head, and the image processing device includes: A first acquisition unit, used to acquire sensor posture data corresponding to a specified time period, wherein the sensor posture data is used to represent the posture of the specified sensor device; A second acquisition unit is used to acquire a head rotation angle according to the sensor posture data, where the head rotation angle is an angle at which the user's head rotates within the specified time period; a third acquisition unit, configured to acquire a person distance corresponding to an end point of the specified time period, wherein the person distance refers to a distance between the panoramic camera and a user body point, wherein the user body point is a point on the median sagittal plane of the user body; a calculation unit, configured to calculate θc according to the head rotation angle, the person-object distance, and the object-object distance, wherein θc is: a direction angle corresponding to the target object in the inertial coordinate system of the panoramic camera, and the object-object distance is the distance between the target object and the origin of the inertial coordinate system; The output unit is used to output the image corresponding to the θc.

9. A panoramic camera, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The panoramic camera is capable of transmitting data with a designated sensor device, wherein the designated sensor device is a sensor worn on a user's head, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images