Shooting path determination method and related equipment and system
By combining the multi-view shooting technology of the main camera and the auxiliary camera on the underwater robot, adjusting the shooting path and rotation angle, the poor shooting effects caused by the diversity of underwater biological forms and posture changes are solved, and the live broadcast retention rate and user Experience.
Patent Information
- Application Number
- CN202411251443.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-07
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When underwater robots shoot underwater creatures, it is difficult to ensure the shooting effect, especially due to the diverse biological shapes and changes in position and posture, the shooting effect is poor, affecting the live broadcast retention rate.
Using an underwater robot that includes the main camera and multiple auxiliary cameras, the target creature is shot simultaneously through the main camera and the auxiliary camera, adjusting the shooting path to ensure the center position of the creature in the image, and adjusting the rotation angle of the panel and auxiliary camera to achieve Tracking and shooting of underwater creatures.
It improves the shooting effect of underwater creatures, enhances users' immersion and live broadcast retention, and ensures the audience's loyalty and stickiness to live broadcast content.
Smart Images

Figure CN119071626A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of underwater robots, and in particular, relates to a shooting path determination method and related equipment and systems. Background Art
[0002] Underwater photography is widely used in teaching, archaeology, military and other fields; however, it requires high physical fitness and diving ability of the photographer. Using underwater robots to shoot is a good choice. If combined with live broadcast equipment on land, the images taken by underwater robots can be broadcast live, and many ordinary people can see the underwater scenery in real time without diving.
[0003] However, underwater creatures have various forms, and their positions and postures are usually changing. It is very likely that the shooting effect cannot be guaranteed when underwater robots are used to shoot underwater creatures. Summary of the invention
[0004] The purpose of this application is to provide a shooting path determination method and related equipment and system. According to the shooting path determination method provided by this application, the shooting effect of underwater creatures can be improved, thereby improving the live broadcast retention rate.
[0005] In a first aspect, an embodiment of the present application provides a method for determining a shooting path, which is applied to an underwater robot, wherein the underwater robot communicates with a live broadcast device, and the underwater robot includes a visual module, wherein the visual module includes a panel, a main camera, and multiple auxiliary cameras, wherein the main camera is fixedly connected to the panel based on a connecting portion, and each of the auxiliary cameras is connected to the panel and can rotate, wherein the panel is connected to a main body of the underwater robot, and the main camera and the multiple auxiliary cameras face a first direction away from the panel, and the panel can rotate in a three-dimensional space with a center point of the connecting portion as an origin. The method includes: based on a first shooting path, using the main camera and the multiple auxiliary cameras to simultaneously shoot a target organism , obtain a first main image set and multiple first auxiliary image sets, wherein the first shooting path includes the first shooting parameters of the panel, the second shooting parameters of the multiple auxiliary cameras and the first moving path of the underwater robot, the first main image set includes multiple frames of first images taken by the main camera, and the first auxiliary image set includes multiple frames of second images taken by the auxiliary camera; transmit the first main image set and the multiple first auxiliary image sets to the live broadcast device; determine the second shooting path according to the first image, and, based on the second shooting path, use the main camera and the multiple auxiliary cameras to shoot, the second shooting path includes the third shooting parameters of the panel, the fourth shooting parameters of the multiple auxiliary cameras and the second moving path of the underwater robot.
[0006] Optionally, the vision module also includes a panel, the main camera and the panel are fixedly connected based on a connecting part, the center line of the first field of view of the main camera is perpendicular to the first plane, the first plane is parallel to the panel, each auxiliary camera is connected to the panel and can rotate, multiple auxiliary cameras are distributed around the main camera, the panel is connected to the main body of the underwater robot, and the main camera and multiple auxiliary cameras are facing a first direction away from the panel, and the panel can rotate in a three-dimensional space with the center point of the connecting part as the origin; wherein the panel can rotate based on the first central axis, the second central axis and the third central axis, the first central axis is the center line of the first field of view, the second central axis and the third central axis are both located at the first A plane is perpendicular to and intersects with the center line of the first field of view, the second central axis is perpendicular to the third central axis, and the first central axis, the second central axis and the third central axis all pass through the connecting portion; wherein the first shooting parameter or the third shooting parameter indicates: the angle at which the panel rotates along the first central axis and / or the second central axis and / or the third central axis; multiple auxiliary cameras can rotate in the second plane, the second plane is perpendicular to the first plane and intersects with the first connecting line, and the first connecting line is a connecting line between the center point of the main camera and the center point of the auxiliary camera; wherein the second shooting parameter or the fourth shooting parameter indicates: the angle between the center line of the second field of view of the auxiliary camera and the panel in the second plane.
[0007] Optionally, the first image includes a central area of the field of view, the first image displays the target organism, a second shooting path is determined according to the first image, and, based on the second shooting path, shooting is performed using a main camera and multiple auxiliary cameras, including: determining that the first image does not meet a preset condition, the preset condition includes: the center point of the organism displayed in the image is located in the central area of the field of view of the image, or a first proportion of pixels among all pixels corresponding to the organism displayed in the image are located in the central area of the field of view of the image, and the first proportion is greater than a preset proportion; calculating a third shooting parameter according to the first image and a first relative position of the target organism relative to the main camera, the third shooting parameter being different from the first shooting parameter; shooting the target organism using the main camera according to the third shooting parameter to obtain a second main image set, the second main image set including multiple frames of the third image; when the third image meets the preset condition, determining that the first moving path is the same as the second moving path, and calculating a fourth shooting parameter according to a first distance between the target organism and the main camera; while shooting the target organism using the main camera, shooting the target organism using multiple auxiliary cameras according to the fourth shooting parameter to obtain multiple second auxiliary image sets, the second auxiliary image set including multiple frames of the fourth image.
[0008] Optionally, the first image includes a central area of the field of view, the first image displays the target organism, the angle at which the panel is rotated along the second central axis belongs to a first angle interval, and the angle at which the panel is rotated along the third central axis belongs to a second angle interval; determining a second shooting path according to the first image, and, based on the second shooting path, using a main camera and multiple auxiliary cameras to shoot, including: determining that the first image does not meet a preset condition, the preset condition including: the center point of the organism displayed in the image is located in the central area of the field of view of the image, or, a first proportion of all pixels corresponding to the organism displayed in the image are located in the central area of the field of view of the image, and the first proportion is greater than a preset proportion; calculating a fifth shooting parameter according to the first image, and a first relative position of the target organism to the main camera, the fifth shooting parameter indicating: the panel is rotated along the second central axis by a third rotation angle, the panel is rotated along the third The central axis rotates a fourth rotation angle; when the third rotation angle does not belong to the first angle interval, or the fourth rotation angle does not belong to the second angle interval, determine a second movement path according to the first relative position, and the second movement path is different from the first movement path; move along the second movement path; calculate a third shooting parameter according to the first image and the first relative position; use the main camera to shoot the target organism according to the third shooting parameter to obtain a second main image set, and the second main image set includes multiple frames of the third image; when the third image meets the preset conditions, calculate the fourth shooting parameter according to the first distance between the target organism and the main camera; while using the main camera to shoot the target organism, use multiple auxiliary cameras to shoot the target organism according to the fourth shooting parameter to obtain multiple second auxiliary image sets, and the second auxiliary image set includes multiple frames of the fourth image.
[0009] Optionally, the first image includes a central area of the field of view, the first image displays the target organism, a second shooting path is determined according to the first image, and, based on the second shooting path, shooting is performed using a main camera and multiple auxiliary cameras, including: determining that the first image meets a preset condition, the preset condition includes: the center point of the organism displayed in the image is located in the central area of the field of view of the image, or a first proportion of all pixels corresponding to the organism displayed in the image are located in the central area of the field of view of the image, and the first proportion is greater than a preset proportion; when a first distance between the target organism and the main camera is greater than a distance threshold, determining a second moving path according to the first distance, a second relative position of the target organism relative to the underwater robot, and the distance threshold; moving along the second moving path; calculating a third shooting parameter according to the first image and the first relative position of the target organism relative to the main camera; shooting the target organism using the main camera according to the third shooting parameter to obtain a second main image set, the second main image set including multiple frames of the third image; when the third image meets the preset condition, calculating a fourth shooting parameter according to the first distance; while shooting the target organism using the main camera, shooting the target organism using multiple auxiliary cameras according to the fourth shooting parameter to obtain multiple second auxiliary image sets, the second auxiliary image set including multiple frames of the fourth image.
[0010] Optionally, the underwater robot also includes a positioning module, which calculates third shooting parameters based on the first image and the first relative position of the target organism relative to the main camera, including: using the positioning module to determine the second relative position of the target organism relative to the underwater robot, and calibrating the first relative position based on the second relative position; determining the third relative position and fourth relative position of the target organism relative to the main camera in a third plane and a fourth plane respectively according to the first relative position, wherein the third plane is the plane where the first central axis and the second central axis are located, and the fourth plane is the plane where the first central axis and the third central axis are located; determining a fourth rotation angle based on the third relative position; determining the third rotation angle based on the fourth relative position; using the main camera to shoot the target organism according to the third shooting parameters, including: rotating the panel along the second central axis by a third rotation angle, rotating the panel along the third central axis by a fourth rotation angle, and using the main camera to shoot the target organism.
[0011] Optionally, a fourth shooting parameter is calculated based on the first distance between the target organism and the main camera, or the fourth shooting parameter is calculated based on the first distance, including: calculating a fifth rotation angle based on the first distance and the third distance of the first connecting line, the fifth rotation angle is in the second plane, and the fifth rotation angle is the angle between the second field of view center line of the auxiliary camera and the panel; using multiple auxiliary cameras to shoot the target organism according to the fourth shooting parameter, including: rotating multiple auxiliary cameras according to the fifth rotation angle, and using multiple auxiliary cameras to shoot the target organism.
[0012] In a second aspect, an embodiment of the present application provides an underwater robot, comprising a visual module, the visual module comprising a panel, a main camera and multiple auxiliary cameras, wherein the main camera is fixedly connected to the panel based on a connecting portion, the center line of the first field of view of the main camera is perpendicular to the first plane, the first plane is parallel to the panel, each auxiliary camera is connected to the panel and can rotate, the multiple auxiliary cameras are distributed around the main camera, the panel is connected to the body of the underwater robot, and the main camera and the multiple auxiliary cameras are facing a first direction away from the panel, the panel can rotate in a three-dimensional space with the center point of the connecting portion as the origin, and the underwater robot is used to perform the steps of the method provided in the first aspect.
[0013] In a third aspect, an embodiment of the present application provides a shooting path determination system, comprising an underwater robot and a live broadcast device, wherein the underwater robot is connected to the live broadcast device, the underwater robot comprises a visual module, the visual module comprises a panel, a main camera and multiple auxiliary cameras; the underwater robot is used to execute the steps of the method provided in the first aspect; the live broadcast device is used to receive a first main image set and multiple first auxiliary image sets from the underwater robot, and play images to the user according to the first main image set and the multiple first auxiliary image sets.
[0014] Optionally, the live broadcast device is further used to detect an operation in which a user instructs not to watch the target organism, and to send a first indication message to the underwater robot, the first indication message instructing not to shoot the target organism; the underwater robot is further used to receive the first indication message from the live broadcast device, and to determine a third shooting path according to the first indication message, and to shoot according to the third shooting path, wherein the third shooting path is different from the first shooting path; or, the live broadcast device is further used to detect an operation in which a user sets a fourth shooting path, and to send a second indication message to the underwater robot, the second indication message including a fourth shooting path, and the fourth shooting path is different from the first shooting path; the underwater robot is further used to receive the second indication message from the live broadcast device, and to shoot according to the fourth shooting path.
[0015] The fourth aspect of an embodiment of the present application provides a shooting path determination device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program.
[0016] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.
[0017] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: by adjusting the shooting path according to the image taken by the main camera, it is possible to track and shoot the target organism, thereby improving the shooting effect of underwater organisms, giving users a visual experience of swimming underwater to track underwater organisms, and providing a stronger sense of immersion, thereby improving the live broadcast retention rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 A schematic diagram of an example of a visual module provided in an embodiment of the present application;
[0019] Figure 2 A schematic diagram of a rotation method of a panel provided in an embodiment of the present application;
[0020] Figure 3 A schematic diagram of the rotation method of the auxiliary camera provided in an embodiment of the present application;
[0021] Figure 4 A schematic diagram of a shooting path determination method 100 provided in an embodiment of the present application;
[0022] Figure 5 A schematic diagram of acquiring a first image based on a first shooting parameter using a main camera according to an embodiment of the present application;
[0023] Figure 6 A schematic diagram of an example of a method for calculating a fourth rotation angle provided in an embodiment of the present application;
[0024] Figure 7 A schematic diagram of a sixth rotation angle provided in an embodiment of the present application;
[0025] Figure 8 A schematic diagram of an example of a first image provided in an embodiment of the present application;
[0026] Fig. 9 A schematic diagram of an example of calculating a fifth rotation angle provided in an embodiment of the present application;
[0027] Fig.10 A schematic diagram of an example of a fourth image captured by multiple auxiliary cameras provided in an embodiment of the present application;
[0028] Fig.11 A schematic diagram of a shooting path determination system provided in an embodiment of the present application;
[0029] Fig.12 A structural block diagram of a device 1000 provided in an embodiment of the present application is shown;
[0030] Fig.13 A schematic diagram of a shooting path determination device 1100 provided in an embodiment of the present application. DETAILED DESCRIPTION
[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0032] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.
[0033] Underwater photography is widely used in teaching, archaeology, military and other fields; however, it requires high physical fitness and diving ability of the photographer. Using underwater robots to shoot is a good choice. If combined with live broadcast equipment on land, the images taken by underwater robots can be broadcast live, and many ordinary people can see the underwater scenery in real time without diving.
[0034] However, underwater creatures have various shapes, and their positions and postures are usually changing. When underwater robots are used to photograph underwater creatures, the photographing effect may not be guaranteed. Therefore, how to improve the photographing effect has become an urgent problem to be solved.
[0035] In view of this, the present application provides a method for determining a shooting path. The method is applied to an underwater robot including a main camera and multiple auxiliary cameras. The main camera is fixedly connected to the panel and can rotate with the panel. Multiple auxiliary cameras are also connected to the panel, and the multiple auxiliary cameras are located around the main camera and can rotate. The method includes: using the main camera and the auxiliary camera to simultaneously shoot the target organism to obtain multiple image sets, and transmitting the multiple image sets to a live broadcast device to achieve live broadcast. And, according to the image taken by the main camera, adjust the shooting path, wherein the shooting path includes the rotation angle of the panel, the rotation angle of the auxiliary camera and the moving path of the underwater robot.
[0036] The above scheme adjusts the shooting path according to the image taken by the main camera, can track and shoot the target organism, improve the shooting effect of underwater organisms, and give users a visual experience of swimming underwater to track underwater organisms, which has a stronger sense of immersion and improves the live broadcast retention rate.
[0037] It is understandable that the live broadcast retention rate is one of the important indicators to measure the effectiveness of live broadcasts. It reflects the audience's loyalty and stickiness to the live broadcast content. Specifically, the live broadcast retention rate refers to the audience's stay time after entering the live broadcast room for the first time and the subsequent return visit rate during the live broadcast process. This data can intuitively reflect the quality of the live broadcast content.
[0038] It should be noted that the image set in the present application can be multiple frames of discontinuous images or video streams, without limitation.
[0039] Before introducing the shooting path determination method provided in the present application, the underwater robot involved in the present application is first introduced.
[0040] Exemplarily, the underwater robot may include a vision module, a positioning module, a navigation module, a driving module, a lighting module, and the like.
[0041] The visual module includes a panel, a main camera and a plurality of auxiliary cameras. The panel is connected to the main body of the underwater robot based on a connection part 1, the main camera is fixedly connected to the panel, and each auxiliary camera is connected to the panel based on a connection part 2 and can rotate. The main camera and the plurality of auxiliary cameras face a first direction away from the panel. The plurality of auxiliary cameras are distributed around the main camera.
[0042] The positioning module is used to determine the relative position of the target organism with respect to the underwater robot, or to determine the distance between the target organism and the underwater robot.
[0043] Among them, the navigation module is used to automatically plan a path to a target location in the map reference system after determining its own position in the map reference system and reach the target location point along the path.
[0044] The driving module is used to drive other modules to operate. For example, the driving module includes a stepper motor, and the driving module is electrically connected to the panel or the auxiliary camera; the driving module controls the rotation angle of the panel or the auxiliary camera by sending a pulse signal. For another example, the driving module includes a servo motor and a servo driver, and the driving module is electrically connected to the panel or the auxiliary camera; the driving module is used to control the rotation angle of the panel or the auxiliary camera.
[0045] Among them, the lighting module is used to provide lighting function for the visual module to improve the shooting effect of the visual module.
[0046] For the convenience of explanation, this application mainly provides a detailed description of the visual module, and does not limit the implementation methods of other modules included in the underwater robot, nor does it limit the connection method between the visual module and the underwater robot body.
[0047] Figure 1 A schematic diagram of an example of a visual module provided in an embodiment of the present application.
[0048] like Figure 1 As shown in (a), the A surface of the display panel faces the first direction. The main camera is fixedly connected to the panel and is located at the center of the panel. The auxiliary cameras 1 to 4 are respectively located around the main camera.
[0049] like Figure 1 As shown in (b) in the figure, side B of the display panel is connected to the main body of the underwater robot. Figure 1 In (b), the multiple cameras located on the A surface are indicated by dotted lines. Exemplarily, part or all of the panel is connected to the underwater robot body. For example, a connector is provided at a position of the B surface of the panel corresponding to the main camera, and the panel is connected to the underwater robot body based on the connector.
[0050] It should be noted that Figure 1 In the figure, the shape or size of the panel is only an example, the position of the main camera is only an example, and the number or position of the auxiliary cameras is only an example, and there is no limitation on these.
[0051] For the visual module, possible rotations of the panel are given. Figure 2 A schematic diagram of the rotation method of the panel provided in an embodiment of the present application.
[0052] like Figure 2 As shown, the A side of the display panel is shown. The panel can rotate based on the first central axis, the second central axis and the third central axis. The double-headed arrows corresponding to the first central axis, the second central axis and the third central axis indicate the direction in which the panel can rotate.
[0053] Among them, the first central axis is the center line of the first field of view, the second central axis and the third central axis are both located in the first plane and are perpendicular to and intersect with the first field of view center line, the second central axis is perpendicular to the third central axis, and the first central axis, the second central axis and the third central axis all pass through the connection between the main camera and the panel.
[0054] It should be noted that Figure 2 The positions and number of auxiliary cameras are for example only, and the shape or size of the panel are for example only and are not limited.
[0055] For the vision module, possible rotation methods of the auxiliary camera are given. Figure 3 A schematic diagram of the rotation method of the auxiliary camera provided in an embodiment of the present application.
[0056] like Figure 3As shown, multiple auxiliary cameras can rotate in a second plane, and the second plane is perpendicular to the first plane and intersects with the first connecting line. Specifically, the intersection line of the first plane and the second plane is represented by a dotted line, and the first connecting line is included in the intersection line and represented by a solid line. The first connecting line is a connecting line between the center point of the main camera and the center point of the auxiliary camera.
[0057] For example, the connection portion 2 may be a hinge, such as a single-axis hinge, which is used to limit the auxiliary camera from rotating within the second plane.
[0058] The double-headed arrow corresponding to the auxiliary camera 1 represents the rotatable direction of the auxiliary camera 1 in the second plane.
[0059] Figure 4 A schematic diagram of a shooting path determination method 100 provided in an embodiment of the present application.
[0060] S101, based on a first shooting path, using a main camera and multiple auxiliary cameras to simultaneously shoot a target organism to obtain a first main image set and multiple first auxiliary image sets.
[0061] The first shooting path includes the first shooting parameters of the panel, the second shooting parameters of the plurality of auxiliary cameras and the first moving path of the underwater robot.
[0062] For example, the first shooting parameter indicates: the angle at which the panel rotates along the first central axis and / or the second central axis and / or the third central axis. The second shooting parameter indicates: the angle between the center line of the second field of view of the auxiliary camera and the panel in the second plane. The first moving path is the moving path referenced by the underwater robot when acquiring the first main image set and multiple first auxiliary image sets.
[0063] The first main image set includes multiple frames of first images taken by the main camera, and the first auxiliary image set includes multiple frames of second images taken by the auxiliary camera.
[0064] S102: Transmit a first main image set and a plurality of first auxiliary image sets to a live broadcast device.
[0065] In the present application, the underwater robot communicates with the live broadcast device. In one possible implementation, the underwater robot communicates directly with the live broadcast device. In another possible implementation, the underwater robot communicates with the live broadcast device through an onshore controller.
[0066] S103: determining a second shooting path according to the first image, and performing shooting using a main camera and a plurality of auxiliary cameras based on the second shooting path.
[0067] The second shooting path includes the third shooting parameters of the panel, the fourth shooting parameters of the plurality of auxiliary cameras and the second moving path of the underwater robot.
[0068] For example, the third shooting parameter indicates: the angle at which the panel rotates along the first central axis and / or the second central axis and / or the third central axis. The fourth shooting parameter indicates: the angle between the center line of the second field of view of the auxiliary camera and the panel in the second plane. The first moving path is the moving path that the underwater robot refers to when continuing to shoot based on the main camera and multiple auxiliary cameras.
[0069] Exemplarily, determining the second shooting path according to the first image can be understood as determining whether to change the first shooting path according to the first image. If the first shooting path is to be changed, the second shooting path is different from the first shooting path, otherwise they are the same.
[0070] Whether to change the first shooting path may be determined according to whether one or more preset conditions are met. The following first introduces several preset conditions involved in the present application, and then further introduces S103.
[0071] For example, preset condition 1 includes: the center point of the organism displayed in the image is located in the central area of the field of view of the image, or a first proportion of all pixels corresponding to the organism displayed in the image are located in the central area of the field of view of the image, and the first proportion is greater than the preset proportion.
[0072] For another example, preset condition 2 includes: the rotation angle of the panel is within a preset angle range.
[0073] For another example, preset condition 3 includes: a first distance between the target organism and the main camera is less than or equal to a distance threshold.
[0074] Several possible examples of S103 are given below.
[0075] Example 1: The first image includes a central area of the field of view, and the first image displays a target organism.
[0076] Step A1, determining that the first image does not meet preset condition 1.
[0077] Step A2: calculating a third shooting parameter according to the first image and a first relative position of the target organism relative to the main camera.
[0078] The third shooting parameter is different from the first shooting parameter.
[0079] Specifically, how to calculate the third shooting parameter will be described below in conjunction with the accompanying drawings.
[0080] Step A3: photograph the target organism using the main camera according to the third photographing parameter to obtain a second main image set.
[0081] The second main image set includes multiple frames of third images.
[0082] Step A4, when the third image meets the preset condition, determining that the first moving path is the same as the second moving path, and calculating a fourth shooting parameter according to the first distance between the target organism and the main camera.
[0083] Specifically, how to calculate the fourth shooting parameter will be described below in conjunction with the accompanying drawings.
[0084] Step A5: while photographing the target organism with the main camera, photograph the target organism with the plurality of auxiliary cameras according to the fourth photographing parameter to obtain a plurality of second auxiliary image sets.
[0085] The second auxiliary image set includes multiple frames of fourth images.
[0086] According to Example 1, when preset condition 1 is not met, preset condition 1 can be met by adjusting the rotation angle of the panel. When the target organism is not located in the central area of the image, the third shooting parameter of the main camera is calculated (that is, the rotation angle of the panel is recalculated). In the image obtained by the main camera based on the third shooting parameter, the target organism can be located in the central area, which means that the preset condition 1 can be met without changing the moving path of the underwater robot. Therefore, the first moving path is the same as the second moving path. Since the panel is connected to the auxiliary camera, after the rotation angle of the panel changes, the viewing angle of the picture taken by the auxiliary camera also changes, so as to ensure that the auxiliary camera can capture the target organism, so it is necessary to calculate the fourth shooting parameter of the auxiliary camera (that is, recalculate the rotation angle of the auxiliary camera). Thus, without changing the moving path, adjusting the rotation angle of the panel and adjusting the rotation angle of the auxiliary camera accordingly can make the target organism located in the central area of the field of view of the picture captured by the main camera, and the auxiliary camera can capture the target organism at the same time.
[0087] Example 2, the first image includes a central area of the field of view, and the first image displays the target organism. Preset condition 2 specifically includes: the angle at which the panel rotates along the second central axis belongs to the first angle interval, and the angle at which the panel rotates along the third central axis belongs to the second angle interval. Exemplarily, the first angle interval and the second angle interval can be determined according to the connection method between the panel and the underwater robot body.
[0088] Step B1, determining that the first image does not satisfy preset condition 1.
[0089] Step B2, calculating a fifth shooting parameter according to the first image and a first relative position between the target organism and the main camera.
[0090] The fifth shooting parameter indicates that: the panel rotates along the second central axis by a third rotation angle, and the panel rotates along the third central axis by a fourth rotation angle.
[0091] Step B3: when the preset condition 2 is not met, determine the second moving path according to the first relative position.
[0092] Wherein, not satisfying the preset condition 2 may include: the third rotation angle does not belong to the first angle interval, and / or the fourth rotation angle does not belong to the second angle interval.
[0093] The second moving path is different from the first moving path.
[0094] Exemplarily, the underwater robot may randomly determine a path different from the first moving path as the second moving path. Alternatively, assuming that the panel theoretically (but not actually) rotates to face the second direction according to the third rotation angle and the fourth rotation angle, the second moving path is determined according to the second direction. Alternatively, the second moving path is determined according to the position of the underwater creature relative to the underwater robot.
[0095] Step B4, moving along the second moving path.
[0096] Step B5, calculating a third shooting parameter according to the first image and the first relative position.
[0097] Specifically, how to calculate the third shooting parameter will be described below in conjunction with the accompanying drawings.
[0098] Step B6: photograph the target organism using the main camera according to the third photographing parameter to obtain a second main image set.
[0099] The second main image set includes multiple frames of third images.
[0100] Step B7: when the third image satisfies the preset condition 1, a fourth shooting parameter is calculated according to the first distance between the target organism and the main camera.
[0101] Specifically, how to calculate the fourth shooting parameter will be described below in conjunction with the accompanying drawings.
[0102] Step B8, while using the main camera to shoot the target organism, use multiple auxiliary cameras to shoot the target organism according to the fourth shooting parameter to obtain multiple second auxiliary image sets.
[0103] The second auxiliary image set includes multiple frames of fourth images.
[0104] According to Example 2, when preset condition 1 is not met, preset condition 1 cannot be met by adjusting the rotation angle of the panel; preset condition 1 can be met by adjusting the moving path of the underwater robot and adjusting the rotation angle of the panel accordingly. When the target organism is not located in the central area of the image, the fifth shooting parameter of the main camera is calculated (that is, the rotation angle of the panel is recalculated). After calculation, the fifth shooting parameter includes the third rotation angle and the fourth rotation angle, which does not meet preset condition 2. It means that changing the rotation angle of the panel cannot meet preset condition 1, so it is necessary to change the moving path of the underwater robot so as to meet preset condition 1. Therefore, the second moving path is calculated according to the first relative position of the target organism and the main camera. After the underwater robot moves according to the second moving path, the third shooting parameter of the main camera is calculated, and the third image is obtained based on the third shooting parameter using the main camera. When the third image meets preset condition 1, the fourth shooting parameter of the auxiliary camera is calculated accordingly (that is, the rotation angle of the auxiliary camera is recalculated). Since the panel is connected to the auxiliary camera, after the rotation angle of the panel changes, the viewing angle of the picture taken by the auxiliary camera also changes, so as to ensure that the auxiliary camera can capture the target organism. Thus, by changing the moving path, adjusting the rotation angle of the panel and correspondingly adjusting the rotation angle of the auxiliary camera, the target organism can be located in the central area of the field of view of the main camera shooting picture, and the auxiliary camera can shoot the target organism at the same time.
[0105] Example 3: The first image includes a central area of the field of view, and the first image displays a target organism.
[0106] Step C1, determining whether the first image satisfies preset condition 1.
[0107] Step C2, determine that preset condition 3 is not met (that is, the first distance between the target organism and the main camera is greater than the distance threshold), and determine the second movement path according to the first distance, the second relative position of the target organism relative to the underwater robot and the distance threshold.
[0108] It can be understood that the purpose of determining the second moving path is that after the underwater robot moves along the second moving path, the distance between the main camera and the target organism can be less than or equal to the distance threshold. In a possible implementation, the moving path of the main camera is determined according to the first distance and the distance threshold, and the second moving path of the underwater robot is calibrated according to the coordinate system conversion between the main camera and the underwater robot.
[0109] Step C3, moving along the second moving path.
[0110] After step C3, preset condition 3 is satisfied.
[0111] Step C4, calculating a third shooting parameter according to the first image and a first relative position of the target organism relative to the main camera.
[0112] Specifically, how to calculate the third shooting parameter will be described below in conjunction with the accompanying drawings.
[0113] Step C5: photograph the target organism using the main camera according to the third photographing parameter to obtain a second main image set.
[0114] The second main image set includes multiple frames of third images.
[0115] Step C6: when the third image satisfies the preset condition 1, a fourth shooting parameter is calculated according to the first distance.
[0116] Alternatively, when the third image does not satisfy preset condition 1, reference may be made to example 1 or example 2 above.
[0117] Specifically, how to calculate the fourth shooting parameter will be described below in conjunction with the accompanying drawings.
[0118] Step C7, while using the main camera to shoot the target organism, use multiple auxiliary cameras to shoot the target organism according to the fourth shooting parameter to obtain multiple second auxiliary image sets.
[0119] The second auxiliary image set includes multiple frames of fourth images.
[0120] According to Example 3, when preset condition 1 is met but preset condition 3 is not met, the movement path of the underwater robot is adjusted to meet preset condition 3, and then the rotation angle of the panel is adjusted to meet preset condition 1. Then the rotation angle of the auxiliary camera is adjusted accordingly to achieve that the target organism is located in the central area of the field of view of the main camera shooting picture, and the auxiliary camera can simultaneously shoot the target organism.
[0121] It is understandable that the present application (such as Examples 1 to 3) is committed to satisfying preset condition 1, which can achieve multiple beneficial effects. On the one hand, since users generally pay more attention to the center of the screen, if the target organism is located in the central area of the field of view, the live broadcast experience is good, which can improve the live broadcast retention rate. By controlling the image captured by the main camera to meet preset condition 1, and adjusting the angle of the auxiliary camera, it is beneficial for the image captured by the auxiliary camera to also meet preset condition 1 (for details, please refer to the introduction to the calculation of the fourth shooting parameter below), thereby ensuring that the image transmitted to the live broadcast device meets preset condition 1 as much as possible to improve the live broadcast retention rate.
[0122] Moreover, compared to shooting with a single camera, after the main camera captures the image of the target creature, the auxiliary camera can be rotated to simultaneously obtain images from multiple perspectives. For underwater creatures whose positions and postures are often changing, shooting images from multiple perspectives can capture more valuable images in the same amount of time, improve shooting effects, and also improve live broadcast retention rates. For example, taking fish as an example, including features such as fins, heads, and tails, compared to single-perspective shooting, multi-perspective shooting can increase the possibility of capturing more features of underwater creatures, and also allow users to view underwater creatures more comprehensively, increasing the attractiveness of live broadcasts to users, thereby increasing live broadcast retention rates.
[0123] Combine the following Figures 5 to 8 An example of calculating the third shooting parameter is described.
[0124] Figure 5 A schematic diagram of acquiring a first image based on a first shooting parameter using a main camera provided in an embodiment of the present application.
[0125] like Figure 5 As shown, a first image is acquired by using a main camera based on a first shooting parameter. In the first image, the target organism is obviously not in the central area of the field of view.
[0126] The plane where the first central axis and the second central axis are located is called the third plane, and the plane where the first central axis and the third central axis are located is called the fourth plane. The third shooting parameter includes a third rotation angle and a fourth rotation angle, wherein the fourth rotation angle is the angle at which the panel is rotated based on the third central axis, that is, the angle at which the panel is rotated in the third plane when viewed from the viewing angle 1 along the third central axis; the third rotation angle is the angle at which the panel is rotated based on the second central axis, that is, the angle at which the panel is rotated in the fourth plane when viewed from the viewing angle 2 along the second central axis.
[0127] The following takes the angle of view 1 as an example to introduce the calculation method of the fourth rotation angle. It can be understood that the calculation method of the third rotation angle is similar to it and will not be described in detail. Therefore, according to the calculation method of the fourth rotation angle and the third rotation angle, the calculation method of the third shooting parameter can be determined.
[0128] Figure 6 A schematic diagram of an example of a method for calculating the fourth rotation angle provided in an embodiment of the present application.
[0129] like Figure 6 As shown in (a), panel #1 (represented by solid lines) represents the panel before rotation, and the first central axis #1 and the second central axis #1 are the central axes of panel #1; panel #2 (represented by dotted lines) represents the panel after rotation, and the first central axis #2 and the second central axis #2 are the central axes of panel #2.
[0130] like Figure 6As shown in (b) in Figure 6 Based on (a) in FIG. 1 , the fourth rotation angle is explained. The fourth rotation angle is ∠β in the figure, which can be understood as the angle of rotation of panel #2 compared with panel #1 in the third plane, and can also be understood as the angle between the first central axis #2 and the first central axis #1.
[0131] It can be understood that after the center line of the first field of view intersects with the target organism, the target organism in the first image can be located in the center area of the field of view. Therefore, it can be considered that after the rotation ∠β, the third image obtained by the main camera can meet the preset condition 1.
[0132] Figure 6 The gray triangle in (b) belongs to the third plane. ∠β can be calculated based on the lengths of any two sides of the gray triangle.
[0133] Exemplarily, the positioning module can be used to determine the second relative position of the target organism relative to the underwater robot, thereby calibrating the first relative position of the target organism relative to the main camera based on the second relative position, and determining the third relative position of the target organism relative to the main camera in a third plane.
[0134] For example, the third relative position can be expressed in the form of coordinates. Taking the intersection of the first central axis #1 and the second central axis #1 as the origin, the coordinates are (0, 0), the first central axis #1 as the y-axis, and the second central axis #1 as the x-axis, the coordinates of the underwater organism are (x1, y1). Among them, the coordinates of the underwater organism can be the coordinates of any point of the underwater organism. The gray triangle is a right triangle, the length of the long right-angled side is y1, and the length of the short right-angled side is x1. Taking the determination of ∠β based on two right-angled sides as an example, tanβ=x1 / y1, ∠β=arctan(x1 / y1).
[0135] Combination Figure 6 , step A2, step B5 or step C4 above may specifically include: using the positioning module to determine the second relative position of the target organism relative to the underwater robot, and calibrating the first relative position according to the second relative position; determining the third relative position and the fourth relative position of the target organism relative to the main camera in the third plane and the fourth plane respectively according to the first relative position, wherein the third plane is the plane where the first central axis and the second central axis are located, and the fourth plane is the plane where the first central axis and the third central axis are located; determining the fourth rotation angle according to the third relative position; determining the third rotation angle according to the fourth relative position.
[0136] Accordingly, the above step A3, step B6 or step C5 may specifically include: rotating the panel along the second central axis by a third rotation angle, rotating the panel along the third central axis by a fourth rotation angle, and photographing the target organism using the main camera.
[0137] Optionally, the calculation method of the third shooting parameter also includes a calculation method of a sixth rotation angle based on the first central axis.
[0138] Figure 7 A schematic diagram of the sixth rotation angle provided in an embodiment of the present application.
[0139] like Figure 7 As shown, the panel rotates by ∠γ based on the first central axis, which is the sixth rotation angle.
[0140] Exemplarily, the sixth rotation angle may be determined according to the position of the target organism in the first image. For example, in a scene where the target organism is almost stationary, if the target organism has an angle offset compared to the viewing angle of the first image, the sixth rotation angle may be considered to improve the viewing comfort of the user.
[0141] Figure 8 A schematic diagram of an example of a first image provided in an embodiment of the present application.
[0142] like Figure 8 As shown in (a) in FIG. 1 , the first image shows almost still underwater organisms (such as corals), and the intersection of two mutually perpendicular dotted lines is the center point of the first image.
[0143] Since the live broadcast device generally broadcasts live to the user with a square screen, the first image is transmitted to the live broadcast device and played to the user, and the picture viewed by the user is similar to the first image. From the user's viewing angle, the coral is tilted in the live broadcast picture, which does not conform to the user's normal viewing habits. Many users may even need to turn their heads in the direction of the tilted coral, or users who use mobile terminals to watch the live broadcast need to turn the mobile terminal device in the direction of the tilted coral to watch, which is very inconvenient.
[0144] Therefore, by determining the sixth rotation angle according to the position of the target organism in the first image, and rotating the panel according to the sixth rotation angle, the position of the target organism in the first image can be made consistent with the user's perspective, so that the picture played by the live broadcast device can be consistent with the user's perspective, thereby improving the user experience.
[0145] like Figure 8 As shown in (b) of FIG. 1 , it is the first image after the panel is rotated according to the sixth rotation angle. Figure 8 The first image shown in (a) can significantly improve the viewing experience of the live broadcast picture and increase the live broadcast retention rate.
[0146] It can be understood that the third rotation angle, the fourth rotation angle and the sixth rotation angle are the angles at which the panel is to be rotated relative to the position before rotation.
[0147] Combine the following Figures 9 and 10 An example of calculating the fourth shooting parameter is described.
[0148] Exemplarily, the above step A4, step B7 or step C6 may specifically include: calculating a fifth rotation angle according to the first distance and the third distance of the first connecting line, the fifth rotation angle being in the second plane, and the fifth rotation angle being the angle between the center line of the second field of view of the auxiliary camera and the panel, rotating multiple auxiliary cameras according to the fifth rotation angle, and using the multiple auxiliary cameras to photograph the target organism.
[0149] Accordingly, the above step A5, step B8 or step C7 may specifically include: rotating the multiple auxiliary cameras according to the fifth rotation angle, and photographing the target organism using the multiple auxiliary cameras.
[0150] Fig. 9 A schematic diagram of an example of calculating the fifth rotation angle provided in an embodiment of the present application.
[0151] like Fig. 9 As shown in (a), taking the first plane facing the first direction as an example, and taking auxiliary camera 1 among multiple auxiliary cameras as an example, a schematic diagram for calculating the fifth rotation angle is given. The first distance is the distance from the center point of the main camera to the target organism (for example, the distance of any point), the first connecting line is located in the first plane, and the length of the first connecting line is the third distance. Since the center line of the first field of view is perpendicular to the first plane, the first connecting line belongs to the first plane, so the angle between the center line of the first field of view and the first connecting line is a right angle. Thus, the center line of the first field of view, the first connecting line, and the center line of the second field of view form a right triangle. ∠α is the fifth rotation angle. tanα=first distance / third distance, ∠α=arctan(first distance / third distance).
[0152] Understandable, Fig. 9 The intersection of the first distance and the third distance shown in (a) is only an example, and the intersection can be within the range where the target organism is located.
[0153] For example, Fig. 9 As shown in (b) in the figure, the intersection point can be the center point of the underwater creature. For other contents, please refer to Fig. 9 The relevant description of (a) in .
[0154] It should be noted that in a specific implementation, multiple auxiliary cameras correspond to multiple second planes and multiple fifth rotation angles. If the distances between the multiple auxiliary cameras and the main camera are equal, the fifth rotation angle of one of the auxiliary cameras can be calculated, and the other auxiliary cameras can be rotated to the same fifth rotation angle in the corresponding second plane.
[0155] Fig.10A schematic diagram of an example of a fourth image captured by multiple auxiliary cameras provided in an embodiment of the present application.
[0156] like Fig.10 As shown in (a) of FIG. 1 , the image currently captured by the main camera is taken as the third image. The third image satisfies the preset condition 1.
[0157] like Fig.10 As shown in (b) of FIG. 1 , the fourth images captured by the auxiliary cameras 1 to 4 correspond to the third images respectively. Fig.10 It can be seen that by shooting the target organism with multiple auxiliary cameras, images of the target organism from different perspectives can be obtained. Fig.10 In the figure, the number and positions of the auxiliary cameras are only examples, and the four fourth images are only examples and are not limited.
[0158] The above scheme provides images of the target organism from different perspectives to the live broadcast device so that the live broadcast device can provide valuable images to the user, thereby improving the live broadcast retention rate.
[0159] It can be understood that the fifth rotation angle is the angle that the auxiliary camera must meet after the auxiliary camera is rotated. Specifically, the degree of rotation of the auxiliary camera is determined according to the angle 1 between the center line of the second field of view and the panel before rotation and the fifth rotation angle. The angle difference between the angle 1 and the fifth rotation angle is the angle that the auxiliary camera must rotate.
[0160] Fig.11 A schematic diagram of a shooting path determination system provided in an embodiment of the present application.
[0161] like Fig.11 As shown, the system includes an underwater robot, a controller and a live broadcast device. The controller can be installed inside the underwater robot, or independently located underwater, or independently located above water, or installed with the live broadcast device. The controller serves as a media device for communication between the underwater robot and the live broadcast device.
[0162] Underwater robots, in addition to Fig.11 The visual module and positioning module shown in the figure can also refer to the above description, which will not be repeated here. The underwater robot is used to perform the steps of the method described above.
[0163] The live broadcast device, corresponding to S102, is used to receive a main image set and multiple auxiliary image sets from the underwater robot, and play pictures to the user according to the main image set and the multiple auxiliary image sets.
[0164] Optionally, in addition to the solution of changing the current shooting path according to the first image described above, the present application may also include: a solution of changing the shooting path according to the user's operation on the live broadcast device. The content of changing the shooting path can refer to the above, the difference is that the basis for triggering the change of the shooting path is different.
[0165] Example A, step D1, the live broadcast device detects operation 1 in which the user indicates not to watch the target creature.
[0166] For example, the user may not be interested in the target organism and choose not to view the current target organism.
[0167] Step D2: the live broadcast device sends a first indication message to the underwater robot, and correspondingly, the underwater robot receives the first indication message from the live broadcast device.
[0168] The first instruction information indicates not to photograph the target organism.
[0169] Step D3: the underwater robot determines a third shooting path according to the first instruction information, and shoots according to the third shooting path.
[0170] The third shooting path is different from the first shooting path.
[0171] Example A-1, the underwater robot can change the first moving path to find and photograph new target organisms; and then adjust the first shooting parameter and the second shooting parameter accordingly.
[0172] Example A-2, first change the first shooting parameter to look for a new target creature, if it can be found, continue to adjust the second shooting parameter, and do not need to adjust the first moving path.
[0173] Example A-3, first change the first shooting parameter to look for a new target creature. If it cannot be found, adjust the first moving path, and then refer to Example A-1.
[0174] The above solution changes the shooting path and searches for new target creatures according to the user's operation, which can increase the user's sense of interaction when watching the live broadcast and give the user an immersive experience of freely swimming on the seabed; it can also better meet the user's viewing needs, allowing the user to judge whether the live broadcast picture is valuable. Thus, the retention rate of live broadcast is improved.
[0175] Example B, step E1, the live broadcast device detects the user's operation of setting the fourth shooting path.
[0176] Exemplarily, the live broadcast device provides the user with functions for setting the shooting path, such as direction controls and rotation controls, etc. The user can control the movement path of the underwater robot based on the direction controls; and can control the rotation angle of the control panel in the three-dimensional control and the rotation angle of the auxiliary camera in the second plane based on the rotation controls.
[0177] Step E2: the live broadcast device sends a second instruction message to the underwater robot, and correspondingly, the underwater robot receives the second instruction message from the live broadcast device.
[0178] The second indication information includes a fourth shooting path, which is different from the first shooting path.
[0179] Step E3: The underwater robot performs shooting according to the fourth shooting path.
[0180] The above solution, which shoots underwater images according to the shooting path provided by the user, can increase the user's sense of interaction when watching the live broadcast, giving the user an immersive experience of freely swimming on the seabed; and can better meet the user's viewing needs, allowing the user to judge whether the live broadcast image is valuable. Thus, the retention rate of live broadcast is improved.
[0181] Optionally, in the present application, before determining whether a certain image satisfies the preset condition 1, it may also include: determining whether there is a target organism in the image. For example, the image may be the first image or the third image mentioned above. For the convenience of explanation, the first image or the third image is referred to as the initial image below.
[0182] Exemplarily, the initial image is input into the first neural network model to obtain the biological detection result.
[0183] The biological detection result indicates whether the target biological being exists in the initial image.
[0184] For example, the first neural network model is trained based on the first training data set. The first training data set includes first input data and first target data. The first input data includes multiple frames of first sample images, and the multiple frames of first sample images respectively display different sample underwater organisms, and the sample underwater organisms display at least one of the sample morphological features. The first target data includes multiple groups of first sample coordinate data, and the multiple frames of first sample images correspond to the multiple groups of first sample coordinate data one by one, and each group of first sample coordinate data is the pixel coordinate of the sample morphological feature in each frame of the first sample image.
[0185] Specific examples are given below for sample underwater organisms and sample morphological characteristics.
[0186] Example 1: The sample underwater organism is a fish-shaped underwater organism, and the sample morphological characteristics include: fins, tail, trunk and head.
[0187] Among them, fish-shaped underwater creatures can include some fish and some underwater mammals. For example, fish include freshwater fish and marine fish. Freshwater fish include black carp, grass carp, silver carp, bighead carp, bream and crucian carp, etc.; marine fish include hairtail, tuna, large yellow croaker, small yellow croaker, etc. Underwater mammals include dolphins and whales, etc. Their bodies at least include: fins, tail, trunk and head.
[0188] Example 2: The sample underwater organism is an underwater organism with tentacles, and the sample morphological characteristics include: head and tentacles.
[0189] Among them, underwater creatures with tentacles can include some underwater coelenterates, such as sea anemones, jellyfish, coral polyps and sea cucumbers, etc.; they can also include some underwater mollusks, such as squids and octopuses, etc. Their bodies include at least two parts, a head and a tentacle part.
[0190] Example 3: The sample underwater organism is an underwater organism with appendages, and the sample morphological characteristics include: cephalothorax, abdomen and appendages.
[0191] Among them, underwater creatures with appendages can include some crustaceans, such as shrimps and crabs, whose bodies are divided into two parts: cephalothorax and abdomen, and they also have powerful appendages for walking or swimming.
[0192] Alternatively, the underwater organisms and morphological characteristics of the samples may be realized in other ways without limitation.
[0193] Taking a frame of the first sample image as an example, assuming that the first sample image 1 shows a crucian carp, and two samples of the crucian carp can be seen in the first sample image, morphological feature 1 (tail) and morphological feature 2 (fin). The first target data 1 corresponds to the first sample image 1, and the first target data 1 includes first sample coordinate data 1, and the first sample coordinate data 1 includes a set of coordinate data. Among them, the first sample coordinate data 1 includes the pixel coordinates of the morphological feature 1 in the first sample image 1, and the pixel coordinates of the morphological feature 2 in the first sample image 1.
[0194] The first input data includes other first sample images, and the first target data includes other first sample coordinate data, which can also refer to the description of the first sample image 1 and the first sample coordinate data 1.
[0195] Recognition result 1: when the biological detection result includes the first coordinate data corresponding to the initial image, the biological detection result indicates that the target biological exists in the initial image. Then, S103 is executed.
[0196] Alternatively, in recognition result 2, when the biological detection result does not include coordinate data, the biological detection result indicates that the target organism does not exist in the initial image. In this case, the underwater robot can change the shooting path, continue to use the main camera to shoot a new initial image, and refer to the above method to use the first neural network for recognition until the recognition result 1 is obtained, and then perform the subsequent operation of determining whether the initial image meets the preset condition 1.
[0197] In the above scheme, after the main camera captures underwater creatures, the main camera and the auxiliary camera are used to capture the target creatures at the same time. Compared with directly controlling the main camera and the auxiliary camera to capture together without identifying whether the target creatures exist, this scheme can improve the success rate of capturing valuable images and save power consumption.
[0198] Optionally, in the present application, S102 can be replaced by: screening out a second image set from a first main image set and a plurality of first auxiliary image sets, and transmitting the second image set to a live broadcast device. For the sake of convenience, the first main image set and a plurality of first auxiliary image sets are referred to as a first image set. Among them, the degree of recognition of the target organism in the second image set is higher than that of other image sets, and the other image sets include: images in the first image set other than the second image set. Among them, the degree of recognition can be understood as the degree to which the target organism can be seen by the user with obvious morphological features. The higher the degree of recognition, the more obvious the morphological features of the target organism are for the user, and the more the user can understand the target organism.
[0199] A possible implementation manner of screening the second image set is given below.
[0200] Exemplarily, the parameters used to characterize the degree of recognition of the target organism include at least one of the following: the number of types of morphological features of the target organism, the total number of morphological features of the target organism, or the area ratio of the morphological features of the target organism in the image showing the target organism. Among them, the area ratio of morphological features is the ratio of the number of pixels used to display the morphological features to the total number of pixels in the entire image. Optionally, when the parameters such as the size and resolution of the photos taken by the main camera and the auxiliary camera are consistent, the area ratio of the morphological features of the target organism in the image showing the target organism can also be replaced by the area ratio of the morphological features of the target organism in the image showing the target organism.
[0201] When determining the degree of recognition of an image, the magnitude of the above parameters can be compared. The larger the value of the above parameters, the higher the degree of recognition. It can be understood that the larger the number of types of morphological features of the target organism and the total number of morphological features of the target organism, the more obvious the morphological features of the target organism displayed in the image; the larger the area proportion of the morphological features of the target organism in the image showing the target organism, the higher the possibility that the morphological features of the target organism can be seen or seen clearly by the user.
[0202] Optionally, the above parameters preferably include the number of types of morphological features of the target organism. Secondly, it is preferred to include the total number of morphological features of the target organism. Secondly, it is preferred to include the area ratio of the morphological features of the target organism in the image showing the target organism. It can be understood that in two images, when the total number of morphological features of the target organism is the same, or when the area ratio of the morphological features of the target organism in the image showing the target organism is the same, the more types of morphological features of the target organism, the higher the degree of recognition is generally. When the area ratio of the morphological features of the target organism in the image showing the target organism is the same, the more pixels the morphological features of the target organism occupy, the higher the degree of recognition is generally.
[0203] In a possible implementation, after obtaining the parameters for characterizing the degree of recognition of the target organism, the degree of recognition can be obtained by weighted calculation. For example, the number of types of morphological features of the target organism corresponds to a first weight, the total number of morphological features of the target organism corresponds to a second weight, and the area ratio of the morphological features of the target organism in the image showing the target organism corresponds to a third weight. The first weight is greater than the second weight, which is greater than the third weight.
[0204] For example, the target organism is a squid, and the parameters of the degree of recognition include the number of types of morphological features of the target organism and the total number of morphological features of the target organism. Image 1 shows the head and 1 tentacle of the squid, and image 2 shows the head and 3 tentacles of the squid. Taking image 1 as an example, the number of types of morphological features of the target organism is 2, and the total number of morphological features of the target organism is 2; taking image 2 as an example, the number of types of morphological features of the target organism is 2, and the total number of morphological features of the target organism is 4. It can be considered that the degree of recognition of image 2 is higher than that of image 1.
[0205] For another example, if the target organism is a grass carp, the parameters of the degree of recognition include the number of types of the morphological features of the target organism and the area ratio of the morphological features of the target organism in the image showing the target organism. Image 3 shows the fins, tail, trunk and head of the grass carp, accounting for 30% of the area. Image 4 shows the tail of the grass carp, accounting for 80% of the area. It can be considered that the degree of recognition of Image 3 is higher than that of Image 4.
[0206] Exemplarily, assume that the first image set is captured within a first time period, and the first image set includes a third image set and Z fourth image sets, wherein the third image set includes X frames of first images, which are captured by a main camera within the first time period, the multiple auxiliary cameras are Z auxiliary cameras, each of the Z fourth image sets includes 1 group of X frames of second images, the Z fourth image sets are captured by the Z auxiliary cameras within the first time period, the first time period includes X moments, the X frames of first images correspond to X moments, each group of X frames of second images corresponds to X moments, X≥2, Z≥2, and X and Z are both integers.
[0207] In a possible implementation, the main camera and Z auxiliary cameras shoot simultaneously in a first time period, shooting a total of Z+1 video segments. The second image set finally sent to the live broadcast device is one of the Z+1 video segments.
[0208] In one possible implementation, the main camera and Z auxiliary cameras shoot simultaneously in the first time period, and shoot a total of Z+1 videos. In order to refine the screening and reduce the processing complexity and save computing power, the first time period is divided into multiple sub-time periods. For example, the first time period includes X moments, one of which includes Y moments, and the Z+1 sub-videos corresponding to each sub-time period are screened. After screening all sub-time periods in the first time period, multiple screened sub-videos are obtained, that is, the second image set.
[0209] The above scheme selects the second image set from the first image set and transmits it to the live broadcast device, that is, transmits images with obvious morphological characteristics of the target organism to the live broadcast device, further improving the value of the images transmitted to the live broadcast device, thereby improving the live broadcast retention rate.
[0210] Fig.12 A structural block diagram of the device 1000 provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.
[0211] Reference Fig.12 In a possible implementation of the embodiment of the present application, the device may specifically include the following modules:
[0212] The transceiver module 1010 is used to simultaneously shoot the target organism using a main camera and multiple auxiliary cameras based on a first shooting path to obtain a first main image set and multiple first auxiliary image sets; and transmit the first main image set and multiple first auxiliary image sets to a live broadcast device.
[0213] The processing module 1020 is used to determine a second shooting path according to the first image, and to shoot using a main camera and multiple auxiliary cameras based on the second shooting path.
[0214] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method and system embodiment section and will not be repeated here.
[0215] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0216] like Fig.13 As shown, an embodiment of the present application also provides a device 1100, which includes: at least one processor 1110, a memory 1120, and a computer program 1121 stored in the memory and executable on the at least one processor, and when the processor executes the computer program, the steps in any of the above-mentioned method embodiments are implemented.
[0217] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0218] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, a mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0219] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0220] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0221] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0222] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0223] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0224] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for determining a shooting path, characterized in that: Applied to an underwater robot, the underwater robot communicates with a live broadcast device, the underwater robot includes a visual module, the visual module includes a panel, a main camera and multiple auxiliary cameras, the main camera is fixedly connected to the panel based on a connecting portion, each of the auxiliary cameras is connected to the panel and can rotate, the panel is connected to the main body of the underwater robot, and the main camera and the multiple auxiliary cameras face a first direction away from the panel, and the panel can rotate in a three-dimensional space with the center point of the connecting portion as the origin, and the method includes: Based on a first shooting path, the target organism is photographed simultaneously by the main camera and the multiple auxiliary cameras to obtain a first main image set and multiple first auxiliary image sets, wherein the first shooting path includes a first shooting parameter of the panel, a second shooting parameter of the multiple auxiliary cameras and a first moving path of the underwater robot, the first main image set includes multiple frames of first images photographed by the main camera, and the first auxiliary image set includes multiple frames of second images photographed by the auxiliary cameras; Transmitting the first main image set and the plurality of first auxiliary image sets to the live broadcast device; A second shooting path is determined according to the first image, and based on the second shooting path, shooting is performed using the main camera and the multiple auxiliary cameras, wherein the second shooting path includes the third shooting parameters of the panel, the fourth shooting parameters of the multiple auxiliary cameras and the second moving path of the underwater robot.
2. The method according to claim 1, characterized in that in, A first field of view center line of the main camera is perpendicular to a first plane, the first plane is parallel to the panel, and the plurality of auxiliary cameras are distributed around the main camera; The panel can rotate based on a first central axis, a second central axis and a third central axis, the first central axis is the center line of the first field of view, the second central axis and the third central axis are both located in the first plane and are perpendicular to and intersect the center line of the first field of view, the second central axis is perpendicular to the third central axis, and the first central axis, the second central axis and the third central axis all pass through the connecting portion; Wherein, the first shooting parameter or the third shooting parameter indicates: the angle at which the panel rotates along the first central axis and / or the second central axis and / or the third central axis; The multiple auxiliary cameras can rotate in a second plane, the second plane is perpendicular to the first plane and intersects with a first connecting line, and the first connecting line is a connecting line between a center point of the main camera and a center point of the auxiliary camera; The second shooting parameter or the fourth shooting parameter indicates: an angle between a center line of a second field of view of the auxiliary camera and the panel in the second plane.
3. The method according to claim 2, characterized in that The first image includes a central area of the field of view, the first image displays the target organism, and determining a second shooting path according to the first image, and shooting using the main camera and the plurality of auxiliary cameras based on the second shooting path, comprises: Determining that the first image does not satisfy a preset condition, the preset condition comprising: a center point of the organism displayed in the image is located in a central area of a visual field of the image, or a first proportion of all pixels corresponding to the organism displayed in the image are located in the central area of the visual field of the image, and the first proportion is greater than a preset proportion; Calculating the third shooting parameter according to the first image and a first relative position of the target organism relative to the main camera, wherein the third shooting parameter is different from the first shooting parameter; Using the main camera to shoot the target organism according to the third shooting parameters to obtain a second main image set, wherein the second main image set includes a plurality of frames of third images; When the third image satisfies the preset condition, determining that the first moving path is the same as the second moving path, and calculating the fourth shooting parameter according to the first distance between the target organism and the main camera; While the target organism is photographed by the main camera, the target organism is photographed by the multiple auxiliary cameras according to the fourth shooting parameters to obtain multiple second auxiliary image sets, where the second auxiliary image sets include multiple frames of fourth images.
4. The method according to claim 2, characterized in that The first image includes a central area of the visual field, the first image displays the target organism, the angle at which the panel is rotated along the second central axis belongs to a first angle interval, and the angle at which the panel is rotated along the third central axis belongs to a second angle interval; The determining a second shooting path according to the first image, and, based on the second shooting path, using the main camera and the plurality of auxiliary cameras to shoot, includes: Determining that the first image does not satisfy a preset condition, the preset condition comprising: a center point of the organism displayed in the image is located in a central area of a visual field of the image, or a first proportion of all pixels corresponding to the organism displayed in the image are located in the central area of the visual field of the image, and the first proportion is greater than a preset proportion; Calculating a fifth shooting parameter according to the first image and a first relative position between the target organism and the main camera, the fifth shooting parameter indicating that: the panel is rotated along the second central axis by a third rotation angle, and the panel is rotated along the third central axis by a fourth rotation angle; When the third rotation angle does not belong to the first angle interval, or the fourth rotation angle does not belong to the second angle interval, determining the second movement path according to the first relative position, and the second movement path is different from the first movement path; moving along the second moving path; Calculating the third shooting parameter according to the first image and the first relative position; Using the main camera to shoot the target organism according to the third shooting parameters to obtain a second main image set, wherein the second main image set includes a plurality of frames of third images; When the third image satisfies the preset condition, calculating the fourth shooting parameter according to the first distance between the target organism and the main camera; While the target organism is photographed by the main camera, the target organism is photographed by the multiple auxiliary cameras according to the fourth shooting parameters to obtain multiple second auxiliary image sets, where the second auxiliary image sets include multiple frames of fourth images.
5. The method according to claim 2, characterized in that The first image includes a central area of the field of view, the first image displays the target organism, and determining a second shooting path according to the first image, and shooting using the main camera and the plurality of auxiliary cameras based on the second shooting path, comprises: Determining that the first image satisfies a preset condition, wherein the preset condition includes: a center point of the organism displayed in the image is located in a central area of a visual field of the image, or a first proportion of all pixels corresponding to the organism displayed in the image are located in the central area of the visual field of the image, and the first proportion is greater than a preset proportion; When the first distance between the target organism and the main camera is greater than a distance threshold, determining the second moving path according to the first distance, a second relative position of the target organism relative to the underwater robot, and the distance threshold; moving along the second moving path; Calculating the third shooting parameter according to the first image and a first relative position of the target organism relative to the main camera; Using the main camera to shoot the target organism according to the third shooting parameters to obtain a second main image set, wherein the second main image set includes a plurality of frames of third images; When the third image satisfies the preset condition, calculating the fourth shooting parameter according to the first distance; While the target organism is photographed by the main camera, the target organism is photographed by the multiple auxiliary cameras according to the fourth shooting parameters to obtain multiple second auxiliary image sets, where the second auxiliary image sets include multiple frames of fourth images.
6. The method according to any one of claims 3 to 5, characterized in that The underwater robot also includes a positioning module. The calculating the third shooting parameter according to the first image and the first relative position of the target organism relative to the main camera includes: determining the second relative position of the target organism relative to the underwater robot by using the positioning module, and calibrating the first relative position according to the second relative position; determining the third relative position and the fourth relative position of the target organism relative to the main camera in a third plane and a fourth plane respectively according to the first relative position, wherein the third plane is the plane where the first central axis and the second central axis are located, and the fourth plane is the plane where the first central axis and the third central axis are located; determining the fourth rotation angle according to the third relative position; determining the third rotation angle according to the fourth relative position; The using the main camera to photograph the target organism according to the third shooting parameters includes: rotating the panel along the second central axis by a third rotation angle, rotating the panel along the third central axis by a fourth rotation angle, and photographing the target organism using the main camera.
7. The method according to claim 6, characterized in that The calculating of the fourth shooting parameter according to the first distance between the target organism and the main camera, or the calculating of the fourth shooting parameter according to the first distance, comprises: calculating a fifth rotation angle according to the first distance and the third distance of the first connecting line, wherein the fifth rotation angle is within the second plane, and the fifth rotation angle is the angle between the center line of the second field of view of the auxiliary camera and the panel; The using the multiple auxiliary cameras to photograph the target creature according to the fourth shooting parameter includes: rotating the multiple auxiliary cameras according to the fifth rotation angle, and using the multiple auxiliary cameras to photograph the target creature.
8. An underwater robot, characterized in that: The invention comprises a visual module, wherein the visual module comprises a panel, a main camera and a plurality of auxiliary cameras, wherein the main camera is fixedly connected to the panel based on a connecting portion, a first field of view centerline of the main camera is perpendicular to a first plane, the first plane is parallel to the panel, each of the auxiliary cameras is connected to the panel and can rotate, the plurality of auxiliary cameras are distributed around the main camera, the panel is connected to the main body of the underwater robot, and the main camera and the plurality of auxiliary cameras face a first direction away from the panel, the panel can rotate in a three-dimensional space with the center point of the connecting portion as the origin, and the underwater robot is used to perform the steps of the method as claimed in any one of claims 1 to 7.
9. A shooting path determination system, comprising an underwater robot and a live broadcast device, characterized in that: The underwater robot is connected to the live broadcast device, and the underwater robot includes a visual module, and the visual module includes a panel, a main camera and a plurality of auxiliary cameras; The underwater robot is used to perform the steps of the method according to any one of claims 1 to 7; The live broadcast device is used to receive the first main image set and the multiple first auxiliary image sets from the underwater robot, and play pictures to the user according to the first main image set and the multiple first auxiliary image sets.
10. The system according to claim 9, characterized in that The live broadcast device is further used to detect an operation of a user indicating not to watch the target organism, and to send a first instruction message to the underwater robot, wherein the first instruction message indicates not to shoot the target organism; the underwater robot is further used to receive the first instruction message from the live broadcast device, and to determine a third shooting path according to the first instruction message, and to shoot according to the third shooting path, wherein the third shooting path is different from the first shooting path; Alternatively, the live broadcast device is further used to detect the user's operation of setting a fourth shooting path, and to send second indication information to the underwater robot, wherein the second indication information includes the fourth shooting path, and the fourth shooting path is different from the first shooting path; the underwater robot is also used to receive the second indication information from the live broadcast device, and to perform shooting according to the fourth shooting path.
Citation Information
Patent Citations
Multi-view visual device
CN109366528A
Shooting method, robot and mobile terminal
CN109995991A
Underwater panoramic shooting method, panoramic shooting equipment, live broadcast system and storage medium
CN112689091A
Multi-view camera and crawler comprising multi-view camera
CN113099174A
Navigation method and device of underwater vehicle, computer equipment and medium
CN114111789A
