Method for identifying underwater organisms and related equipment and system

Abstract

The present application provides a method for identifying underwater creatures and related equipment and systems. The method is applied to an underwater robot including a main camera and multiple auxiliary cameras. The method includes: first shooting with the main camera, and identifying whether the main camera has captured underwater creatures. When it is determined that underwater creatures have been captured, multiple auxiliary cameras are rotated to simultaneously capture the underwater creatures to obtain a first image set. Subsequently, a second image set with obvious characteristics of the target creature in the first image set is screened out and transmitted to a live broadcast device for playback. In the above scheme, after the main camera captures the underwater creature, the main camera and the auxiliary camera are used to capture the target creature at the same time, which can improve the success rate of capturing valuable images. In addition, a second image set is screened out from the first image set and transmitted to the live broadcast device, further improving the value of the images transmitted to the live broadcast device, thereby improving the live broadcast retention rate.

Description

Technical Field

[0001] The present application belongs to the field of underwater robots, and in particular, relates to a method for identifying underwater creatures and related equipment and systems. Background Art

[0002] Underwater photography is widely used in teaching, archaeology and other fields; however, it requires high physical fitness and diving ability of the photographer. Using underwater robots to shoot is a good choice. If combined with live broadcast equipment on land, the images taken by underwater robots can be broadcast live, and many ordinary people can see the underwater scenery in real time without diving.

[0003] The live broadcast retention rate is one of the important indicators to measure the live broadcast effect. It reflects the audience's loyalty and stickiness to the live broadcast content. Specifically, the live broadcast retention rate refers to the audience's stay time after entering the live broadcast room for the first time and the subsequent return visit rate during the live broadcast. This data can intuitively reflect the quality of the live broadcast content. Therefore, in order to improve the live broadcast retention rate, the underwater robot needs to capture as many valuable pictures as possible underwater. Summary of the invention

[0004] The purpose of this application is to provide a method for identifying underwater creatures and related equipment and systems. According to the method for identifying underwater creatures provided by this application, the success rate of capturing valuable images can be improved, and the retention rate of live broadcasts can be improved.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying underwater organisms, which is applied to an underwater robot. The underwater robot communicates with a live broadcast device. The underwater robot includes a visual module, and the visual module includes a main camera and multiple auxiliary cameras. The method includes: using the main camera to obtain an initial image; identifying whether the initial image shows a target organism; when the initial image shows the target organism, using the main camera and multiple auxiliary cameras to simultaneously shoot the target organism to obtain a first image set, the first image set including multiple frames of first images shot by the main camera, and multiple frames of second images shot by multiple auxiliary cameras; screening out a second image set from the first image set, the target organisms in the second image set being more recognizable than other image sets, and the other image sets including: images in the first image set except the second image set; transmitting the second image set to the live broadcast device.

[0006] Optionally, identifying whether the initial image shows the target organism includes: inputting the initial image into a first neural network model to obtain a biological detection result, the biological detection result indicating whether the initial image contains the target organism; wherein the first neural network model is trained based on a first training data set, the first training data set includes first input data and first target data, the first input data includes multiple frames of first sample images, the multiple frames of first sample images respectively display different sample underwater organisms, the sample underwater organisms display sample morphological features, the first target data includes multiple groups of first sample coordinate data, the multiple frames of first sample images correspond one-to-one to the multiple groups of first sample coordinate data, each group of first sample coordinate data is respectively the pixel coordinates of the sample morphological features in each frame of the first sample image; wherein, in the case where the biological detection result includes the first coordinate data corresponding to the initial image, the biological detection result indicates that the initial image contains the target organism, or, in the case where the biological detection result does not include the coordinate data, the biological detection result indicates that the initial image does not contain the target organism.

[0007] Optionally, the parameter used to characterize the degree of recognizability of the target organism includes at least one of the following: the number of types of morphological features of the target organism, the total number of morphological features of the target organism, or the area ratio of the morphological features of the target organism in an image showing the target organism.

[0008] Optionally, the first image set is captured within a first time period, the first image set includes a third image set and Z fourth image sets, wherein the third image set includes X frames of first images, the X frames of first images are captured by a main camera within the first time period, the multiple auxiliary cameras are Z auxiliary cameras, each fourth image set in the Z fourth image sets includes 1 group of X frames of second images, the Z fourth image sets are captured by the Z auxiliary cameras within the first time period, the first time period includes X moments, the X frames of first images correspond to X moments respectively, each group of X frames of second images corresponds to X moments respectively, X≥2, Z≥2, and X and Z are both integers; screening out the second image set from the first image set includes: obtaining Y frames of first images and Z groups of Y frames of second images corresponding to Y moments among the X moments, inputting the Y frames of first images into a second neural network model, obtaining first recognition degree information, the first recognition degree information is used to characterize the degree of recognition of the target organism in the Y frames of first images, and the first recognition degree information includes: in the Y frames of first images, the target organism The method comprises the steps of: inputting Z groups of Y frame second images into a second neural network model to obtain Z groups of second recognition degree information, each group of second recognition degree information is used to characterize the recognition degree of the target organism in each group of Y frame second images, and each group of second recognition degree information includes: in each group of Y frame second images, the second number of types of morphological features of the target organism, the second total number of morphological features of the target organism, or the second area ratio of morphological features of the target organism in the image showing the target organism; according to the first recognition degree information and the Z groups of second recognition degree information, selecting a Y frame third image from the Y frame first image and the Z groups of Y frame second images, the recognition degree of the Y frame third image is higher than that of other Y frame images, and the other Y frame images are images other than the Y frame third image in the Y frame first image and the Z groups of Y frame second images; setting the image set corresponding to the Y frame third image to the second image set, 1≤Y≤X and Y is an integer.

[0009] Optionally, the second neural network model is trained based on a second training data set, the second training data set includes second input data and second target data, the second input data includes multiple frames of second sample images, the multiple frames of second sample images respectively display different sample underwater organisms, the sample underwater organisms display sample morphological features, the second target data includes multiple sets of sample identification data, the multiple frames of second sample images correspond one-to-one to the multiple sets of sample identification degree information, the multiple sets of sample identification degree information include: in the second sample image, the number of types of sample morphological features, the total number of sample morphological features, and the number of pixels occupied by the sample morphological features.

[0010] Optionally, the sample underwater organism is a fish-shaped underwater organism, and the sample morphological characteristics include: fins, tail, trunk and head; or, the sample underwater organism is a tentacle-shaped underwater organism, and the sample morphological characteristics include: head and tentacles; or, the sample underwater organism is an appendage-shaped underwater organism, and the sample morphological characteristics include: cephalothorax, abdomen and appendages.

[0011] Optionally, the visual module also includes a panel, a main camera and the panel are fixedly connected based on a connecting part, each auxiliary camera is connected to the panel and can rotate, multiple auxiliary cameras are distributed around the main camera, the panel is connected to the main body of the underwater robot, and the main camera and multiple auxiliary cameras are facing a first direction away from the panel, and the panel can rotate in a three-dimensional space with the center point of the connecting part as the origin; using the main camera to obtain the initial image, including: based on an initial shooting path, using the main camera to obtain the initial image, wherein the initial shooting path includes: an initial rotation angle of the panel and an initial moving trajectory of the underwater robot; the method also includes: when the initial image does not show the target organism, changing the initial shooting path, and based on the changed shooting path, using the main camera to obtain the corresponding image.

[0012] Optionally, the visual module also includes a panel, the main camera is fixedly connected to the panel, the first field of view center line of the main camera is perpendicular to the first plane, the first plane is parallel to the panel, each auxiliary camera is connected to the panel and can rotate, multiple auxiliary cameras are distributed around the main camera, the panel is connected to the main body of the underwater robot, and the main camera and multiple auxiliary cameras are facing a first direction away from the panel, multiple auxiliary cameras can rotate in a second plane, the second plane is perpendicular to the first plane and intersects with a first connecting line, the first connecting line is a connecting line between the center point of the main camera and the center point of the auxiliary camera, the underwater robot also includes a positioning module, and the method also includes: using the positioning module to measure the second distance between the target organism and the underwater robot, and calibrating the first distance of the target organism relative to the main camera according to the second distance; calculating the rotation angle according to the first distance and the third distance of the first connecting line, the rotation angle is in the second plane, and the rotation angle is the angle between the second field of view center line of the auxiliary camera and the panel; rotating multiple auxiliary cameras according to the rotation angle.

[0013] In the second aspect, an embodiment of the present application provides an underwater robot, comprising a visual module, the visual module comprising a panel, a main camera and multiple auxiliary cameras, wherein the main camera is fixedly connected to the panel, the center line of the first field of view of the main camera is perpendicular to the first plane, the first plane is parallel to the panel, each auxiliary camera is connected to the panel and can rotate, the multiple auxiliary cameras are distributed around the main camera, the panel is connected to the main body of the underwater robot, and the main camera and the multiple auxiliary cameras are facing a first direction away from the panel, and the underwater robot is used to perform the steps of the method described in the first aspect.

[0014] In a third aspect, an embodiment of the present application provides a system for identifying underwater creatures, comprising an underwater robot and a live broadcast device, wherein the underwater robot is connected to the live broadcast device, the underwater robot comprises a visual module, the visual module comprises a panel, a main camera and multiple auxiliary cameras, wherein the main camera is fixedly connected to the panel, a first field of view centerline of the main camera is perpendicular to a first plane, the first plane is parallel to the panel, each auxiliary camera is connected to the panel and can rotate, multiple auxiliary cameras are distributed around the main camera, the panel is connected to the main body of the underwater robot, and the main camera and the multiple auxiliary cameras are facing a first direction away from the panel, the underwater robot is used to execute the steps of the method described in the first aspect; the live broadcast device is used to receive a second image set from the underwater robot, and play pictures to the user according to the second image set.

[0015] The fourth aspect of an embodiment of the present application provides a device for identifying underwater organisms, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.

[0016] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0017] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: firstly, underwater creatures are photographed by the main camera, and then the target creatures are photographed by the main camera and the auxiliary camera at the same time to obtain an image set, which can improve the success rate of capturing valuable images. In addition, the captured image set is screened out, and the image set with a high degree of recognition is transmitted to the live broadcast device, that is, the value of the images transmitted to the live broadcast device is further improved, thereby improving the live broadcast retention rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a method 100 for identifying underwater organisms provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a system for identifying underwater organisms provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of an example of a visual module provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of the rotation mode of the panel provided in the embodiment of the present application;

[0022] Figure 5 A schematic diagram of the rotation method of the auxiliary camera provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of acquiring a first image based on a first shooting parameter using a main camera according to an embodiment of the present application;

[0024] Figure 7 A schematic diagram of an example of a method for calculating a fourth rotation angle provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of a sixth rotation angle provided in an embodiment of the present application;

[0026] Fig. 9 A schematic diagram of an example of a first image provided in an embodiment of the present application;

[0027] Fig.10 A schematic diagram of an example of calculating a fifth rotation angle provided in an embodiment of the present application;

[0028] Fig.11 A schematic diagram of an example of a fourth image captured by multiple auxiliary cameras provided in an embodiment of the present application;

[0029] Fig.12 A structural block diagram of a device 1000 provided in an embodiment of the present application is shown;

[0030] Fig.13 A schematic diagram of a device 1100 for identifying underwater creatures provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0032] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0033] At present, underwater photography is widely used in teaching, archaeology and other fields; however, it requires high physical fitness and diving ability of the photographer. Using underwater robots to shoot is a good choice. If combined with live broadcast equipment on land, the pictures taken by underwater robots can be broadcast live, and many ordinary people can see the underwater scenery in real time without diving.

[0034] The live broadcast retention rate is one of the important indicators to measure the live broadcast effect. It reflects the audience's loyalty and stickiness to the live broadcast content. Specifically, the live broadcast retention rate refers to the audience's stay time after entering the live broadcast room for the first time and the subsequent return visit rate during the live broadcast. This data can intuitively reflect the quality of the live broadcast content. Therefore, in order to improve the live broadcast retention rate, the underwater robot needs to capture as many valuable pictures as possible underwater.

[0035] Therefore, how to capture as many valuable images as possible becomes an urgent problem to be solved.

[0036] In view of this, the present application provides a method for identifying underwater creatures. The method is applied to an underwater robot including a main camera and multiple auxiliary cameras. The method comprises: first shooting with the main camera, and identifying whether the main camera has captured underwater creatures. When it is determined that underwater creatures have been captured, multiple auxiliary cameras are rotated to simultaneously capture the underwater creatures to obtain a first image set. Subsequently, a second image set with obvious characteristics of the target creature in the first image set is screened out, and transmitted to a live broadcast device for playback.

[0037] In the above scheme, after the main camera captures underwater creatures, the main camera and the auxiliary camera are used to capture the target creatures at the same time, which can improve the success rate of capturing valuable images. In addition, the second image set is selected from the first image set and transmitted to the live broadcast device, further improving the value of the images transmitted to the live broadcast device, thereby improving the live broadcast retention rate.

[0038] Figure 1 A schematic diagram of a method 100 for identifying underwater organisms provided in an embodiment of the present application.

[0039] The method 100 is applied to an underwater robot, which communicates with a live broadcast device. The underwater robot includes a vision module, and the vision module includes a main camera and multiple auxiliary cameras.

[0040] S101, using a main camera to acquire an initial image.

[0041] That is, the underwater robot uses the main camera to take pictures.

[0042] S102, identifying whether the initial image shows a target organism.

[0043] Exemplarily, the initial image is input into the first neural network model to obtain the biological detection result.

[0044] The biological detection result indicates whether the target biological being exists in the initial image.

[0045] For example, the first neural network model is trained based on the first training data set. The first training data set includes first input data and first target data. The first input data includes multiple frames of first sample images, and the multiple frames of first sample images respectively display different sample underwater organisms, and the sample underwater organisms display at least one of the sample morphological features. The first target data includes multiple groups of first sample coordinate data, and the multiple frames of first sample images correspond to the multiple groups of first sample coordinate data one by one, and each group of first sample coordinate data is the pixel coordinate of the sample morphological feature in each frame of the first sample image.

[0046] Specific examples are given below for sample underwater organisms and sample morphological characteristics.

[0047] Example 1: The sample underwater organism is a fish-shaped underwater organism, and the sample morphological characteristics include: fins, tail, trunk and head.

[0048] Among them, fish-shaped underwater creatures can include some fish and some underwater mammals. For example, fish include freshwater fish and marine fish. Freshwater fish include black carp, grass carp, silver carp, bighead carp, bream and crucian carp, etc.; marine fish include hairtail, tuna, large yellow croaker, small yellow croaker, etc. Underwater mammals include dolphins and whales, etc. Their bodies at least include: fins, tail, trunk and head.

[0049] Example 2: The sample underwater organism is an underwater organism with tentacles, and the sample morphological characteristics include: head and tentacles.

[0050] Among them, underwater creatures with tentacles can include some underwater coelenterates, such as sea anemones, jellyfish, coral polyps and sea cucumbers, etc.; they can also include some underwater mollusks, such as squids and octopuses, etc. Their bodies include at least two parts, a head and a tentacle part.

[0051] Example 3: The sample underwater organism is an underwater organism with appendages, and the sample morphological characteristics include: cephalothorax, abdomen and appendages.

[0052] Among them, underwater creatures with appendages can include some crustaceans, such as shrimps and crabs, whose bodies are divided into two parts: cephalothorax and abdomen, and they also have powerful appendages for walking or swimming.

[0053] Alternatively, the underwater organisms and morphological characteristics of the samples may be realized in other ways without limitation.

[0054] Taking a frame of the first sample image as an example, assuming that the first sample image 1 shows a crucian carp, and two samples of the crucian carp can be seen in the first sample image, morphological feature 1 (tail) and morphological feature 2 (fin). The first target data 1 corresponds to the first sample image 1, and the first target data 1 includes first sample coordinate data 1, and the first sample coordinate data 1 includes a set of coordinate data. Among them, the first sample coordinate data 1 includes the pixel coordinates of the morphological feature 1 in the first sample image 1, and the pixel coordinates of the morphological feature 2 in the first sample image 1.

[0055] The first input data includes other first sample images, and the first target data includes other first sample coordinate data, which can also refer to the description of the first sample image 1 and the first sample coordinate data 1.

[0056] Recognition result 1: when the biological detection result includes the first coordinate data corresponding to the initial image, the biological detection result indicates that the target biological exists in the initial image. Then, S103 is executed.

[0057] Alternatively, in recognition result 2, when the biological detection result does not include coordinate data, the biological detection result indicates that the target organism does not exist in the initial image. In this case, the underwater robot can change the shooting path, continue to use the main camera to shoot a new initial image, and perform recognition with reference to the method of S102 until the recognition result 1 is obtained, and then execute S103. Specifically, the scheme related to changing the shooting path will be described in detail below with reference to the accompanying drawings.

[0058] In the above scheme, after the main camera captures underwater creatures, the main camera and the auxiliary camera are used to capture the target creatures at the same time. Compared with directly controlling the main camera and the auxiliary camera to capture together without identifying whether the target creatures exist, this can improve the success rate of capturing valuable images and save power consumption.

[0059] S103, when the initial image shows the target organism, using a main camera and a plurality of auxiliary cameras to simultaneously photograph the target organism to obtain a first image set.

[0060] The first image set includes multiple frames of first images taken by a main camera and multiple frames of second images taken by multiple auxiliary cameras.

[0061] Exemplarily, S103 involves collaborative shooting with the main camera and the auxiliary camera, which will be described in detail below with reference to the accompanying drawings.

[0062] S104: Filter out a second image set from the first image set.

[0063] The target organisms in the second image set are more recognizable than those in other image sets, and the other image sets include: images in the first image set except the second image set.

[0064] The degree of recognition can be understood as the degree to which the target organism can be seen by the user. The higher the degree of recognition, the more obvious the morphological features of the target organism are to the user, and the more the user can understand the target organism.

[0065] Exemplarily, the parameters used to characterize the degree of recognition of the target organism include at least one of the following: the number of types of morphological features of the target organism, the total number of morphological features of the target organism, or the area ratio of the morphological features of the target organism in the image showing the target organism. Among them, the area ratio of morphological features is the ratio of the number of pixels used to display the morphological features to the total number of pixels in the entire image. Optionally, when the parameters such as the size and resolution of the photos taken by the main camera and the auxiliary camera are consistent, the area ratio of the morphological features of the target organism in the image showing the target organism can also be replaced by the area ratio of the morphological features of the target organism in the image showing the target organism.

[0066] When determining the degree of recognition of an image, the magnitude of the above parameters can be compared. The larger the value of the above parameters, the higher the degree of recognition. It can be understood that the larger the number of types of morphological features of the target organism and the total number of morphological features of the target organism, the more obvious the morphological features of the target organism displayed in the image; the larger the area proportion of the morphological features of the target organism in the image showing the target organism, the higher the possibility that the morphological features of the target organism can be seen or seen clearly by the user.

[0067] Optionally, the above parameters preferably include the number of types of morphological features of the target organism. Secondly, it is preferred to include the total number of morphological features of the target organism. Secondly, it is preferred to include the area ratio of the morphological features of the target organism in the image showing the target organism. It can be understood that in two images, when the total number of morphological features of the target organism is the same, or when the area ratio of the morphological features of the target organism in the image showing the target organism is the same, the more types of morphological features of the target organism, the higher the degree of recognition is generally. When the area ratio of the morphological features of the target organism in the image showing the target organism is the same, the more pixels the morphological features of the target organism occupy, the higher the degree of recognition is generally.

[0068] In a possible implementation, after obtaining the parameters for characterizing the degree of recognition of the target organism, the degree of recognition can be obtained by weighted calculation. For example, the number of types of morphological features of the target organism corresponds to a first weight, the total number of morphological features of the target organism corresponds to a second weight, and the area ratio of the morphological features of the target organism in the image showing the target organism corresponds to a third weight. The first weight is greater than the second weight, which is greater than the third weight.

[0069] For example, the target organism is a squid, and the parameters of the degree of recognition include the number of types of morphological features of the target organism and the total number of morphological features of the target organism. Image 1 shows the head and 1 tentacle of the squid, and image 2 shows the head and 3 tentacles of the squid. Taking image 1 as an example, the number of types of morphological features of the target organism is 2, and the total number of morphological features of the target organism is 2; taking image 2 as an example, the number of types of morphological features of the target organism is 2, and the total number of morphological features of the target organism is 4. It can be considered that the degree of recognition of image 2 is higher than that of image 1.

[0070] For another example, if the target organism is a grass carp, the parameters of the degree of recognition include the number of types of the morphological features of the target organism and the area ratio of the morphological features of the target organism in the image showing the target organism. Image 3 shows the fins, tail, trunk and head of the grass carp, accounting for 30% of the area. Image 4 shows the tail of the grass carp, accounting for 80% of the area. It can be considered that the degree of recognition of Image 3 is higher than that of Image 4.

[0071] Exemplarily, assume that the first image set is captured within a first time period, and the first image set includes a third image set and Z fourth image sets, wherein the third image set includes X frames of first images, which are captured by a main camera within the first time period, the multiple auxiliary cameras are Z auxiliary cameras, each of the Z fourth image sets includes 1 group of X frames of second images, the Z fourth image sets are captured by the Z auxiliary cameras within the first time period, the first time period includes X moments, the X frames of first images correspond to X moments, each group of X frames of second images corresponds to X moments, X≥2, Z≥2, and X and Z are both integers.

[0072] In a possible implementation, the main camera and Z auxiliary cameras shoot simultaneously in a first time period, shooting a total of Z+1 video segments. The second image set finally sent to the live broadcast device is one of the Z+1 video segments.

[0073] In one possible implementation, the main camera and Z auxiliary cameras shoot simultaneously in the first time period, and shoot a total of Z+1 videos. In order to refine the screening and reduce the processing complexity and save computing power, the first time period is divided into multiple sub-time periods. For example, the first time period includes X moments, one of which includes Y moments, and the Z+1 sub-videos corresponding to each sub-time period are screened. After screening all sub-time periods in the first time period, multiple screened sub-videos are obtained, that is, the second image set.

[0074] As an implementation of S104, taking one of the sub-periods as an example, Y frames of first images and Z groups of Y frames of second images corresponding to Y moments in X moments are obtained. The screening process can be implemented by a neural network.

[0075] Step 1: input the Y-frame first image into the second neural network model to obtain first recognition degree information.

[0076] The first recognition degree information is used to characterize the degree of recognition of the target organism in the first image of the Y frame. The first recognition degree information includes: the first number of types of morphological features of the target organism in the first image of the Y frame, the first total number of morphological features of the target organism, or the first area ratio of the morphological features of the target organism in the image showing the target organism.

[0077] Step 2, input Z groups of Y frames of second images into the second neural network model respectively, and obtain Z groups of second recognition degree information respectively.

[0078] Each set of second recognition degree information is used to characterize the recognition degree of the target organism in each set of Y-frame second images. Each set of second recognition degree information includes: in each set of Y-frame second images, the second number of types of morphological features of the target organism, the second total number of morphological features of the target organism, or the second area ratio of the morphological features of the target organism in the image showing the target organism;

[0079] Step 3: Filter out Y frames of third images from Y frames of first images and Z groups of Y frames of second images according to the first recognition degree information and Z groups of second recognition degree information.

[0080] The Y frame third image is more recognizable than other Y frame images, and the other Y frame images are the Y frame first image and the images in the Z group of Y frame second images except the Y frame third image.

[0081] Step 4: Set the image set corresponding to the third image of frame Y as the second image set, where 1≤Y≤X and Y is an integer.

[0082] Exemplarily, the second neural network model is obtained by training based on the second training data set. The second training data set includes second input data and second target data, the second input data includes multiple frames of second sample images, the multiple frames of second sample images respectively display different sample underwater organisms, and the sample underwater organisms display sample morphological features, the second target data includes multiple groups of sample identification data, the multiple frames of second sample images correspond one-to-one to the multiple groups of sample identification degree information, and the multiple groups of sample identification degree information include: in the second sample image, the number of types of sample morphological features, the total number of sample morphological features, and the number of pixels occupied by the sample morphological features.

[0083] Among them, the sample underwater organisms and sample morphological characteristics can refer to the relevant content of S102.

[0084] The above scheme selects the second image set from the first image set and transmits it to the live broadcast device, that is, transmits images with obvious morphological characteristics of the target organism to the live broadcast device, further improving the value of the images transmitted to the live broadcast device, thereby improving the live broadcast retention rate.

[0085] S105: Transmit a second image set to the live broadcast device.

[0086] In a possible implementation, the underwater robot does not transmit other image sets.

[0087] In another possible implementation, the underwater robot transmits a first image set and a priority identifier, where the priority identifier is used to indicate that the second image set in the first image set is an image set with a higher degree of recognition within the first time period.

[0088] Figure 2 A schematic diagram of a system for identifying underwater organisms provided in an embodiment of the present application.

[0089] like Figure 2 As shown, the system includes an underwater robot, a controller and a live broadcast device. The controller can be installed inside the underwater robot, or independently located underwater, or independently located above water, or installed with the live broadcast device. The controller serves as a media device for communication between the underwater robot and the live broadcast device.

[0090] An underwater robot is used to execute the steps of the above method.

[0091] The live broadcast device is used to receive the second image set from the underwater robot, and broadcast the second image set to the user.

[0092] In a possible implementation, the live broadcast device does not receive other image sets.

[0093] In another possible implementation, the live broadcast device receives a first image set and a priority identifier, where the priority identifier is used to indicate that the second image set in the first image set is an image set with a higher degree of recognition within the first time period. The live broadcast device may choose to play the picture only based on the second image combination. Alternatively, the display screen of the live broadcast device includes a first area and a second area, and the viewing priority of the first area is higher than the viewing priority of the second area. The live broadcast device plays the picture based on the second image set in the first area, and plays the picture based on other image sets in the second area.

[0094] Exemplarily, S102 is performed by a controller. That is, S102 can be performed by an underwater robot or by a separate controller.

[0095] Before introducing the solutions related to changing the shooting path and the solution of coordinated shooting between the main camera and the auxiliary camera, the underwater robot involved in this application is first introduced.

[0096] Exemplarily, the underwater robot may include a vision module, a positioning module, a navigation module, a driving module, a lighting module, and the like.

[0097] The visual module includes a panel, a main camera and a plurality of auxiliary cameras. The panel is connected to the main body of the underwater robot based on a connection part 1, the main camera is fixedly connected to the panel, and each auxiliary camera is connected to the panel based on a connection part 2 and can rotate. The main camera and the plurality of auxiliary cameras face a first direction away from the panel. The plurality of auxiliary cameras are distributed around the main camera.

[0098] The positioning module is used to determine the relative position of the target organism with respect to the underwater robot, or to determine the distance between the target organism and the underwater robot.

[0099] The navigation module is used to automatically plan a path to a target location in the map reference system after determining its own position in the map reference system and reach the target location point along the path.

[0100] The driving module is used to drive other modules to operate. For example, the driving module includes a stepper motor, and the driving module is electrically connected to the panel or the auxiliary camera; the driving module controls the rotation angle of the panel or the auxiliary camera by sending a pulse signal. For another example, the driving module includes a servo motor and a servo driver, and the driving module is electrically connected to the panel or the auxiliary camera; the driving module is used to control the rotation angle of the panel or the auxiliary camera.

[0101] Among them, the lighting module is used to provide lighting function for the visual module to improve the shooting effect of the visual module.

[0102] For the convenience of explanation, this application mainly provides a detailed description of the visual module, and does not limit the implementation methods of other modules included in the underwater robot, nor does it limit the connection method between the visual module and the underwater robot body.

[0103] Figure 3 A schematic diagram of an example of a visual module provided in an embodiment of the present application.

[0104] like Figure 3 As shown in (a), the A surface of the display panel faces the first direction. The main camera is fixedly connected to the panel and is located at the center of the panel. The auxiliary cameras 1 to 4 are respectively located around the main camera.

[0105] like Figure 3 As shown in (b) in the figure, side B of the display panel is connected to the main body of the underwater robot. Figure 3 In (b), the multiple cameras located on the A surface are indicated by dotted lines. Exemplarily, part or all of the panel is connected to the underwater robot body. For example, a connector is provided at a position of the B surface of the panel corresponding to the main camera, and the panel is connected to the underwater robot body based on the connector.

[0106] It should be noted that Figure 3 In the figure, the shape or size of the panel is only an example, the position of the main camera is only an example, and the number or position of the auxiliary cameras is only an example, and there is no limitation on these.

[0107] For the visual module, possible rotations of the panel are given. Figure 4 A schematic diagram of the rotation method of the panel provided in an embodiment of the present application.

[0108] like Figure 4 As shown, the A side of the display panel is shown. The panel can rotate based on the first central axis, the second central axis and the third central axis. The double-headed arrows corresponding to the first central axis, the second central axis and the third central axis indicate the direction in which the panel can rotate.

[0109] Among them, the first central axis is the center line of the first field of view, the second central axis and the third central axis are both located in the first plane and are perpendicular to and intersect with the first field of view center line, the second central axis is perpendicular to the third central axis, and the first central axis, the second central axis and the third central axis all pass through the connection between the main camera and the panel.

[0110] It should be noted that Figure 4 The positions and number of auxiliary cameras are for example only, and the shape or size of the panel are for example only and are not limited.

[0111] For the vision module, possible rotation methods of the auxiliary camera are given. Figure 5 A schematic diagram of the rotation method of the auxiliary camera provided in an embodiment of the present application.

[0112] like Figure 5 As shown, multiple auxiliary cameras can rotate in a second plane, and the second plane is perpendicular to the first plane and intersects with the first connecting line. Specifically, the intersection line of the first plane and the second plane is represented by a dotted line, and the first connecting line is included in the intersection line and represented by a solid line. The first connecting line is a connecting line between the center point of the main camera and the center point of the auxiliary camera.

[0113] For example, the connection portion 2 may be a hinge, such as a single-axis hinge, which is used to limit the auxiliary camera from rotating within the second plane.

[0114] The double-headed arrow corresponding to the auxiliary camera 1 represents the rotatable direction of the auxiliary camera 1 in the second plane.

[0115] The following is a detailed description of a solution related to changing the shooting path, which involves the coordinated shooting of the main camera and the auxiliary camera.

[0116] Step X1, based on the first shooting path, use the main camera and multiple auxiliary cameras to simultaneously shoot the target organism to obtain a first main image set and multiple first auxiliary image sets.

[0117] The first shooting path includes the first shooting parameters of the panel, the second shooting parameters of the plurality of auxiliary cameras and the first moving path of the underwater robot.

[0118] For example, the first shooting parameter indicates: the angle at which the panel rotates along the first central axis and / or the second central axis and / or the third central axis. The second shooting parameter indicates: the angle between the center line of the second field of view of the auxiliary camera and the panel in the second plane. The first moving path is the moving path referenced by the underwater robot when acquiring the first main image set and multiple first auxiliary image sets.

[0119] The first main image set includes multiple frames of first images taken by the main camera, and the first auxiliary image set includes multiple frames of second images taken by the auxiliary camera.

[0120] Step X2: transmitting a first main image set and a plurality of first auxiliary image sets to a live broadcast device.

[0121] In the present application, the underwater robot communicates with the live broadcast device. In one possible implementation, the underwater robot communicates directly with the live broadcast device. In another possible implementation, the underwater robot communicates with the live broadcast device through an onshore controller.

[0122] Step X3: determining a second shooting path according to the first image, and shooting using a main camera and multiple auxiliary cameras based on the second shooting path.

[0123] The second shooting path includes the third shooting parameters of the panel, the fourth shooting parameters of the plurality of auxiliary cameras and the second moving path of the underwater robot.

[0124] For example, the third shooting parameter indicates: the angle at which the panel rotates along the first central axis and / or the second central axis and / or the third central axis. The fourth shooting parameter indicates: the angle between the center line of the second field of view of the auxiliary camera and the panel in the second plane. The first moving path is the moving path that the underwater robot refers to when continuing to shoot based on the main camera and multiple auxiliary cameras.

[0125] Exemplarily, determining the second shooting path according to the first image can be understood as determining whether to change the first shooting path according to the first image. If the first shooting path is to be changed, the second shooting path is different from the first shooting path, otherwise they are the same.

[0126] Whether to change the first shooting path can be determined according to whether one or more preset conditions are met. The following first introduces several preset conditions involved in this application, and then further introduces step X3.

[0127] For example, preset condition 1 includes: the center point of the organism displayed in the image is located in the central area of ​​the field of view of the image, or a first proportion of all pixels corresponding to the organism displayed in the image are located in the central area of ​​the field of view of the image, and the first proportion is greater than the preset proportion.

[0128] For another example, preset condition 2 includes: the rotation angle of the panel is within a preset angle range.

[0129] For another example, preset condition 3 includes: a first distance between the target organism and the main camera is less than or equal to a distance threshold.

[0130] Several possible examples of step X3 are given below.

[0131] Example 1: The first image includes a central area of ​​the field of view, and the first image displays a target organism.

[0132] Step A1, determining that the first image does not meet preset condition 1.

[0133] Step A2: calculating a third shooting parameter according to the first image and a first relative position of the target organism relative to the main camera.

[0134] The third shooting parameter is different from the first shooting parameter.

[0135] Specifically, how to calculate the third shooting parameter will be described below in conjunction with the accompanying drawings.

[0136] Step A3: photograph the target organism using the main camera according to the third photographing parameter to obtain a second main image set.

[0137] The second main image set includes multiple frames of third images.

[0138] Step A4, when the third image meets the preset condition, determining that the first moving path is the same as the second moving path, and calculating a fourth shooting parameter according to the first distance between the target organism and the main camera.

[0139] Specifically, how to calculate the fourth shooting parameter will be described below in conjunction with the accompanying drawings.

[0140] Step A5: while photographing the target organism with the main camera, photograph the target organism with the plurality of auxiliary cameras according to the fourth photographing parameter to obtain a plurality of second auxiliary image sets.

[0141] The second auxiliary image set includes multiple frames of fourth images.

[0142] According to Example 1, when preset condition 1 is not met, preset condition 1 can be met by adjusting the rotation angle of the panel. When the target organism is not located in the central area of ​​the image, the third shooting parameter of the main camera is calculated (that is, the rotation angle of the panel is recalculated). In the image obtained by the main camera based on the third shooting parameter, the target organism can be located in the central area, which means that the preset condition 1 can be met without changing the moving path of the underwater robot. Therefore, the first moving path is the same as the second moving path. Since the panel is connected to the auxiliary camera, after the rotation angle of the panel changes, the viewing angle of the picture taken by the auxiliary camera also changes, so as to ensure that the auxiliary camera can capture the target organism, so it is necessary to calculate the fourth shooting parameter of the auxiliary camera (that is, recalculate the rotation angle of the auxiliary camera). Thus, without changing the moving path, adjusting the rotation angle of the panel and adjusting the rotation angle of the auxiliary camera accordingly can make the target organism located in the central area of ​​the field of view of the picture captured by the main camera, and the auxiliary camera can capture the target organism at the same time.

[0143] Example 2, the first image includes a central area of ​​the field of view, and the first image displays the target organism. Preset condition 2 specifically includes: the angle at which the panel rotates along the second central axis belongs to the first angle interval, and the angle at which the panel rotates along the third central axis belongs to the second angle interval. Exemplarily, the first angle interval and the second angle interval can be determined according to the connection method between the panel and the underwater robot body.

[0144] Step B1, determining that the first image does not satisfy preset condition 1.

[0145] Step B2, calculating a fifth shooting parameter according to the first image and a first relative position between the target organism and the main camera.

[0146] The fifth shooting parameter indicates that: the panel rotates along the second central axis by a third rotation angle, and the panel rotates along the third central axis by a fourth rotation angle.

[0147] Step B3: when the preset condition 2 is not met, determine the second moving path according to the first relative position.

[0148] Wherein, not satisfying the preset condition 2 may include: the third rotation angle does not belong to the first angle interval, and / or the fourth rotation angle does not belong to the second angle interval.

[0149] The second moving path is different from the first moving path.

[0150] Exemplarily, the underwater robot may randomly determine a path different from the first moving path as the second moving path. Alternatively, assuming that the panel theoretically (but not actually) rotates to face the second direction according to the third rotation angle and the fourth rotation angle, the second moving path is determined according to the second direction. Alternatively, the second moving path is determined according to the position of the underwater creature relative to the underwater robot.

[0151] Step B4, moving along the second moving path.

[0152] Step B5, calculating a third shooting parameter according to the first image and the first relative position.

[0153] Specifically, how to calculate the third shooting parameter will be described below in conjunction with the accompanying drawings.

[0154] Step B6: photograph the target organism using the main camera according to the third photographing parameter to obtain a second main image set.

[0155] The second main image set includes multiple frames of third images.

[0156] Step B7: when the third image satisfies the preset condition 1, a fourth shooting parameter is calculated according to the first distance between the target organism and the main camera.

[0157] Specifically, how to calculate the fourth shooting parameter will be described below in conjunction with the accompanying drawings.

[0158] Step B8, while using the main camera to shoot the target organism, use multiple auxiliary cameras to shoot the target organism according to the fourth shooting parameter to obtain multiple second auxiliary image sets.

[0159] The second auxiliary image set includes multiple frames of fourth images.

[0160] According to Example 2, when preset condition 1 is not met, preset condition 1 cannot be met by adjusting the rotation angle of the panel; preset condition 1 can be met by adjusting the moving path of the underwater robot and adjusting the rotation angle of the panel accordingly. When the target organism is not located in the central area of ​​the image, the fifth shooting parameter of the main camera is calculated (that is, the rotation angle of the panel is recalculated). After calculation, the fifth shooting parameter includes the third rotation angle and the fourth rotation angle, which does not meet preset condition 2. It means that changing the rotation angle of the panel cannot meet preset condition 1, so it is necessary to change the moving path of the underwater robot so as to meet preset condition 1. Therefore, the second moving path is calculated according to the first relative position of the target organism and the main camera. After the underwater robot moves according to the second moving path, the third shooting parameter of the main camera is calculated, and the third image is obtained based on the third shooting parameter using the main camera. When the third image meets preset condition 1, the fourth shooting parameter of the auxiliary camera is calculated accordingly (that is, the rotation angle of the auxiliary camera is recalculated). Since the panel is connected to the auxiliary camera, after the rotation angle of the panel changes, the viewing angle of the picture taken by the auxiliary camera also changes, so as to ensure that the auxiliary camera can capture the target organism. Thus, by changing the moving path, adjusting the rotation angle of the panel and correspondingly adjusting the rotation angle of the auxiliary camera, the target organism can be located in the central area of ​​the field of view of the main camera shooting picture, and the auxiliary camera can shoot the target organism at the same time.

[0161] Example 3: The first image includes a central area of ​​the field of view, and the first image displays a target organism.

[0162] Step C1, determining whether the first image satisfies preset condition 1.

[0163] Step C2, determine that preset condition 3 is not met (that is, the first distance between the target organism and the main camera is greater than the distance threshold), and determine the second movement path according to the first distance, the second relative position of the target organism relative to the underwater robot and the distance threshold.

[0164] It can be understood that the purpose of determining the second moving path is that after the underwater robot moves along the second moving path, the distance between the main camera and the target organism can be less than or equal to the distance threshold. In a possible implementation, the moving path of the main camera is determined according to the first distance and the distance threshold, and the second moving path of the underwater robot is calibrated according to the coordinate system conversion between the main camera and the underwater robot.

[0165] Step C3, moving along the second moving path.

[0166] After step C3, preset condition 3 is satisfied.

[0167] Step C4, calculating a third shooting parameter according to the first image and a first relative position of the target organism relative to the main camera.

[0168] Specifically, how to calculate the third shooting parameter will be described below in conjunction with the accompanying drawings.

[0169] Step C5: photograph the target organism using the main camera according to the third photographing parameter to obtain a second main image set.

[0170] The second main image set includes multiple frames of third images.

[0171] Step C6: when the third image satisfies the preset condition 1, a fourth shooting parameter is calculated according to the first distance.

[0172] Alternatively, when the third image does not satisfy preset condition 1, reference may be made to example 1 or example 2 above.

[0173] Specifically, how to calculate the fourth shooting parameter will be described below in conjunction with the accompanying drawings.

[0174] Step C7, while using the main camera to shoot the target organism, use multiple auxiliary cameras to shoot the target organism according to the fourth shooting parameter to obtain multiple second auxiliary image sets.

[0175] The second auxiliary image set includes multiple frames of fourth images.

[0176] According to Example 3, when preset condition 1 is met but preset condition 3 is not met, the movement path of the underwater robot is adjusted to meet preset condition 3, and then the rotation angle of the panel is adjusted to meet preset condition 1. Then the rotation angle of the auxiliary camera is adjusted accordingly to achieve that the target organism is located in the central area of ​​the field of view of the main camera shooting picture, and the auxiliary camera can simultaneously shoot the target organism.

[0177] It is understandable that the present application (such as Examples 1 to 3) is committed to satisfying preset condition 1, which can achieve multiple beneficial effects. On the one hand, since users generally pay more attention to the center of the screen, if the target organism is located in the central area of ​​the field of view, the live broadcast experience is good, which can improve the live broadcast retention rate. By controlling the image captured by the main camera to meet preset condition 1, and adjusting the angle of the auxiliary camera, it is beneficial for the image captured by the auxiliary camera to also meet preset condition 1 (for details, please refer to the introduction to the calculation of the fourth shooting parameter below), thereby ensuring that the image transmitted to the live broadcast device meets preset condition 1 as much as possible to improve the live broadcast retention rate.

[0178] Moreover, compared to shooting with a single camera, after the main camera captures the image of the target creature, the auxiliary camera can be rotated to simultaneously obtain images from multiple perspectives. For underwater creatures whose positions and postures are often changing, shooting images from multiple perspectives can capture more valuable images in the same amount of time, improve shooting effects, and also improve live broadcast retention rates. For example, taking fish as an example, including features such as fins, heads, and tails, compared to single-perspective shooting, multi-perspective shooting can increase the possibility of capturing more features of underwater creatures, and also allow users to view underwater creatures more comprehensively, increasing the attractiveness of live broadcasts to users, thereby increasing live broadcast retention rates.

[0179] Combine the following Figures 6 to 9 An example of calculating the third shooting parameter is described.

[0180] Figure 6 A schematic diagram of acquiring a first image based on a first shooting parameter using a main camera provided in an embodiment of the present application.

[0181] like Figure 6 As shown, a first image is acquired by using a main camera based on a first shooting parameter. In the first image, the target organism is obviously not in the central area of ​​the field of view.

[0182] The plane where the first central axis and the second central axis are located is called the third plane, and the plane where the first central axis and the third central axis are located is called the fourth plane. The third shooting parameter includes a third rotation angle and a fourth rotation angle, wherein the fourth rotation angle is the angle at which the panel is rotated based on the third central axis, that is, the angle at which the panel is rotated in the third plane when viewed from the viewing angle 1 along the third central axis; the third rotation angle is the angle at which the panel is rotated based on the second central axis, that is, the angle at which the panel is rotated in the fourth plane when viewed from the viewing angle 2 along the second central axis.

[0183] The following takes the angle of view 1 as an example to introduce the calculation method of the fourth rotation angle. It can be understood that the calculation method of the third rotation angle is similar to it and will not be described in detail. Therefore, according to the calculation method of the fourth rotation angle and the third rotation angle, the calculation method of the third shooting parameter can be determined.

[0184] Figure 7 A schematic diagram of an example of a method for calculating the fourth rotation angle provided in an embodiment of the present application.

[0185] like Figure 7 As shown in (a), panel #1 (represented by solid lines) represents the panel before rotation, and the first central axis #1 and the second central axis #1 are the central axes of panel #1; panel #2 (represented by dotted lines) represents the panel after rotation, and the first central axis #2 and the second central axis #2 are the central axes of panel #2.

[0186] like Figure 7As shown in (b) in Figure 7 Based on (a) in FIG. 1 , the fourth rotation angle is explained. The fourth rotation angle is ∠β in the figure, which can be understood as the angle of rotation of panel #2 compared with panel #1 in the third plane, and can also be understood as the angle between the first central axis #2 and the first central axis #1.

[0187] It can be understood that after the center line of the first field of view intersects with the target organism, the target organism in the first image can be located in the center area of ​​the field of view. Therefore, it can be considered that after the rotation ∠β, the third image obtained by the main camera can meet the preset condition 1.

[0188] Figure 7 The gray triangle in (b) belongs to the third plane. ∠β can be calculated based on the lengths of any two sides of the gray triangle.

[0189] Exemplarily, the positioning module can be used to determine the second relative position of the target organism relative to the underwater robot, thereby calibrating the first relative position of the target organism relative to the main camera based on the second relative position, and determining the third relative position of the target organism relative to the main camera in a third plane.

[0190] For example, the third relative position can be expressed in the form of coordinates. Taking the intersection of the first central axis #1 and the second central axis #1 as the origin, the coordinates are (0, 0), the first central axis #1 as the y-axis, and the second central axis #1 as the x-axis, the coordinates of the underwater organism are (x1, y1). Among them, the coordinates of the underwater organism can be the coordinates of any point of the underwater organism. The gray triangle is a right triangle, the length of the long right-angled side is y1, and the length of the short right-angled side is x1. Taking the determination of ∠β based on two right-angled sides as an example, tanβ=x1 / y1, ∠β=arctan(x1 / y1).

[0191] Combination Figure 7 , step A2, step B5 or step C4 above may specifically include: using the positioning module to determine the second relative position of the target organism relative to the underwater robot, and calibrating the first relative position according to the second relative position; determining the third relative position and the fourth relative position of the target organism relative to the main camera in the third plane and the fourth plane respectively according to the first relative position, wherein the third plane is the plane where the first central axis and the second central axis are located, and the fourth plane is the plane where the first central axis and the third central axis are located; determining the fourth rotation angle according to the third relative position; determining the third rotation angle according to the fourth relative position.

[0192] Accordingly, the above step A3, step B6 or step C5 may specifically include: rotating the panel along the second central axis by a third rotation angle, rotating the panel along the third central axis by a fourth rotation angle, and photographing the target organism using the main camera.

[0193] Optionally, the calculation method of the third shooting parameter also includes a calculation method of a sixth rotation angle based on the first central axis.

[0194] Figure 8 A schematic diagram of the sixth rotation angle provided in an embodiment of the present application.

[0195] like Figure 8 As shown, the panel rotates by ∠γ based on the first central axis, which is the sixth rotation angle.

[0196] Exemplarily, the sixth rotation angle may be determined according to the position of the target organism in the first image. For example, in a scene where the target organism is almost stationary, if the target organism has an angle offset compared to the viewing angle of the first image, the sixth rotation angle may be considered to improve the viewing comfort of the user.

[0197] Fig. 9 A schematic diagram of an example of a first image provided in an embodiment of the present application.

[0198] like Fig. 9 As shown in (a) in FIG. 1 , the first image shows almost still underwater organisms (such as corals), and the intersection of two mutually perpendicular dotted lines is the center point of the first image.

[0199] Since the live broadcast device generally broadcasts live to the user with a square screen, the first image is transmitted to the live broadcast device and played to the user, and the picture viewed by the user is similar to the first image. From the user's viewing angle, the coral is tilted in the live broadcast picture, which does not conform to the user's normal viewing habits. Many users may even need to turn their heads in the direction of the tilted coral, or users who use mobile terminals to watch the live broadcast need to turn the mobile terminal device in the direction of the tilted coral to watch, which is very inconvenient.

[0200] Therefore, by determining the sixth rotation angle according to the position of the target organism in the first image, and rotating the panel according to the sixth rotation angle, the position of the target organism in the first image can be made consistent with the user's perspective, so that the picture played by the live broadcast device can be consistent with the user's perspective, thereby improving the user experience.

[0201] like Fig. 9 As shown in (b) of FIG. 1 , it is the first image after the panel is rotated according to the sixth rotation angle. Fig. 9 The first image shown in (a) can significantly improve the viewing experience of the live broadcast picture and increase the live broadcast retention rate.

[0202] It can be understood that the third rotation angle, the fourth rotation angle and the sixth rotation angle are the angles at which the panel is to be rotated relative to the position before rotation.

[0203] Combine the following Figure 10 to Figure 11 An example of calculating the fourth shooting parameter is described.

[0204] Exemplarily, the above step A4, step B7 or step C6 may specifically include: calculating a fifth rotation angle according to the first distance and the third distance of the first connecting line, the fifth rotation angle being in the second plane, and the fifth rotation angle being the angle between the center line of the second field of view of the auxiliary camera and the panel, rotating multiple auxiliary cameras according to the fifth rotation angle, and using the multiple auxiliary cameras to photograph the target organism.

[0205] Accordingly, the above step A5, step B8 or step C7 may specifically include: rotating the multiple auxiliary cameras according to the fifth rotation angle, and photographing the target organism using the multiple auxiliary cameras.

[0206] Fig.10 A schematic diagram of an example of calculating the fifth rotation angle provided in an embodiment of the present application.

[0207] like Fig.10 As shown in (a), taking the first plane facing the first direction as an example, and taking auxiliary camera 1 among multiple auxiliary cameras as an example, a schematic diagram for calculating the fifth rotation angle is given. The first distance is the distance from the center point of the main camera to the target organism (for example, the distance of any point), the first connecting line is located in the first plane, and the length of the first connecting line is the third distance. Since the center line of the first field of view is perpendicular to the first plane, the first connecting line belongs to the first plane, so the angle between the center line of the first field of view and the first connecting line is a right angle. Thus, the center line of the first field of view, the first connecting line, and the center line of the second field of view form a right triangle. ∠α is the fifth rotation angle. tanα=first distance / third distance, ∠α=arctan(first distance / third distance).

[0208] Understandable, Fig.10 The intersection of the first distance and the third distance shown in (a) is only an example, and the intersection can be within the range where the target organism is located.

[0209] For example, Fig.10 As shown in (b) in the figure, the intersection point can be the center point of the underwater creature. For other contents, please refer to Fig.10 The relevant description of (a) in .

[0210] It should be noted that in a specific implementation, multiple auxiliary cameras correspond to multiple second planes and multiple fifth rotation angles. If the distances between the multiple auxiliary cameras and the main camera are equal, the fifth rotation angle of one of the auxiliary cameras can be calculated, and the other auxiliary cameras can be rotated to the same fifth rotation angle in the corresponding second plane.

[0211] Fig.11A schematic diagram of an example of a fourth image captured by multiple auxiliary cameras provided in an embodiment of the present application.

[0212] like Fig.11 As shown in (a) of FIG. 1 , the image currently captured by the main camera is taken as the third image. The third image satisfies the preset condition 1.

[0213] like Fig.11 As shown in (b) of FIG. 1 , the fourth images captured by the auxiliary cameras 1 to 4 correspond to the third images respectively. Fig.11 It can be seen that by shooting the target organism with multiple auxiliary cameras, images of the target organism from different perspectives can be obtained. Fig.11 In the figure, the number and positions of the auxiliary cameras are only examples, and the four fourth images are only examples and are not limited.

[0214] The above scheme provides images of the target organism from different perspectives to the live broadcast device so that the live broadcast device can provide valuable images to the user, thereby improving the live broadcast retention rate.

[0215] It can be understood that the fifth rotation angle is the angle that the auxiliary camera must meet after the auxiliary camera is rotated. Specifically, the degree of rotation of the auxiliary camera is determined according to the angle 1 between the center line of the second field of view and the panel before rotation and the fifth rotation angle. The angle difference between the angle 1 and the fifth rotation angle is the angle that the auxiliary camera must rotate.

[0216] Fig.12 A structural block diagram of the device 1000 provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0217] Reference Fig.12 In a possible implementation of the embodiment of the present application, the device may specifically include the following modules:

[0218] The transceiver module 1010 is used to obtain an initial image using the main camera; and transmit the second image set to the live broadcast device.

[0219] The processing module 1020 is used to identify whether the initial image shows the target organism; when the initial image shows the target organism, use the main camera and the multiple auxiliary cameras to simultaneously shoot the target organism to obtain a first image set; and filter out a second image set from the first image set.

[0220] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method and system embodiment section and will not be repeated here.

[0221] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0222] like Fig.13 As shown, an embodiment of the present application also provides a device 1100, which includes: at least one processor 1110, a memory 1120, and a computer program 1121 stored in the memory and executable on the at least one processor, and when the processor executes the computer program, the steps in any of the above-mentioned method embodiments are implemented.

[0223] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0224] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, a mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0225] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0226] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0227] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0228] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0229] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0230] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for identifying underwater organisms, characterized in that: Applied to an underwater robot, the underwater robot communicates with a live broadcast device, the underwater robot includes a visual module, the visual module includes a main camera and multiple auxiliary cameras, the method includes: Acquire an initial image using the main camera; identifying whether the initial image shows a target organism; In the case where the initial image shows the target organism, the target organism is photographed by the main camera and the plurality of auxiliary cameras simultaneously to obtain a first image set, wherein the plurality of auxiliary cameras are Z auxiliary cameras, and the first image set includes a plurality of frames of first images photographed by the main camera, and Z groups of a plurality of frames of second images photographed by the plurality of auxiliary cameras, where Z ≥ 2 and Z is an integer; Screening out a second image set from the first image set, wherein the target organism in the second image set has a higher degree of recognition than other image sets, and the other image sets include: images in the first image set other than the second image set; Transmitting the second image set to the live broadcast device; The step of selecting the second image set from the first image set includes: Inputting the multiple frames of first images into a second neural network to obtain first recognition degree information, the first recognition degree information being used to characterize the degree of recognition of the target organism in the multiple frames of first images, the first recognition degree information comprising: at least one of a first number of types of morphological features of the target organism in the multiple frames of first images, a first total number of morphological features of the target organism, or a first area ratio of the morphological features of the target organism in an image showing the target organism; Inputting the Z groups of multiple frames of second images into a second neural network respectively, and obtaining Z groups of second recognition degree information respectively, each group of the second recognition degree information is used to characterize the degree of recognition of the target organism in each group of the multiple frames of second images, and each group of the second recognition degree information includes: in each group of the multiple frames of second images, at least one of the second number of types of morphological features of the target organism, the second total number of morphological features of the target organism, or the second area ratio of the morphological features of the target organism in the image showing the target organism; According to the first recognition degree information and the Z group of second recognition degree information, a plurality of frames of third images are selected from the plurality of frames of first images and the Z group of plurality of frames of second images, wherein the plurality of frames of third images have a higher degree of recognition than other plurality of frames of images, and the other plurality of frames of images are images other than the plurality of frames of third images from the plurality of frames of first images and the Z group of plurality of frames of second images; The image set corresponding to the multiple frames of third images is set as the second image set.

2. The method according to claim 1, characterized in that The identifying whether the initial image shows a target organism comprises: Inputting the initial image into a first neural network model to obtain a biological detection result, wherein the biological detection result indicates whether a target organism exists in the initial image; Wherein, the first neural network model is obtained by training according to a first training data set, the first training data set includes first input data and first target data, the first input data includes multiple frames of first sample images, the multiple frames of first sample images respectively display different sample underwater organisms, the sample underwater organisms display sample morphological features, the first target data includes multiple groups of first sample coordinate data, the multiple frames of first sample images correspond to the multiple groups of first sample coordinate data one by one, and each group of the first sample coordinate data is the pixel coordinate of the sample morphological feature in each frame of the first sample image; Wherein, when the biological detection result includes the first coordinate data corresponding to the initial image, the biological detection result indicates that the target organism exists in the initial image; or, when the biological detection result does not include the coordinate data, the biological detection result indicates that the target organism does not exist in the initial image.

3. The method according to claim 1 or 2, characterized in that The first image set is captured within a first time period, the first image set includes a third image set and Z fourth image sets, wherein the third image set includes X frames of the first images, the X frames of the first images are captured by the main camera within the first time period, each of the Z fourth image sets includes a group of X frames of the second images, the Z fourth image sets are captured by the Z auxiliary cameras within the first time period, the first time period includes X moments, the X frames of the first images correspond to the X moments, each group of X frames of the second images corresponds to the X moments, X≥2, and X is an integer; The step of selecting the second image set from the first image set further includes: acquiring Y frames of the first image and Z groups of Y frames of the second image corresponding to Y moments in the X moments respectively; The step of inputting the multiple frames of the first image into the second neural network to obtain the first recognition degree information includes: inputting Y frames of the first image into the second neural network model to obtain the first recognition degree information, wherein the first recognition degree information is specifically used to characterize the degree of recognition of the target organism in the Y frames of the first image, and the first recognition degree information specifically includes: a first number of types of morphological features of the target organism in the Y frames of the first image, a first total number of morphological features of the target organism, or a first area ratio of the morphological features of the target organism in the image showing the target organism; The step of inputting the Z groups of multiple frames of second images into the second neural network to obtain Z groups of second recognition degree information respectively includes: inputting Z groups of Y frames of the second images into the second neural network model to obtain the Z groups of second recognition degree information respectively, each group of the second recognition degree information is specifically used to characterize the degree of recognition of the target organism in each group of Y frames of the second images, and each group of the second recognition degree information specifically includes: in each group of Y frames of the second images, the second number of types of morphological features of the target organism, the second total number of morphological features of the target organism, or the second area ratio of the morphological features of the target organism in the image showing the target organism; The step of selecting a plurality of frames of third images from the plurality of frames of first images and the plurality of frames of second images in the Z group according to the first recognition degree information and the plurality of frames of second recognition degree information includes: selecting a third image of Y frames from the first image of Y frames and the second image of Y frames in the Z group according to the first recognition degree information and the plurality of frames of second recognition degree information, wherein the third image of Y frames has a higher recognition degree than other images of Y frames, and the other images of Y frames are images of the first image of Y frames and the second image of Y frames in the Z group except the third image of Y frames, where 1≤Y≤X and Y is an integer; The step of setting the image set corresponding to the multiple frames of third images as the second image set includes: setting the image set corresponding to the Y frames of third images as the second image set.

4. The method according to claim 3, characterized in that The second neural network model is trained based on a second training data set, the second training data set includes second input data and second target data, the second input data includes multiple frames of second sample images, the multiple frames of second sample images respectively display different sample underwater organisms, the sample underwater organisms display sample morphological characteristics, the second target data includes multiple groups of sample identification data, the multiple frames of second sample images correspond one-to-one to multiple groups of sample identification degree information, the multiple groups of sample identification degree information include: in the second sample image, the number of types of the sample morphological characteristics, the total number of the sample morphological characteristics and the number of pixels occupied by the sample morphological characteristics.

5. The method according to claim 2 or 4, characterized in that The sample underwater organism is a fish-shaped underwater organism, and the sample morphological features include: fins, tail, trunk and head; Alternatively, the sample underwater organism is an underwater organism with tentacles, and the sample morphological features include: a head and a tentacle; Alternatively, the sample underwater organism is an underwater organism with appendages, and the sample morphological characteristics include: cephalothorax, abdomen and appendages.

6. The method according to claim 1 or 2, characterized in that: The visual module further includes a panel, the main camera is fixedly connected to the panel based on a connecting portion, each of the auxiliary cameras is connected to the panel and can rotate, the multiple auxiliary cameras are distributed around the main camera, the panel is connected to the main body of the underwater robot, and the main camera and the multiple auxiliary cameras face a first direction away from the panel, and the panel can rotate in a three-dimensional space with the center point of the connecting portion as the origin; The acquiring the initial image by using the main camera comprises: acquiring the initial image by using the main camera based on an initial shooting path, wherein the initial shooting path comprises: an initial rotation angle of the panel and an initial movement trajectory of the underwater robot; The method further includes: when the initial image does not show the target organism, changing the initial shooting path, and acquiring a corresponding image using the main camera based on the changed shooting path.

7. The method according to claim 1 or 2, characterized in that: in, The visual module also includes a panel, the main camera is fixedly connected to the panel, the first field of view center line of the main camera is perpendicular to a first plane, the first plane is parallel to the panel, each of the auxiliary cameras is connected to the panel and can rotate, the multiple auxiliary cameras are distributed around the main camera, the panel is connected to the main body of the underwater robot, and the main camera and the multiple auxiliary cameras face a first direction away from the panel, the multiple auxiliary cameras can rotate in a second plane, the second plane is perpendicular to the first plane and intersects with a first connecting line, the first connecting line is a connecting line between the center point of the main camera and the center point of the auxiliary camera, the underwater robot also includes a positioning module, and the method also includes: Using the positioning module to measure a second distance between the target organism and the underwater robot, and calibrating a first distance between the target organism and the main camera according to the second distance; Calculate a rotation angle according to the first distance and the third distance of the first connecting line, wherein the rotation angle is within the second plane and is the angle between the center line of the second field of view of the auxiliary camera and the panel; The plurality of auxiliary cameras are rotated according to the rotation angle.

8. An underwater robot, characterized in that: The invention comprises a visual module, wherein the visual module comprises a panel, a main camera and a plurality of auxiliary cameras, wherein the main camera is fixedly connected to the panel, a first field of view centerline of the main camera is perpendicular to a first plane, the first plane is parallel to the panel, each of the auxiliary cameras is connected to the panel and can rotate, the plurality of auxiliary cameras are distributed around the main camera, the panel is connected to the body of the underwater robot, and the main camera and the plurality of auxiliary cameras face a first direction away from the panel, and the underwater robot is used to perform the steps of the method as claimed in any one of claims 1 to 7.

9. A system for identifying underwater organisms, comprising an underwater robot and a live broadcast device, characterized in that: The underwater robot is connected to a live broadcast device, the underwater robot includes a visual module, the visual module includes a panel, a main camera and multiple auxiliary cameras, wherein the main camera is fixedly connected to the panel, the first field of view centerline of the main camera is perpendicular to a first plane, the first plane is parallel to the panel, each of the auxiliary cameras is connected to the panel and can rotate, the multiple auxiliary cameras are distributed around the main camera, the panel is connected to the main body of the underwater robot, and the main camera and the multiple auxiliary cameras face a first direction away from the panel, the underwater robot is used to perform the steps of the method as described in any one of claims 1 to 7; the live broadcast device is used to receive the second image set from the underwater robot, and play pictures to the user according to the second image set.

Images