Image processing method, device, shooting device and movable platform

Abstract

An image processing method, device, shooting device and movable platform, the method comprising: obtaining a binocular image of a target scene; determining the depth information of the target scene according to the binocular image; obtaining the semantic information and position information of each target in the target scene according to the depth information and a semantic segmentation map of a monocular image in the binocular image. When performing target recognition, the present invention combines the depth information of the target scene with the semantic segmentation map of the monocular image of the target scene, and can more accurately obtain the semantic information and position information of each target in the target scene, realize the distinction between categories with missing distance and similar texture in the target scene, and provide support for building an accurate and practical semantic map; the target recognition method of the present invention is particularly suitable for target scenes with relatively complex backgrounds.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image processing method, device, shooting device and movable platform. Background Art

[0002] In the related art, when identifying a target in a scene, a single image taken for the scene is semantically segmented to obtain a semantic segmentation map, and the target is identified based on the semantic segmentation map. The above target recognition method is difficult to distinguish between categories with missing distances and similar textures in scenes, especially scenes with complex backgrounds, such as distinguishing between grass and the ground, and distinguishing between front and rear vehicles. This is difficult to achieve using the above target recognition method. Summary of the invention

[0003] The invention provides an image processing method, an image processing device, a shooting device and a movable platform.

[0004] Specifically, the present invention is achieved through the following technical solutions:

[0005] According to a first aspect of the present invention, there is provided an image processing method, the method comprising:

[0006] Acquire a binocular image of the target scene;

[0007] Determining depth information of the target scene according to the binocular image;

[0008] Semantic information and position information of each target in the target scene are obtained according to the depth information and a semantic segmentation map of a monocular image in the binocular images.

[0009] According to a second aspect of the present invention, there is provided an image processing apparatus, the apparatus comprising:

[0010] A storage device for storing program instructions;

[0011] One or more processors call program instructions stored in the storage device, and when the program instructions are executed, the one or more processors are individually or collectively configured to:

[0012] Acquire a binocular image of the target scene;

[0013] Determining depth information of the target scene according to the binocular image;

[0014] Semantic information and position information of each target in the target scene are obtained according to the depth information and a semantic segmentation map of a monocular image in the binocular images.

[0015] According to a third aspect of the present invention, there is provided a photographing device, the photographing device comprising:

[0016] An image acquisition module, used to acquire binocular images of the target scene;

[0017] A storage device for storing program instructions;

[0018] One or more processors call program instructions stored in the storage device, and when the program instructions are executed, the one or more processors are individually or collectively configured to:

[0019] Acquire a binocular image of a target scene acquired by the image acquisition module;

[0020] Determining depth information of the target scene according to the binocular image;

[0021] Semantic information and position information of each target in the target scene are obtained according to the depth information and a semantic segmentation map of a monocular image in the binocular images.

[0022] According to a fourth aspect of the present invention, there is provided a movable platform, the movable platform comprising:

[0023] An image acquisition module, used to acquire binocular images of the target scene;

[0024] A storage device for storing program instructions;

[0025] One or more processors call program instructions stored in the storage device, and when the program instructions are executed, the one or more processors are individually or collectively configured to:

[0026] Acquire a binocular image of a target scene acquired by the image acquisition module;

[0027] Determining depth information of the target scene according to the binocular image;

[0028] Semantic information and position information of each target in the target scene are obtained according to the depth information and a semantic segmentation map of a monocular image in the binocular images.

[0029] It can be seen from the technical solutions provided by the above embodiments of the present invention that, when performing target recognition, the present invention combines the depth information of the target scene with the semantic segmentation map of the monocular image of the target scene, so as to more accurately obtain the semantic information and position information of each target in the target scene, and realize the distinction between categories with missing distance and similar texture in the target scene, thereby providing support for constructing accurate and practical semantic maps; the target recognition method of the present invention is particularly suitable for target scenes with relatively complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0031] Figure 1 is a method flow chart of an image processing method in one embodiment of the present invention;

[0032] Figure 2a is a monocular image in a binocular image of a target scene in one embodiment of the present invention;

[0033] Figure 2b for Figure 2a A schematic diagram showing the representation of depth information of the target scene;

[0034] Figure 2c It is a flowchart of a specific implementation method of determining depth information of a target scene based on a binocular image in one embodiment of the present invention;

[0035] Figure 3 It is a flowchart of a specific implementation method of obtaining semantic information and position information of each target in a target scene according to depth information and a semantic segmentation map of a monocular image in a binocular image in one embodiment of the present invention;

[0036] Figure 4a is a schematic diagram of an application scenario of an image processing method in one embodiment of the present invention;

[0037] Figure 4b yes Figure 4a Schematic diagram of the depth map and binocular image of the scene;

[0038] Figure 5a is a schematic diagram of another application scenario of the image processing method in one embodiment of the present invention;

[0039] Figure 5b yes Figure 5a Schematic diagram of the depth map and binocular image of the scene;

[0040] Figure 6a is another application scenario schematic diagram of the image processing method in one embodiment of the present invention;

[0041] Figure 6b yes Figure 6a Schematic diagram of the depth map and binocular image of the scene;

[0042] Figure 7It is a flowchart of another specific implementation method of obtaining semantic information and position information of each target in a target scene according to depth information and a semantic segmentation map of a monocular image in a binocular image in one embodiment of the present invention;

[0043] Figure 8 is a specific method flow chart of an image processing method in one embodiment of the present invention;

[0044] Fig. 9 is a structural block diagram of an image processing device in one embodiment of the present invention;

[0045] Fig.10 is a structural block diagram of a photographing device in one embodiment of the present invention;

[0046] Fig.11 It is a structural schematic diagram of a movable platform in one embodiment of the present invention. DETAILED DESCRIPTION

[0047] Traditional target recognition methods have difficulty distinguishing categories in the scene with missing distances and similar textures, such as grass and ground, front and rear vehicles, adjacent walls, and other categories in the scene.

[0048] When performing target recognition, the present invention combines the depth information of the target scene with the semantic segmentation map of the monocular image of the target scene, which can more accurately obtain the semantic information and position information of each target in the target scene, and realizes the distinction between categories with missing distance and similar texture in the target scene.

[0049] The present invention can provide pixel-level semantic recognition for scenes under the perspective of a mobile platform, and construct a semantic map to provide key strategic semantic category support, such as information on drivable areas, people, and vehicles. Due to the lack of distance information and color information, the semantic segmentation results of a single monocular image in a binocular image are poor. For some difficult-to-distinguish categories, such as grass and ground, and front and rear vehicles, the present invention relies on binocular images. When performing target recognition, it combines the depth information of the target scene with the semantic segmentation map of the monocular image of the target scene, and can more accurately obtain the semantic information and position information of each target in the scene under the perspective of the mobile platform, and realize the distinction of categories with missing distance and similar texture in the scene under the perspective of the mobile platform, providing support for other intelligent functions of the mobile platform.

[0050] The movable platform of the present invention has a shooting function, and the movable platform can be a vehicle, an unmanned aerial vehicle, a handheld gimbal, an unmanned ship, etc. Among them, the vehicle can be an unmanned vehicle, a remote control car, etc., and the unmanned aerial vehicle can be an aerial drone or other unmanned aerial vehicles with a shooting function.

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0053] Figure 1 is a method flow chart of an image processing method in one embodiment of the present invention; see Figure 1 The image processing method of the embodiment of the present invention may include the following steps:

[0054] S101: Acquire a binocular image of a target scene;

[0055] The binocular image acquisition method can be selected as needed. For example, in some embodiments, a binocular camera is used to capture a binocular image of the target scene. The binocular camera can be a binocular camera, which is mounted on a movable platform or can be used directly. Of course, the binocular camera can also be integrated on a movable platform.

[0056] In some embodiments, a monocular camera is used to shoot at different positions to obtain a binocular image of the target scene. The different positions correspond to the shooting positions of the binocular camera. The monocular camera of this embodiment can be a camera of a monocular camera, which can be mounted on a movable platform for use or can be used directly; of course, the above-mentioned monocular camera can also be integrated on a movable platform.

[0057] For binocular images of the same target scene, one or more groups of binocular images of the target scene may be acquired, and one group of binocular images includes two monocular images, namely a left-eye image and a right-eye image.

[0058] S102: Determine depth information of the target scene according to the binocular image;

[0059] Among them, the depth information may include: the relative distance information of each target in the target scene in a preset coordinate system. Optionally, the depth information includes: the distance information of each target in the target scene relative to the shooting device that shoots the target scene, such as the distance of each target relative to the lens, or the distance of each target relative to other positions of the shooting device. The above-mentioned preset coordinate system can be a world coordinate system or a custom coordinate system. It can be understood that in other embodiments, absolute distance information can also be used to represent the depth information.

[0060] Depth information can be presented in the form of feature maps or data. Figure 2a A monocular image in a binocular image of a target scene; Figure 2b for Figure 2a The depth information of the target scene is shown. In this embodiment, a feature map is used to present the depth information of the target scene.

[0061] In the related art, the depth information is determined by using the triangle similarity principle. The calculation process of this method is relatively complicated and takes a long time. In order to reduce the time of determining the depth information, in this embodiment, a deep learning method is used to determine the depth information, such as Figure 2c As shown in FIG, a specific implementation method of determining the depth information of the target scene based on the binocular image. Figure 2c When determining the depth information of the target scene based on the binocular image, the image information of the binocular image is input into the pre-trained first convolutional neural network to determine the depth information of the target scene. The image information includes the color information of each channel of the corresponding monocular image, such as RGB components; in addition, in this embodiment, one or more sets of image information of the binocular image are input into the first convolutional neural network to determine the depth information of the target scene. Moreover, the depth information finally determined can be represented by a feature map, the length and width of the feature map are the same as the length and width of the monocular image in the binocular image.

[0062] The network structure of the first convolutional neural network can be designed as needed. For example, in a feasible implementation, the first convolutional neural network may include multiple first network units connected in sequence, and the first network units are used to extract features from their respective inputs; optionally, the first convolutional neural network includes three first network units connected in sequence, the input of the first first network unit is the image information of the binocular image, the input of the middle first network unit is the output of the first network unit, and the input of the last first network unit is the output of the middle first network unit; optionally, the output of the first first network unit and the output of the middle first network unit are used together as the input of the last first network unit to deepen the depth of the first convolutional neural network.

[0063] The first network unit of this embodiment may include at least one of a convolutional layer, a batch normalization layer, and a nonlinear activation layer, wherein the convolutional layer, the batch normalization layer, and the nonlinear activation layer all select conventional operations, which are not specifically described in this embodiment. Of course, the first network unit may also include other network layers, not limited to convolutional layers, batch normalization layers, and / or nonlinear activation layers.

[0064] Also, see again Figure 2cIn some embodiments, before determining the depth information of the target scene based on the binocular image, the image processing method may further include: preprocessing the binocular image so that the two monocular images constituting the binocular image have the same size and the corresponding image points on the two monocular images match; in some embodiments, before determining the depth information of the target scene based on the binocular image, the image processing method may further include: eliminating the distortion of the two monocular images of the binocular image by binocular correction. Through the above preprocessing, the matching degree of the binocular image is improved, thereby improving the accuracy of the depth information.

[0065] S103: Obtain semantic information and position information of each target in the target scene according to the depth information and a semantic segmentation map of a monocular image in the binocular image.

[0066] At the same time, the depth information of the target scene and the semantic segmentation map of the monocular image of the target scene are combined for target recognition to obtain more accurate semantic information and position information of each target in the target scene, and to distinguish between categories with missing distance and similar texture in the target scene.

[0067] In this embodiment, the semantic information at least includes information for indicating the category of the target, such as information indicating the category of the target, such as whether the target is a vehicle, a pedestrian, a road, or the sky.

[0068] There are many optional implementations of step S103. Figure 3 In some embodiments, the process of obtaining the semantic information and position information of each target in the target scene according to the depth information and the semantic segmentation map of a monocular image in the binocular image may include but is not limited to the following steps:

[0069] S301: performing semantic segmentation on a monocular image in the binocular image to obtain a semantic segmentation map of the monocular image;

[0070] The monocular image can be a left-eye image or a right-eye image. Since the left-eye image is usually used as a reference when capturing binocular images, this embodiment chooses to perform semantic segmentation on the left-eye image to obtain a semantic segmentation map of the left-eye image.

[0071] The existing semantic segmentation algorithm can be used to implement step S301. In this embodiment, a monocular image in the binocular image is input into the second convolutional neural network, and the second convolutional neural network determines the semantic segmentation map of the monocular image according to the preset target classification rules and the image information of the monocular image. For the second convolutional neural network, please refer to the subsequent description.

[0072] S302: Determine semantic information and position information of each target in the target scene according to the depth information and the semantic segmentation map.

[0073] The implementation process of determining the semantic information and position information of each target in the target scene according to the depth information and the semantic segmentation map may include but is not limited to the following steps:

[0074] (1) Distinguish multiple objects in the semantic segmentation map that have the same or similar object categories and are adjacent to each other based on the depth information and the initial semantic information of each object in the semantic segmentation map;

[0075] For multiple objects with the same or similar object categories and adjacent positions in the semantic segmentation map, the initial semantic information may identify these objects as the same object, resulting in inaccurate object recognition. The multiple objects in this step can be the grass and ground in the target scene, the front and rear vehicles, adjacent walls, etc., or multiple objects with the same or similar object categories and adjacent positions.

[0076] (2) Obtain semantic information and boundary information of multiple targets.

[0077] Of course, the position information of each target in the target scene determined based on the depth information and the semantic segmentation map may also be other position information of the corresponding target, not limited to the boundary information.

[0078] In the following embodiments, the depth information is represented based on a depth map.

[0079] The image processing method provided in this embodiment can be applied to semantic segmentation of images with occlusion, such as Figure 4a As shown in the top view, when car A and car B are in the position shown in the figure, the binocular camera (including camera a and camera b) observes from the direction of the arrow that car A is partially blocked by car B. After the binocular camera shoots along the observation angle, the following can be obtained: Figure 4b The following figure shows monocular image a (captured by camera a) and monocular image b (captured by camera b). Based on monocular image a and monocular image b, we can get the following: Figure 4b The depth map shown in the above figure (where different filling patterns represent different depths) can be used to distinguish two vehicles at different distances in front by combining the depth map with the initial semantic information of each target in the semantic segmentation map of the monocular image a, or by combining the depth map with the initial semantic information of each target in the semantic segmentation map of the monocular image b. The type of vehicle (i.e., target category) can also be further distinguished.

[0080] Of course, this embodiment can also distinguish more complex occlusion situations, such as Figure 5a As shown in the top view, from the perspective of the binocular camera in the direction of the arrow, car C blocks part of car B and car A, and car B blocks part of car A. After the binocular camera shoots along the perspective, the following can be obtained: Figure 5bThe following figure shows monocular image a (captured by camera a) and monocular image b (captured by camera b). Based on monocular image a and monocular image b, we can get Figure 5b The depth map shown in the above figure can distinguish different occlusion relationships in front by combining the depth map with the initial semantic information of each target in the semantic segmentation map of the monocular image a, or by combining the depth map with the initial semantic information of each target in the semantic segmentation map of the monocular image b. It can also further distinguish the type of vehicle (that is, the target category).

[0081] The image processing method provided in this embodiment can also be applied to semantic segmentation of images containing objects with similar textures, such as Figure 6a In the top view shown in the figure, there is a corner wall in front of the binocular camera. Wall D is closer to the binocular camera than wall E, and wall D and wall E have similar textures. After the binocular camera shoots along the observation angle, the following can be obtained: Figure 6b The following figure shows monocular image a (captured by camera a) and monocular image b (captured by camera b). Based on monocular image a and monocular image b, we can get Figure 6b The depth map shown in the above figure can distinguish the front-to-back relationship between wall D and wall E by combining the depth map with the initial semantic information of each target in the semantic segmentation map of monocular image a, or by combining the depth map with the initial semantic information of each target in the semantic segmentation map of monocular image b. It can also further distinguish the boundary information as a wall with a corner.

[0082] See also Figure 7 In some embodiments, the process of obtaining the semantic information and position information of each target in the target scene according to the depth information and the semantic segmentation map of a monocular image in the binocular image may include but is not limited to the following steps:

[0083] S701: inputting the depth information and the image information of one monocular image in the binocular image into a pre-trained second convolutional neural network to obtain semantic information and position information of each target in the target scene;

[0084] Among them, the second convolutional neural network is used to determine the semantic segmentation map of the monocular image according to the preset target classification rules and the image information of the monocular image; and obtain the semantic information and position information of each target in the target scene based on the depth information and the semantic segmentation map.

[0085] In this embodiment, in order to achieve target classification more accurately, the image training set used to train the second convolutional neural network includes image training sets of multiple target categories, and each category of image training set includes at least one subcategory image training set; optionally, the target categories include at least two of the following: vehicles, sky, roads, static obstacles, and dynamic obstacles; of course, the target categories are not limited to the above-listed categories, and can also be set to other categories. In addition, the subcategories of vehicles can be specifically divided into cars, trucks, buses, trains, RVs, etc., the subcategories of static obstacles can be specifically divided into buildings, walls, guardrails, telephone poles, traffic lights, traffic signs, etc., and the subcategories of dynamic obstacles can include pedestrians, bicycles, motorcycles, etc.

[0086] The target classification rules of this embodiment correspond to the target categories, that is, the second convolutional neural network can identify targets belonging to the above target categories in the monocular image.

[0087] The network structure of the second convolutional neural network can be designed as needed. For example, in a feasible implementation, the second convolutional neural network includes a plurality of second network units connected in sequence, and the second network units are used to perform target classification on their respective inputs; optionally, the second convolutional neural network includes three second network units connected in sequence, the first second network unit inputs depth information and image information of a monocular image in a binocular image, the middle second network unit inputs the output of the first network unit, and the last second network unit inputs the output of the middle second network unit; optionally, the output of the first second network unit and the output of the middle second network unit are used together as the input of the last second network unit to deepen the depth of the second convolutional neural network.

[0088] The second network layer of this embodiment includes at least one of a convolutional layer, a batch normalization layer, and a nonlinear activation layer, wherein the convolutional layer, the batch normalization layer, and the nonlinear activation layer all select conventional operations, which are not specifically described in this embodiment. Of course, the second network unit may also include other network layers, not limited to convolutional layers, batch normalization layers, and / or nonlinear activation layers.

[0089] In addition, the position information in step S701 may be boundary information of each target in the target scene, or may be other position information of each target in the target scene.

[0090] In some embodiments, both step S301 and step S302 are implemented in the second convolutional neural network.

[0091] See also Figure 8In some embodiments, the image information after binocular image preprocessing is input into a first convolutional neural network to determine the depth information of the target scene; then the depth information of the target scene and the image information of a monocular image in the binocular image are input into a second convolutional neural network to obtain the semantic information and position information of each target in the target scene.

[0092] Optionally, in some embodiments, the semantic information may include: a recognition result and a corresponding recognition confidence, wherein the recognition result is used to indicate information about the category to which the target belongs, and the recognition confidence is used to indicate the accuracy of the recognition result, and the confidence can be used to remove misidentified targets and improve the accuracy of target recognition.

[0093] Further, see Figure 8 In some embodiments, after obtaining the semantic information and location information of each target in the target scene, the image processing method may further include: generating a semantic graph of the target scene according to the recognition result, the corresponding recognition confidence and the location information, so as to intuitively present the target recognition result based on the semantic graph. The implementation process of generating the semantic graph of the target scene according to the recognition result, the corresponding recognition confidence and the location information may include but is not limited to the following steps:

[0094] (1) Determine the target corresponding to the recognition result in the semantic segmentation map based on the recognition result and location information;

[0095] According to the recognition result and the position information, the outline of the target corresponding to the recognition result can be displayed in the semantic segmentation map.

[0096] (3) If the recognition confidence corresponding to the recognition result is greater than a preset confidence threshold, the target corresponding to the recognition result is marked in the semantic segmentation map as a mark of the preset target category of the recognition result.

[0097] The annotated semantic segmentation map is the semantic map of the target scene. In this embodiment, the target category where the target recognition result is located is intuitively presented in the semantic segmentation map through annotation.

[0098] In this embodiment, the label of each target category is pre-set, and the label of the target category can be represented by color, pattern, etc., wherein different target categories have different corresponding labels. Optionally, different target categories have different corresponding colors, for example, the color corresponding to the sky is blue, the color corresponding to the ground is brown, the color corresponding to the grass is green, etc.; Optionally, the colors corresponding to different subcategories under the same target category are the same color, but the colors corresponding to different subcategories under the same target category have different depths.

[0099] In addition, if the recognition confidence corresponding to the recognition result is less than or equal to the preset confidence threshold, it is determined that the recognition result may be misidentified. For the misidentified target, the target category information of the target can be directly ignored to avoid affecting the semantic segmentation result.

[0100] Corresponding to the image processing method of the above embodiment, the embodiment of the present invention also provides an image processing device, see Fig. 9 The image processing device 100 includes: a first storage device 110 and one or more first processors 120.

[0101] Among them, the first storage device 110 is used to store program instructions; one or more first processors 120 call the program instructions stored in the first storage device 110. When the program instructions are executed, the one or more first processors 120 are individually or collectively configured to: obtain a binocular image of the target scene; determine the depth information of the target scene based on the binocular image; obtain semantic information and position information of each target in the target scene based on the depth information and a semantic segmentation map of a monocular image in the binocular image.

[0102] The first processor 120 may implement the present invention. Figure 1 , Figure 2c , Figure 3 , Figure 7 as well as Figure 8 The image processing method of the illustrated embodiment can be described with reference to the image processing method of the above-mentioned embodiment to explain the image processing device 100 of the present embodiment.

[0103] It should be noted that the image processing device 100 of this embodiment can be a device with image processing capabilities such as a computer, or it can be a shooting device with a camera function, such as a camera, a camcorder, a smart phone, a smart terminal, a shooting stabilizer, an unmanned aerial vehicle, etc.

[0104] Corresponding to the image processing method of the above embodiment, the embodiment of the present invention further provides a photographing device, see Fig.10 The shooting device 200 includes: a first image acquisition module 210, a second storage device 220 and one or more second processors 230.

[0105] Among them, the first image acquisition module 210 is used to acquire a binocular image of the target scene; the second storage device 220 is used to store program instructions; one or more second processors 230 call the program instructions stored in the second storage device 220. When the program instructions are executed, the one or more second processors 230 are individually or collectively configured to: obtain a binocular image of the target scene acquired by the first image acquisition module 210; determine the depth information of the target scene based on the binocular image; obtain the semantic information and position information of each target in the target scene based on the depth information and a semantic segmentation map of a monocular image in the binocular image.

[0106] Optionally, the first image acquisition module 210 includes a lens and an imaging sensor matched with the lens, such as an image sensor such as a CCD or CMOS.

[0107] The second processor 230 can implement the present invention. Figure 1 , Figure 2c , Figure 3 , Figure 7 as well as Figure 8 The image processing method of the illustrated embodiment can be described with reference to the image processing method of the above-mentioned embodiment to explain the shooting device 200 of the present embodiment.

[0108] The shooting device 200 can be a camera with a video recording function, a camcorder, a smart phone, a smart terminal, a shooting stabilizer (such as a handheld gimbal), an unmanned aerial vehicle (such as a drone), etc.

[0109] The present invention provides a movable platform. Fig.11 The movable platform 300 includes: a second image acquisition module 310, a third storage device 320 and one or more third processors 330.

[0110] Among them, the second image acquisition module 310 is used to acquire a binocular image of the target scene; the third storage device 320 is used to store program instructions; one or more third processors 330 call the program instructions stored in the third storage device 320. When the program instructions are executed, the one or more third processors 330 are configured individually or collectively to: obtain a binocular image of the target scene acquired by the second image acquisition module 310; determine the depth information of the target scene based on the binocular image; obtain the semantic information and position information of each target in the target scene based on the depth information and a semantic segmentation map of a monocular image in the binocular image.

[0111] The second image acquisition module 310 of this embodiment may be a camera, or a structure with a shooting function formed by a combination of a lens and an imaging sensor (such as CCD, CMOS, etc.).

[0112] The third processor 330 can implement the present invention. Figure 1 , Figure 2c , Figure 3 , Figure 7 as well as Figure 8 The image processing method of the illustrated embodiment can be described with reference to the image processing method of the above-mentioned embodiment to explain the movable platform 300 of this embodiment.

[0113] In a feasible implementation, the movable platform 300 is a drone. It can be understood that the drone is an aerial photography drone, and other drones that do not have a camera function do not belong to the protection subject of this embodiment. The drone can be a multi-rotor drone or a fixed-wing drone, and the embodiment of the present invention does not specifically limit the type of drone. Furthermore, the second image acquisition module 310 can be mounted on a fuselage (not marked) through a gimbal (not marked), and the second image acquisition module 310 is stabilized by the gimbal, wherein the gimbal can be a two-axis gimbal or a three-axis gimbal, and the embodiment of the present invention does not specifically limit this.

[0114] The above-mentioned storage device may include a volatile memory (volatile memory), such as a random-access memory (RAM); the storage device may also include a non-volatile memory (non-volatile memory), such as a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the storage device 110 may also include a combination of the above-mentioned types of memory.

[0115] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0116] In addition, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the image processing method of the above embodiment are implemented.

[0117] The computer-readable storage medium may be the internal storage unit of the pan-tilt head described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of the pan-tilt head, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both the internal storage unit of the pan-tilt head and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the pan-tilt head, and may also be used to temporarily store data that has been output or is to be output.

[0118] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0119] The above disclosure is only part of the embodiments of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a binocular image of the target scene; Determine the depth information of the target scene according to the binocular image; before determining the depth information of the target scene according to the binocular image, the method further includes: preprocessing the binocular image so that the two monocular images constituting the binocular image have the same size; Obtaining semantic information and position information of each target in the target scene according to the depth information and a semantic segmentation map of a monocular image in the binocular image, including: Perform semantic segmentation on a monocular image in the binocular image to obtain a semantic segmentation map of the monocular image; and determine semantic information and position information of each target in the target scene according to the depth information and the semantic segmentation map.

2. The method according to claim 1, characterized in that The depth information includes: relative distance information of each target in the target scene in a preset coordinate system.

3. The method according to claim 2, characterized in that The depth information includes: distance information of each target in the target scene relative to a shooting device that shoots the target scene.

4. The method according to any one of claims 1 to 3, characterized in that: Determining the depth information of the target scene according to the binocular image includes: The image information of the binocular image is input into a pre-trained first convolutional neural network to determine the depth information of the target scene.

5. The method according to claim 4, characterized in that The first convolutional neural network includes a plurality of first network units connected in sequence, and the first network units are used to extract features from respective inputs; The first network unit includes at least one of a convolutional layer, a batch normalization layer, and a non-linear activation layer.

6. The method according to claim 1, characterized in that Determining semantic information and position information of each target in the target scene according to the depth information and the semantic segmentation map includes: Distinguishing multiple targets in the semantic segmentation map that have the same or similar target categories and are adjacent to each other according to the depth information and the initial semantic information of each target in the semantic segmentation map; Semantic information and boundary information of the multiple objects are obtained.

7. The method according to claim 1, characterized in that The obtaining, according to the depth information and a semantic segmentation map of a monocular image in the binocular image, semantic information and position information of each target in the target scene comprises: Inputting the depth information and image information of a monocular image in the binocular image into a pre-trained second convolutional neural network to obtain semantic information and position information of each target in the target scene; Among them, the second convolutional neural network is used to determine the semantic segmentation map of the monocular image according to the preset target classification rules and the image information of the monocular image; and obtain the semantic information and position information of each target in the target scene based on the depth information and the semantic segmentation map.

8. The method according to claim 7, characterized in that The image training set used to train the second convolutional neural network includes image training sets of multiple target categories, and the image training set of each category includes at least one subcategory image training set; The target classification rule corresponds to the target category.

9. The method according to claim 8, characterized in that The target categories include at least two of the following: Vehicles, sky, roads, static obstacles and dynamic obstacles.

10. The method according to claim 7, characterized in that The second convolutional neural network includes a plurality of second network units connected in sequence, and the second network units are used to perform target classification on respective inputs; The second network unit includes at least one of a convolutional layer, a batch normalization layer, and a non-linear activation layer.

11. The method according to claim 1, characterized in that: The semantic information includes: a recognition result and a corresponding recognition confidence.

12. The method according to claim 11, characterized in that After obtaining the semantic information and position information of each target in the target scene, the method further includes: A semantic graph of the target scene is generated according to the recognition result, the corresponding recognition confidence and the location information.

13. The method according to claim 12, characterized in that The generating a semantic graph of the target scene according to the recognition result, the corresponding recognition confidence and the position information includes: Determine, according to the recognition result and the location information, a target corresponding to the recognition result in the semantic segmentation map; If the recognition confidence corresponding to the recognition result is greater than a preset confidence threshold, the target corresponding to the recognition result is marked in the semantic segmentation map as a mark of the preset target category of the recognition result.

14. An image processing device, characterized in that: The device comprises: A storage device for storing program instructions; One or more processors call program instructions stored in the storage device, and when the program instructions are executed, the one or more processors are individually or collectively configured to: Acquire a binocular image of the target scene; Determine the depth information of the target scene according to the binocular image; before determining the depth information of the target scene according to the binocular image, the method further includes: preprocessing the binocular image so that the two monocular images constituting the binocular image have the same size; Obtaining semantic information and position information of each target in the target scene according to the depth information and a semantic segmentation map of a monocular image in the binocular image, including: Perform semantic segmentation on a monocular image in the binocular image to obtain a semantic segmentation map of the monocular image; and determine semantic information and position information of each target in the target scene according to the depth information and the semantic segmentation map.

15. The image processing device according to claim 14, characterized in that: The depth information includes: relative distance information of each target in the target scene in a preset coordinate system.

16. The image processing device according to claim 15, characterized in that: The depth information includes: distance information of each target in the target scene relative to a shooting device that shoots the target scene.

17. The image processing device according to any one of claims 14 to 16, characterized in that: The one or more processors are further configured, individually or collectively, to: The image information of the binocular image is input into a pre-trained first convolutional neural network to determine the depth information of the target scene.

18. The image processing device according to claim 17, characterized in that: The first convolutional neural network includes a plurality of first network units connected in sequence, and the first network units are used to extract features from respective inputs; The first network unit includes at least one of a convolutional layer, a batch normalization layer, and a non-linear activation layer.

19. The image processing device according to claim 14, characterized in that: The one or more processors are further configured, individually or collectively, to: Distinguishing multiple targets in the semantic segmentation map that have the same or similar target categories and are adjacent to each other according to the depth information and the initial semantic information of each target in the semantic segmentation map; Semantic information and boundary information of the multiple objects are obtained.

20. The image processing device according to claim 14, characterized in that: The one or more processors are further configured, individually or collectively, to: Inputting the depth information and image information of a monocular image in the binocular image into a pre-trained second convolutional neural network to obtain semantic information and position information of each target in the target scene; Among them, the second convolutional neural network is used to determine the semantic segmentation map of the monocular image according to the preset target classification rules and the image information of the monocular image; and obtain the semantic information and position information of each target in the target scene based on the depth information and the semantic segmentation map.

21. The image processing device according to claim 20, characterized in that: The image training set used to train the second convolutional neural network includes image training sets of multiple target categories, and the image training set of each category includes at least one subcategory image training set; The target classification rule corresponds to the target category.

22. The image processing device according to claim 21, characterized in that: The target categories include at least two of the following: Vehicles, sky, roads, static obstacles and dynamic obstacles.

23. The image processing device according to claim 20, characterized in that: The second convolutional neural network includes a plurality of second network units connected in sequence, and the second network units are used to perform target classification on respective inputs; The second network unit includes at least one of a convolutional layer, a batch normalization layer, and a non-linear activation layer.

24. The image processing device according to claim 14, characterized in that: The semantic information includes: a recognition result and a corresponding recognition confidence.

25. The image processing device according to claim 24, characterized in that: After obtaining the semantic information and position information of each target in the target scene, the one or more processors are further configured individually or collectively to: A semantic graph of the target scene is generated according to the recognition result, the corresponding recognition confidence and the location information.

26. The image processing device according to claim 25, characterized in that: The one or more processors are further configured, individually or collectively, to: Determine, according to the recognition result and the location information, a target corresponding to the recognition result in the semantic segmentation map; If the recognition confidence corresponding to the recognition result is greater than a preset confidence threshold, the target corresponding to the recognition result is marked in the semantic segmentation map as a mark of the preset target category of the recognition result.

27. A photographing device, characterized in that: The photographing device comprises: An image acquisition module, used to obtain a binocular image of a target scene; A storage device for storing program instructions; One or more processors call program instructions stored in the storage device, and when the program instructions are executed, the one or more processors are individually or collectively configured to implement the method of any one of claims 1-13.

28. A movable platform, characterized in that: The movable platform comprises: An image acquisition module, used to obtain a binocular image of a target scene; A storage device for storing program instructions; One or more processors call program instructions stored in the storage device, and when the program instructions are executed, the one or more processors are individually or collectively configured to implement the method of any one of claims 1-13.

29. The movable platform according to claim 28, characterized in that The movable platform is at least one of an unmanned aerial vehicle and a vehicle.

Images