Image processing method and its device, equipment, storage medium, and computer program

Abstract

An embodiment of the present invention discloses an image processing method and an apparatus, a device, a storage medium, and a program product thereof, wherein the method includes obtaining a placement area and a drivable area within an image to be pasted, placing an object image including the object to be pasted in the placement area to generate a composite image, and generating label information indicating whether the object to be pasted is a falling object based on the positional relationship in the composite image between the drivable area and the placement area.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) The present invention is based on a Chinese patent application filed with the China Patent Office on July 14, 2023, bearing application number 202310868168.3 and entitled "Image processing method and apparatus, devices, storage media, and program products thereof," and claims priority to the Chinese patent application, the entire contents of which are incorporated herein by reference.

[0002] The present invention relates to the field of computer vision technology, and in particular to an image processing method and an apparatus, device, storage medium, and program product thereof. [Background technology]

[0003] In advanced driving assistance systems (ADAS) and autonomous driving, if there are fallen objects or obstacles on the road, it is necessary to avoid the objects or obstacles. However, the number of samples that can be collected from fallen objects is small and diverse, making it difficult to learn about fallen objects. Related technology detects fallen objects by training a falling object detection model using a small amount of fallen object data and generalizing the model, but the performance of the trained falling object detection model is relatively low. Summary of the Invention [Problem to be solved by the invention]

[0004] In view of this, embodiments of the present invention provide at least an image processing method, an apparatus, a device, a storage medium, and a program product therefor. [Means for solving the problem]

[0005] The technical solutions of the embodiments of the present invention are realized as follows:

[0006] In one aspect, embodiments of the present invention provide a method of image processing, the method comprising: Acquiring a placement area and a drivable area within the image to be pasted; arranging an object image including a pasting object in the arrangement area to generate a composite image; and generating label information indicating whether the object to be attached is a fallen object based on a positional relationship between the travelable area and the placement area in the composite image.

[0007] In another aspect, embodiments of the present invention provide an image processing apparatus, the apparatus comprising: a first acquisition module configured to acquire a placement area and a drivable area within the pasting target image; a first generation module configured to place an object image including a pasting object in the placement area to generate a composite image; and a second generation module configured to generate label information indicating whether the object to be attached is a falling object or not, based on the positional relationship in the composite image between the travelable area and the placement area.

[0008] In another aspect, an embodiment of the present invention provides a computing device comprising a memory and a processor, wherein the memory is configured to store a computer program executable by the processor, and the processor is configured to execute the program to perform some or all of the steps of the above method.

[0009] In another aspect, an embodiment of the present invention provides a computer-readable storage medium having stored thereon a computer program that causes a processor to perform some or all of the steps of the above-described method.

[0010] In another aspect, embodiments of the present invention provide a computer program comprising computer readable code which, when executed on a computing device, causes a processor of the computing device to perform some or all of the steps of the above-described methods.

[0011] In another aspect, embodiments of the present invention provide a computer program product comprising a computer program or instructions, said computer program or instructions causing a processor to perform some or all of the steps of the above-described methods. [Effects of the Invention]

[0012] In an embodiment of the present invention, first, a placement area for placing an object to be pasted and a drivable area within the object to be pasted are obtained from the object to be pasted image, then an object image including the object to be pasted is placed in the placement area to obtain a composite image, and in this way, the placement area within the object to be pasted is replaced with the object image, and the composite image includes the object image and the drivable area. Finally, label information indicating whether the object to be pasted is a falling object is generated based on the positional relationship between the drivable area and the placement area, and by analyzing the positional relationship between the drivable area and the placement area in this way, the positional relationship between the position of the object image in the composite image and the drivable area can be determined, thereby making it possible to determine whether the object to be pasted is within the drivable area and to generate label information indicating whether the object to be pasted is a falling object. In this way, by generating a synthetic image including the label information, the synthetic image can be used as falling object sample data, and in the process of generating the synthetic image, label information can be generated by analyzing the positional relationship between the drivable area and the placement area.As a result, when generating falling object sample data, a large amount of diverse falling object sample data can be obtained regardless of the object to be pasted, and further, sufficient falling object sample data can be provided for the learning process of the falling object detection model.

[0013] It is to be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory and are not intended to limit the technical solutions of the present invention. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a schematic diagram of an implementation process of an image processing method according to an embodiment of the present invention; [Figure 2] FIG. 4 is a schematic diagram of another implementation process of an image processing method according to an embodiment of the present invention; [Figure 3] FIG. 10 is a schematic diagram of yet another implementation process of an image processing method according to an embodiment of the present invention; [Figure 4A] 1 is a schematic illustration of an application scenario of an image processing method according to an embodiment of the present invention; [Figure 4B] 2 is a schematic diagram of another application scenario of the image processing method according to an embodiment of the present invention; [Figure 5] 3 is a schematic diagram of yet another application scenario of the image processing method according to an embodiment of the present invention; [Figure 6] FIG. 10 is a schematic diagram of yet another implementation process of an image processing method according to an embodiment of the present invention; [Figure 7] 1 is a schematic diagram of an implementation framework for an image processing method according to an embodiment of the present invention; [Figure 8] 1 is an exemplary structural diagram of an image processing device according to an embodiment of the present invention; [Figure 9] 1 is a schematic diagram of hardware entities of a computer device according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0015] The drawings herein, which are incorporated in and constitute a part of this specification, illustrate embodiments according to the present invention and are used together with this specification to explain the technical solutions of the embodiments of the present invention.

[0016] In order to make the objectives, technical solutions and advantages of the present invention clearer, the following will further describe in detail the technical solutions of the present invention with reference to drawings and examples, and the described examples should not be regarded as limiting the present invention, and all other examples that can be obtained by those skilled in the art without paying any creative effort shall fall within the protection scope of the present invention.

[0017] In the following description, the term "some embodiments" refers to a subset of all possible embodiments, and it will be understood that the terms "some embodiments" may refer to the same or different subsets of all possible embodiments, which may be combined with each other where there is no contradiction.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. The terms used herein are only for the purpose of describing embodiments of the present invention and are not intended to limit the present invention.

[0019] Before describing the embodiments of the present invention in more detail, the nouns and terms referred to in the embodiments of the present invention will first be explained. The nouns and terms referred to in the embodiments of the present invention are interpreted as follows.

[0020] 1) Computer vision refers to machine vision, which uses cameras and computers instead of human eyes to identify, track, and measure targets, and then performs graphics processing to generate images suitable for observation by the human eye or detection by a machine.

[0021] 2) Deep learning aims to reduce the amount of calculation required for a model, reduce the number / volume of parameters in the model, and shorten the inference time of the model.

[0022] An embodiment of the present invention provides an image processing method, which can be executed by a processor of a computer device. Here, the computer device may refer to a device with data processing capabilities, such as a server, a notebook computer, a tablet computer, a desktop computer, a smart TV, or a mobile device (e.g., a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, or a portable game console). Figure 1 is a schematic diagram of the process of implementing the image processing method according to the embodiment of the present invention. As shown in Figure 1, the method includes the following steps S101 to S103.

[0023] In step S101, the placement area and the drivable area in the image to be pasted are obtained.

[0024] In some embodiments, the target image may be an image with complex image content or an image with simple image content, the captured target image may be a two-dimensional image or a three-dimensional image, or the target image may be an image including a traffic scene.

[0025] In some possible implementations, the target image may be acquired by collecting images of a traffic scene using an image collection device, or may be received from another device as a target image of a traffic scene. Alternatively, the target image may be randomly selected from an image database, such as an image collected from a traffic road scene or a parking lot. The placement area within the target image may be any area within the target image, and the placement area may be within or outside the drivable area. The placement areas of different target images may be located at the same position in the image or at different positions. The placement area within the target image may be represented as a binary image having the same image size as the target image, a black background, and a white placement area.

[0026] The drivable area within the image to be pasted refers to the area where vehicles can pass, and includes areas where vehicles can pass, such as the road surface area on the road within the image to be pasted, parking lots, underground parking lots, etc. The drivable area within the image to be pasted can be expressed as a binary image with the same image size as the image to be pasted, with a black background and a white drivable area.

[0027] In one specific example, the target image is an image collected from a traffic road scene, and the drivable area is the area where the road surface is located in the image. The placement area may be any area in the image, or may be an area on or off the road surface, such as a green belt on the road surface, a building in the air or on the roadside, etc.

[0028] In step S102, an object image including an object to be pasted is placed in the placement area to generate a composite image.

[0029] In some embodiments, the attachment object may be a two-dimensional object or a three-dimensional object. The attachment object may be any type of object, such as an animal, a plant, or other type. Here, the animal may be a dog, a cat, a sheep, or the like, the plant may be a tree, a flower, or the like, and the other type may be a relatively large object that may pose a hazard to a vehicle. The object image including the attachment object may be an image of the attachment object itself, or an image including the attachment object image and a background. The object image may be obtained by collecting images of the attachment object using an image collection device, may be arbitrarily selected from an image database of images including the attachment object, or may be an object image received from another device. The object image may include a two-dimensional object image or a three-dimensional object image, and the attachment image may include a two-dimensional attachment image or a three-dimensional attachment image. In this way, both the object image and the attachment image include two-dimensional images and three-dimensional images, thereby enabling a greater variety of composite images to be generated using the object image and the attachment image.

[0030] In some possible implementations, the composite image is obtained by pasting the object image onto the placement area to cover the placement area, so that the composite image includes the object image and the drivable area.

[0031] In step S103, label information indicating whether the object to be attached is a fallen object is generated based on the positional relationship between the travelable area and the placement area.

[0032] In some embodiments, the positional relationship between the drivable area and the placement area represents the relationship between the drivable area and the placement area. The label information represents whether the attached object is a fallen object or not, and includes a fallen object label and a non-falling object label. The label information can be represented by any type of identifier, for example, 0 represents a non-falling object label and 1 represents a falling object label, or it can be represented by pinyin or alphabet.

[0033] If the placement area belongs to the drivable area, it indicates that the placement area is within the range of the drivable area, that is, it indicates that the object image is included within the range of the drivable area, thereby indicating that the pasted object is located within the range of the drivable area and affects the vehicle's driving, and based on this, it is determined that the pasted object is a fallen object, and label information indicating that the pasted object is a fallen object is generated.

[0034] If the placement area does not belong to the drivable area, it indicates that the placement area is outside the drivable area, thereby indicating that the object image is outside the drivable area, thereby indicating that the attached object is not included within the drivable area and will not affect the vehicle's driving. Based on this, it is determined that the attached object is not a fallen object, and label information indicating that the attached object is not a fallen object is generated. In this way, by annotating the label information to the composite image, a wide variety of sample images annotated with fallen or non-falling objects can be obtained.

[0035] In an embodiment of the present invention, by replacing the placement area in the image to be pasted with an object image, the composite image can include an object image and a drivable area. By analyzing the positional relationship between the drivable area and the placement area, the positional relationship between the position of the object image in the composite image and the drivable area can be determined. By analyzing the positional relationship between the drivable area and the placement area in this way, label information can be generated. This allows for a large amount of diverse falling object sample data to be obtained when generating falling object sample data, regardless of the object to be pasted. Furthermore, sufficient falling object sample data can be provided for the learning process of the falling object detection model, thereby improving the accuracy of the falling object detection model.

[0036] In some embodiments, the object image is superimposed on the placement area to obtain a composite image, that is, the above step S102 can be realized by the following process: The object image is overlaid on the placement area to obtain the composite image.

[0037] In some possible implementations, the object image is pasted into the placement area to obtain the composite image. The area of the placement area is equal to or greater than the area of the object image, and the object image is pasted into the placement area so as to cover a portion of the placement area. Alternatively, the composite image can be obtained by replacing an area in the placement area that has the same area as the object image with the object image. By overlaying the object image on at least a portion of the placement area in this way, the obtained composite image can completely display the pasted object and the drivable area, making it easier to analyze whether the pasted object is located in the drivable area.

[0038] In some embodiments, the label information includes a fallen object label and a non-falling object label. If the placement area is within the drivable area, the generated label information is a fallen object label and is annotated using the fallen object label. If the placement area is not within the drivable area, the generated label information is a non-falling object label and is annotated using the non-falling object label. That is, the above step S103 can be realized by the steps shown in FIG. 2.

[0039] In step S201, if the placement area is within the travelable area, the object to be attached is determined to be a fallen object, and the fallen object label is generated.

[0040] In some embodiments, if the placement area is within the drivable area, it indicates that the object image is pasted within the drivable area, indicating that the pasted object is located within the drivable area and affects vehicle movement, thereby determining that the pasted object is a fallen object affecting traffic. Based on this, a fallen object label is generated indicating that the pasted object is a fallen object. For example, if the placement area is located on a road surface within the drivable area, i.e., if the pasted object falls onto the road surface, the pasted object is a fallen object on the road surface, and a fallen object label is generated for the pasted object. In the process of determining whether the pasted object is a fallen object, fallen object training is performed using actually placed pasted objects, so the characteristics of the object are already taken into account in the placement process when learning about fallen objects. Because the detection is performed taking into account the characteristics of fallen objects, areas where segmentation recognition is uncertain are not determined to be fallen objects.

[0041] In step S202, if the placement area is outside the drivable area, the object to be pasted in the composite image is determined to be a non-falling object, and the non-falling object label is generated.

[0042] In some embodiments, if the placement area is outside the drivable area, it indicates that the object image is pasted outside the drivable area, indicating that the pasted object is located outside the drivable area and does not affect the vehicle's driving, and further indicating that the pasted object is not a fallen object. Based on this, a non-falling object label indicating that the pasted object is not a fallen object is generated.

[0043] In step S203, the object to be pasted is annotated in the composite image using the label information.

[0044] In some embodiments, after determining whether the attached object is a falling object, the attached object is annotated using the generated label information to obtain an annotated composite image including the label information. In this manner, the annotated composite image becomes fallen object sample data annotated with the label information. If the attached object is a falling object, the attached object is annotated with a falling object label, and by annotating the attached object with the falling object label, positive example sample data of the fallen object annotated with the falling object label can be obtained. If the attached object is a non-falling object, the attached object is annotated with a non-falling object label. By annotating the attached object with the non-falling object label, negative example sample data of the fallen object annotated with the non-falling object label can be obtained.

[0045] In an embodiment of the present invention, if the object to be attached is within the drivable area, a falling object label indicating that the object is a falling object is generated. The falling object label is then used to annotate the object as a falling object in the composite image, thereby obtaining a composite image that can be used as positive sample data for falling objects. If the object to be attached is outside the drivable area, a non-falling object label indicating that the object is not a falling object is generated. The non-falling object label is then used to annotate the object as a falling object in the composite image, thereby obtaining a composite image that can be used as negative sample data for falling objects. In this way, a large amount of diverse falling object sample data can be obtained using the annotated positive sample data and negative sample data for falling objects. Furthermore, whether the object to be attached is a falling object is determined by determining whether it is within the drivable area. In this way, whether the object to be attached is a falling object does not depend on the properties of the object to be attached itself, so relatively stable falling object sample data can be obtained.

[0046] In some possible implementations, annotation of a pasted object can be achieved in several ways:

[0047] In method 1, segment annotation is performed on the object to be pasted in the composite image using the label information.

[0048] In some embodiments, segment annotation is used to annotate the pasted object for each segment of the segmented pasted object. In the composite image, the pasted object is divided into multiple segments based on the overlapping relationship between the location of the pasted object and the drivable area to obtain segmented pasted objects, and the label information can be used to annotate the pasted object for each segment of the segmented pasted object. For example, if a portion of the pasted object is within the drivable area and another portion is outside the drivable area, the pasted object can be divided into two segments, one segment can be annotated as a falling object, and the other segment can be annotated as a non-falling object.

[0049] In Method 2, the object to be pasted is annotated with a bounding box in the composite image using the label information.

[0050] In some embodiments, the bounding box may be a rectangular box that can enclose the object to be pasted, and the rectangular box is used to annotate the object to be pasted based on the label information to obtain the object to be pasted annotated with the label information in the rectangular box, thereby obtaining an annotated composite image that can be used as falling object sample data.

[0051] In method 3, the object image is annotated in the composite image using the label information.

[0052] In some embodiments, the label information is used to annotate the entire object image in the placement region within the composite image, thereby obtaining an object image annotated with the label information, thereby obtaining an annotated composite image that can be used as falling object sample data. In this way, by annotating whether the pasted object is a falling object using one or more of a plurality of methods, it is possible to accurately and quickly annotate whether the pasted object is a positive example sample or a negative example sample, thereby facilitating classification of the falling object sample data.

[0053] In some embodiments, the object image including the pasted object can be realized in several ways:

[0054] In method 1, the object image including the pasted object is selected from the object image library.

[0055] In some embodiments, the object image library includes images containing any type of attached object. The images in the object image library may be two-dimensional images or three-dimensional images. By selecting any one frame of an image from the object image library as the object image, the resulting falling object sample data based on the selected object image can have a greater variety.

[0056] In method 2, the object image is generated based on the pasted object.

[0057] In some embodiments, the object image is obtained by performing an image acquisition on the pasted object, or by scanning the pasted object, or by drawing the pasted object.

[0058] In the above-mentioned methods 1 and 2, an object image including any type of pasted object can be obtained by selecting an object image from an object image library or generating an object image, thereby diversifying the types of object images, and thus obtaining a wide variety of falling object sample data based on the diverse types of object images.

[0059] In some embodiments, the drivable area in the image to be pasted can be obtained in the following ways.

[0060] In method 1, the drivable area is determined from the image to be pasted based on input instruction information.

[0061] In some embodiments, the instruction information is used to indicate the location of the drivable area in the target image, and the instruction information may be input autonomously by a user, i.e., the user can manually select the drivable area in the target image. For example, the user can manually draw the drivable area on the target image to enclose the drivable area. Alternatively, the user can input the instruction information using a button to draw the drivable area based on the location of the drivable area in the instruction information.

[0062] In method 2, area estimation is performed on the image to be pasted based on the image content of the image to be pasted, and the drivable area is determined.

[0063] In some embodiments, the image content of the target image includes the background of the target image and various objects in the foreground. By recognizing various objects in the foreground, a vehicle-passable area is obtained. For example, by dividing the foreground into roads, garages, parking lots, and other areas that the vehicle can pass through, a drivable area is obtained.

[0064] In method 3, the drivable area is determined based on the annotated information in the image to be pasted.

[0065] In some embodiments, the annotated information in the image to be pasted can represent the type of object included in each region of the image to be pasted, for example, if the image to be pasted includes a building, a road, and a green plant, the annotated information can include a building label, a road label, a green plant label, etc. This makes it possible to know which regions are drivable regions based on the annotated information, thereby accurately dividing the drivable regions in the image to be pasted.

[0066] Methods 1 to 3 described above allow the drivable area to be selected manually or automatically, and furthermore, the drivable area can be determined by area estimation, resulting in a wider and more accurate overlap range of the drivable area obtained within the image to be pasted.

[0067] In some embodiments, after generating a composite image annotated with label information indicating whether the object to be attached is a falling object, i.e., after step S103, the annotated composite image is used as falling object sample data to train a falling object detection model, thereby obtaining a trained falling object detection module. This can be achieved by the steps shown in FIG. 3.

[0068] In step S301, a composite image is obtained in which an annotation is made as to whether the object to be pasted is a fallen object or not.

[0069] In some embodiments, whether the object to be pasted is a fallen object depends on whether the object to be pasted is located within the drivable area. Therefore, even if the object is the same, the corresponding label information may differ depending on the placement position. For example, if the object to be pasted is a dog, and the image of the dog is located within the drivable area in the image to be pasted in the first frame, and the image of the dog is located outside the drivable area in the image to be pasted in the second frame, in the resulting two-frame composite image, the label of the dog in the composite image in the first frame is a fallen object label, and the label of the dog in the composite image in the second frame is a non-falling object label. In this way, whether the object to be pasted is a fallen object is determined based on the position where the object image fell in the image to be pasted, regardless of the attributes of the generated fallen object itself, thereby improving the stability of the resulting annotated composite image.

[0070] In step S302, the composite image annotated as to whether the pasted object is a falling object or not is used as falling object sample data to train a falling object detection model waiting to be trained, thereby obtaining a trained falling object detection model.

[0071] In some embodiments, the annotated synthetic images can be used as falling object sample data to provide a large amount of sample data containing a variety of falling objects to a waiting falling object detection model, which can then be used to train the waiting falling object detection model and improve the generalization performance of the trained falling object detection model.

[0072] In some embodiments, the falling object sample data is divided into positive sample data and negative sample data, and the falling object detection model waiting to be trained is trained using a contrastive learning method, that is, the above step S302 can be realized by the following steps S321 to S323 (not shown).

[0073] In step S321, feature extraction is performed on the multi-frame falling object sample data to obtain an image feature set.

[0074] In some embodiments, the image feature set is obtained by performing feature extraction on the falling object sample data of multiple frames using a feature extractor to obtain image features of the falling object sample data of each frame, wherein the image features of the falling object sample data of each frame include label information indicating whether the sample data is a falling object.

[0075] In step S322, based on the label information of the multi-frame falling object sample data, image features in the image feature set that are annotated with falling object labels are set as positive example samples, and image features that are annotated with non-falling object labels are set as negative example samples.

[0076] In some embodiments, it is possible to determine whether the attached object in the falling object sample data is a falling object based on the label information. In the image feature set of the falling object sample data, image features having a falling object label are classified as positive example samples based on the annotated label information, and image features having a non-falling object label are classified as negative example samples, thereby classifying the image feature set into two types of samples, positive example samples and negative example samples, and subjecting them to comparative training in a falling object detection model awaiting training.

[0077] In step S323, the falling object detection model waiting to be trained is trained based on the image features of the positive sample and the image features of the negative sample, thereby obtaining the trained falling object detection model.

[0078] In some embodiments, the image features of the positive example samples and the image features of the negative example samples are used to provide sample data with clear boundaries to the falling object detection model waiting to be trained, thereby adjusting the network parameters of the falling object detection model waiting to be trained according to the image features of the positive example samples and the image features of the negative example samples, thereby obtaining a trained falling object detection model with high detection accuracy.

[0079] In an embodiment of the present invention, by dividing the image feature set of falling object sample data into positive example samples and negative example samples, the boundary between falling objects and non-falling objects can be made clearer by contrasting the positive example samples with the negative example samples in the learning process of a falling object detection model waiting to be trained, thereby making the detection results of the trained falling object detection model more accurate.

[0080] In some embodiments, the distance between image features with the same label is decreased and the distance between image features with different labels is increased to clarify the boundary between positive and negative example samples, i.e., the above step S323 can be realized by the following steps:

[0081] In the first step, the distance between image features having the same label information among the image features of the positive example samples and the image features of the negative example samples is decreased, and the distance between image features having different label information is increased, thereby obtaining adjusted image features of the positive example samples and adjusted image features of the negative example samples.

[0082] In some embodiments, among the image features of the positive example samples and the image features of the negative example samples, the distance between the image features belonging to the positive example samples and the distance between the image features belonging to the negative example samples are decreased, respectively, and the distance between the image features belonging to the positive example samples and the image features belonging to the negative example samples is increased, thereby obtaining adjusted image features of the positive example samples and adjusted image features of the negative example samples.

[0083] In a second step, the network parameters of the falling object detection model waiting to be trained are adjusted based on the adjusted image features of the positive example samples and the adjusted image features of the negative example samples to obtain the trained falling object detection model.

[0084] In some embodiments, the adjusted image features of the positive example samples and the adjusted image features of the negative example samples obtained after adjusting the distance between the features are used to adjust the network parameters of the falling object detection model to be trained, thereby performing a training process for the falling object detection model to be trained and obtaining a trained falling object detection model. In this way, by reducing the distance between features with the same label, the distance between features with the same label becomes closer, and by increasing the distance between features with different labels, the distance between features with different labels becomes farther, thereby allowing the falling object detection model to more accurately train the falling object detection model to classify tagged objects with different labels more accurately.

[0085] In the following, the application of the image processing method according to the embodiment of the present invention to a real scenario will be described, taking the generation of falling object samples in an autonomous driving scenario as an example.

[0086] Related technologies detect falling objects by training a falling object detection model with a small amount of falling object data and generalizing the model. However, it is difficult to generalize falling objects that cannot be defined as unique objects, and it is difficult to train a model for falling objects with a small amount of data. Furthermore, segmentation models are prone to false positives for falling object regions, and methods that detect falling objects by estimating regions where segmentation fails have low accuracy. However, these methods require collaboration with the segmentation model and do not take into account the object-like nature of the falling object, and may simply estimate regions with low segmentation reliability.

[0087] In an embodiment of the present invention, the problem of a lack of falling object data is solved by generating falling object data, and by learning the generated falling objects, it becomes possible to detect falling objects taking into account the object-likeness of the falling object.

[0088] In some embodiments, in falling object learning, if an object is placed on the road surface, the object is a fallen object, and if the object is placed off the road surface, the object is not a fallen object. As shown in FIG. 4A, dog 41 is on the road surface in FIG. 4A, so dog 41 is determined to be a fallen object. In FIG. 4B, dog 42 is not off the road surface, so dog 42 is determined to not be a fallen object. A fallen object detection model is trained by generating training data of the falling object to be trained, and a trained model is obtained.

[0089] In some possible implementations, based on the assumption that detected fallen objects are on the road surface, objects placed on the road surface are learned as fallen objects, and objects placed off the road surface are learned as non-falling objects. Even if an object placed on the road surface and an object placed off the road surface are the same object, the object placed off the road surface is determined to be not a fallen object, thereby preventing the object itself from being learned as a fallen object. Regardless of the type of object, if it is a fallen object, these features are brought closer together. Conversely, even if it is the same object, if it is not a fallen object, these features are moved farther apart. For example, fallen objects and non-falling objects are separated by increasing the distance between positive and negative samples. As shown in Figure 5, dog 51 is determined to be a fallen object because it is on the road surface, dog 52 is determined to be a non-falling object because it is off the road surface, sphere 53 is determined to be a fallen object because it is on the road surface, and sphere 54 is determined to be a non-falling object because it is off the road surface. Dog 51 and dog 52 are the same object, but have different label information; one is a fallen object and the other is a non-falling object, so the features corresponding to dog 51 and dog 52 are moved away from each other. Sphere 53 and sphere 54 are the same object, but one is a fallen object and the other is a non-falling object, so the features corresponding to sphere 53 and sphere 54 are moved away from each other. The features corresponding to dog 51 and sphere 53 are moved closer to each other, and the features corresponding to dog 52 and sphere 54 are moved closer to each other. In this embodiment of the present invention, since fallen objects themselves can be learned directly, there is no need for collaboration with a segmentation model, and since objects actually placed on the road surface are learned as fallen objects, detection that takes into account object-likeliness becomes possible.

[0090] FIG. 6 is a schematic diagram of another implementation process of an image processing method according to an embodiment of the present invention, and the following description will be made with reference to the steps shown in FIG.

[0091] In step S601, the process of generating falling object sample data begins.

[0092] In step S602, an image to be pasted is selected from the image database.

[0093] In some possible implementations, the image database includes background images to be stitched together, for example, driving data image sets such as CityScapes and BDD100K.

[0094] In step S603, an object image including the pasted object is extracted from the object image library.

[0095] In some possible implementations, the object database includes an image set of various objects to be stitched, such as COCO and VOC detection target data. The stitched objects are trained as candidates for "falling objects." The stitched objects are determined to be falling objects or not depending on their stitching positions.

[0096] In step S604, the placement position of the object to be pasted is determined in the image to be pasted, and pasting is performed.

[0097] In step S605, it is determined whether the object to be pasted is pasted on the road.

[0098] In some possible implementations, if the pasted object is on the road, proceed to step S606, and if the pasted object is not on the road, proceed to step S608.

[0099] In step S606, if the pasted object is on a road, the pasted object is annotated as a positive sample.

[0100] In some possible implementations, if the pasted object is pasted on a road, it can be determined that a fallen object has fallen on the road, so the pasted object is annotated as a positive example sample.

[0101] In step S607, the object to be pasted is detected as a fallen object.

[0102] In step S608, if the pasted object is not on the road, the pasted object is annotated as a negative sample.

[0103] In some possible implementations, if the pasted object is pasted in a position outside the road, such as in the sky, it is determined that it is unlikely to be a fallen object, and the pasted object is annotated as a negative example sample.

[0104] In step S609, it is determined that the object to be detected is not a falling object.

[0105] In an embodiment of the present invention, data on falling objects is not required during learning. By changing the labels of the same object as falling or non-falling depending on its location, it becomes possible to detect falling objects based on their location, regardless of the attributes of the falling objects themselves generated during learning.

[0106] In some possible implementations, steps S601 to S609 can be implemented using the framework diagram shown in FIG. 7 , which is a schematic diagram of an implementation framework for an image processing method according to an embodiment of the present invention. In FIG. 7 , image 71 is an image to be pasted, and object 72 is an object to be pasted. Object 72 may be a 2D image, a 3D image, or a generated object. Four different placement areas are set in image 71, such as four placement areas 73 corresponding to image 71, and a drivable area 74 is extracted from image 71. Object 72 is placed in each of the four placement areas to obtain composite images 701, 702, 703, and 704. Here, in composite images 701 and 702, the pasted object is located within drivable area 74 and is a positive example sample. In composite images 703 and 704, the pasted object is located outside drivable area 74 and is a negative example sample. Then, a feature extractor 705 is used to extract features from the composite images 701, 702, 703, and 704. A falling object detection head 706 is used to perform falling object detection 77 on the extracted features to detect whether the extracted features contain falling objects. A contrast learning head 707 is used to classify the extracted feature vectors, classifying the feature vectors into falling object vectors 75 and non-falling object vectors 76. Here, the distance between vectors of the same type (e.g., falling objects or non-falling objects) is decreased, and the distance between vectors of different types (e.g., falling objects and non-falling objects) is increased.

[0107] In an embodiment of the present invention, there is no need to collect actual falling object sample data. Even for the same object, different placement areas generate different label information. By analyzing based on the position of the drivable area and placement area in the image, falling object detection can be performed without relying on the attributes of the falling object itself. Furthermore, this method can train a falling object detection model independent of segmentation, and is therefore not affected by the segmentation model. Users can create a falling object detection model without any falling object data, thereby enabling stable falling object detection that is not dependent on any specific object, thereby improving the accuracy of falling object detection using a trained falling object detection model.

[0108] Based on the above-described embodiments, an embodiment of the present invention provides an image processing device, and each unit included in the device and each module included in each unit can be implemented by a processor in a computer device, and of course, can also be implemented by a specific logic circuit. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0109] FIG. 8 is an exemplary structural diagram of an image processing device according to an embodiment of the present invention. As shown in FIG. 8, the image processing device 800 includes: A first acquisition module 801 configured to acquire a placement area and a drivable area within the image to be pasted; a first generating module 802 configured to place an object image including a pasting object in the placement area to generate a composite image; and a second generation module 803 configured to generate label information indicating whether the object to be attached is a falling object or not, based on the positional relationship in the composite image between the travelable area and the placement area.

[0110] In some embodiments, the first generating module 802 is further configured to overlay the object image onto the placement area to obtain the composite image.

[0111] In some embodiments, the label information includes the falling object label and the non-falling object label, and the second generation module 803: a first determination submodule configured to determine that the pasting object in the composite image is a fallen object and generate the fallen object label when the placement area is within the drivable area; and a second determination submodule configured to determine that the pasted object in the composite image is a non-falling object if the placement area is outside the drivable area, and to generate the non-falling object label.

[0112] The first adjusting unit, the device further comprises: A first annotation module is configured to annotate the attachment object in the composite image using the label information.

[0113] The first adjustment unit, the first annotation module, a first annotation submodule configured to perform segment annotation on the pasting object in the composite image using the label information, wherein the segment annotation annotates the pasting object of each segment of the segmented pasting object; a second annotation sub-module configured to annotate the pasting object with a bounding box in the composite image using the label information; and a third annotation sub-module configured to annotate the object image in the composite image using the label information.

[0114] The first adjusting unit, the device further comprises: a first selection module configured to select the object image including the pasted object from an object image library; and a third generation module configured to generate the object image based on the pasting object.

[0115] The first adjustment unit, the first acquisition module 801, a third determination submodule configured to determine the drivable area from the pasting target image based on input instruction information, wherein the instruction information is used to indicate a position of the drivable area in the pasting target image; a first estimation submodule configured to perform area estimation on the pasting target image based on image content of the pasting target image and determine the drivable area; and a fourth determination sub-module configured to determine the drivable area based on annotated information in the pasting target image.

[0116] First adjustment unit: the object image includes a two-dimensional object image or a three-dimensional object image, and the stitching target image includes a two-dimensional stitching target image or a three-dimensional stitching target image.

[0117] The first adjusting unit, the device further comprises: a second acquisition module configured to acquire a composite image annotated with whether the attachment object is a falling object; and a first learning module configured to use a composite image annotated as to whether the pasted object is a falling object as falling object sample data to train a falling object detection model waiting to be trained, thereby obtaining a trained falling object detection model.

[0118] The first adjustment unit, the first learning module, a first extraction sub-module configured to perform feature extraction on the multi-frame falling object sample data to obtain an image feature set; a first classification submodule configured to classify image features annotated with falling object labels as positive samples and image features annotated with non-falling object labels as negative samples in the image feature set based on label information of the multi-frame falling object sample data; and a first learning submodule configured to learn the falling object detection model waiting to be learned based on the image features of the positive example samples and the image features of the negative example samples, thereby obtaining the learned falling object detection model.

[0119] In some embodiments, the first learning sub-module: a first adjustment unit configured to decrease a distance between image features having the same label information and increase a distance between image features having different label information in the image features of the positive example samples and the image features of the negative example samples, thereby obtaining adjusted image features of the positive example samples and adjusted image features of the negative example samples; and a second adjustment unit configured to adjust network parameters of the falling object detection model waiting to be trained based on the adjusted image features of the positive example samples and the adjusted image features of the negative example samples, thereby obtaining the trained falling object detection model.

[0120] The above description of the apparatus embodiments is similar to the description of the above method embodiments, and has the same beneficial effects as the method embodiments. In some embodiments, functions or modules included in the apparatus according to the embodiments of the present invention can be used to perform the methods described in the above method embodiments, and technical details not disclosed in the apparatus embodiments of the present invention can be understood by referring to the description of the method embodiments of the present invention.

[0121] In addition, in embodiments of the present invention, when the above-described image processing method is implemented in the form of a software functional module and sold or used as an independent product, it can be stored on a computer-readable storage medium. Based on this understanding, an essential part of the technical solution of embodiments of the present invention, i.e., a part that contributes to the related art, can be embodied in the form of a software product, which is stored on a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to execute all or part of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash memory, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk. Thus, embodiments of the present invention are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0122] An embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor executes the program to perform some or all of the steps of the above method.

[0123] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, the computer program causing a processor to perform some or all of the steps of the above-described method. The computer-readable storage medium may be transitory or non-transitory.

[0124] An embodiment of the present invention provides a computer program comprising computer readable code, which, when executed on a computing device, causes a processor of the computing device to perform some or all of the steps of the above-described method.

[0125]

[0013] An embodiment of the present invention provides a computer program product including a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program causing the computer to perform some or all of the steps of the above-described method when read and executed by a computer. The computer program product may be specifically implemented by hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product such as a software development kit (SDK).

[0126] It should be noted that the above descriptions of the embodiments tend to emphasize the differences between the embodiments, and their commonalities or similarities can be mutually referenced. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above embodiments of the method, and have the same beneficial effects as the embodiments of the method. Technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of the present invention can be understood by referring to the descriptions of the embodiments of the method of the present invention.

[0127] Note that Figure 9 is a schematic diagram of the hardware entities of a computer device according to an embodiment of the present invention. As shown in Figure 9, the hardware entities of the computer device 900 include a processor 901, a communication interface 902, and a memory 903.

[0128] The processor 901 typically controls the overall operation of the computing device 900 .

[0129] The communication interface 902 enables the computing device to communicate with other terminals or servers over a network.

[0130] The memory 903 stores instructions and applications executable by the processor 901, and can also cache data (e.g., image data, audio data, voice communication data, and video communication data) that is processed or has been processed by the processor 901 and each module in the computer device 900, and can be implemented by flash memory (FLASH) or random access memory (RAM). Data can be transferred between the processor 901, communication interface 902, and memory 903 via a bus 904.

[0131] It should be understood that "one embodiment" or "one embodiment" referred to throughout this specification means that a particular feature, structure, or characteristic associated with an embodiment is included in at least one embodiment of the present invention. Thus, "in one embodiment" or "in one embodiment" in this specification do not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that in each embodiment of the present invention, the order of the above-described steps / processes does not imply a chronological order of execution, and the order of execution of each step / process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. The above-described order of the embodiments of the present invention is not intended to indicate the superiority or inferiority of the embodiments, but is intended for convenience of description.

[0132] It should be noted that in embodiments of the present invention, "comprises," "includes," or any other variation thereof is intended to cover a non-exclusive inclusion, whereby a process, method, item, or apparatus comprising a set of elements not only includes those elements, but also other elements not expressly listed, or includes the inherent elements of those processes, methods, items, or apparatus. Unless otherwise limited, an element defined by the phrase "comprises" does not exclude the presence of other similar elements in the process, method, item, or apparatus that includes that element.

[0133] It should be understood that the disclosed devices and methods in some embodiments provided by the present invention can be realized in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is merely a division of logical functions. In actual implementation, other division methods may exist. For example, multiple units or components may be combined or integrated into another system, and some features may be ignored or not implemented. Furthermore, the couplings or direct couplings or communication connections between the components shown or discussed may be indirect couplings or communication connections via several interfaces, devices, or units, and may be in electrical, mechanical, or other forms.

[0134] The units described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed across multiple network units, and some or all of the units therein may be selected according to actual needs to achieve the objective of the technical solution in this embodiment.

[0135] Furthermore, the functional units in each embodiment of the present invention may all be integrated into a single processing unit, or each unit may function independently as a single unit, or two or more units may be integrated into a single unit, and the above-mentioned integrated units may be realized in the form of hardware or in the form of a functional unit of hardware and software.

[0136] As can be understood by those skilled in the art, all or some of the steps of the above method embodiments can be performed by hardware associated with program instructions, and the program can be stored in a computer-readable storage medium, and when the program is executed, the steps of the above method embodiments are performed, and the storage medium includes a medium that can store program code, such as a mobile storage device, a read-only memory (ROM), a magnetic memory, or an optical disk.

[0137] Alternatively, when the above-described integrated units of the present invention are realized in the form of software functional modules and sold or used as independent products, they can be stored in a single computer-readable storage medium. Based on this understanding, the essential parts of the technical solutions of the present invention, i.e., the parts that contribute to the related art, can be embodied in the form of a software product, and the computer software product is stored in a single storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a mobile storage device, a ROM, a magnetic disk, or an optical disk.

[0138] The above content is merely an embodiment of the present invention, and the protection scope of the present invention is not limited thereto. Any modifications or replacements that can be easily thought up by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention.

Claims

1. 1. An image processing method, comprising: Acquiring a placement area and a drivable area within the image to be pasted; arranging an object image including a pasting object in the arrangement area to generate a composite image; generating label information indicating whether the object to be attached is a fallen object based on a positional relationship in the composite image between the travelable area and the placement area.

2. The label information includes a falling object label and a non-falling object label, and generating label information indicating whether the object to be attached is a falling object or not based on a positional relationship between the travelable area and the placement area in the composite image is If the placement area is within the drivable area, determining that the object to be pasted in the composite image is a fallen object and generating the fallen object label; If the placement area is outside the drivable area, determining that the pasted object in the composite image is a non-falling object and generating the non-falling object label. The image processing method according to claim 1 .

3. After generating label information indicating whether the object to be attached is a fallen object based on the positional relationship between the travelable area and the placement area, the image processing method includes: further comprising annotating the object to be pasted in the composite image using the label information.

3. The image processing method according to claim 1.

4. Annotating the object to be pasted using the label information in the composite image includes: performing segment annotation on the pasting object in the composite image using the label information, wherein the pasting object of each segment of the segmented pasting object is annotated by the segment annotation; and annotating the object to be pasted with a bounding box in the composite image using the label information; annotating the object image in the composite image using the label information. The image processing method according to claim 3 .

5. The object image is selecting the object image containing the pasted object from an object image library; or The object image is obtained by generating the object image based on the pasting object.

5. The image processing method according to claim 1.

6. Acquiring a drivable area within the image to be pasted includes: determining the drivable area from the pasting target image based on input instruction information, the instruction information being used to indicate a position of the drivable area in the pasting target image; performing area estimation for the image to be pasted based on the image content of the image to be pasted, and determining the drivable area; determining the drivable area based on annotated information in the image to be pasted; The image processing method according to any one of claims 1 to 5.

7. After generating label information indicating whether the object to be attached is a fallen object based on the positional relationship between the travelable area and the placement area, the image processing method includes: acquiring a composite image in which an annotation is made as to whether the object to be pasted is a fallen object; The method further includes: using the composite image, in which the annotation is made as to whether the pasted object is a falling object or not, as falling object sample data to train a falling object detection model that is waiting to be trained, thereby obtaining a trained falling object detection model. The image processing method according to any one of claims 1 to 6.

8. The composite image in which the pasting target object is annotated as a falling object or not is used as falling object sample data to train a falling object detection model waiting to be trained, thereby obtaining a trained falling object detection model. performing feature extraction on the multi-frame falling object sample data to obtain an image feature set; Based on label information of the multi-frame falling object sample data, image features annotated with falling object labels in the image feature set are determined as positive example samples, and image features annotated with non-falling object labels are determined as negative example samples; and training the falling object detection model waiting to be trained based on the image features of the positive sample and the image features of the negative sample to obtain the trained falling object detection model. The image processing method according to claim 7.

9. training the falling object detection model waiting to be trained based on the image features of the positive sample and the image features of the negative sample to obtain the trained falling object detection model, In the image features of the positive example samples and the image features of the negative example samples, the distance between image features having the same label information is decreased, and the distance between image features having different label information is increased, thereby obtaining adjusted image features of the positive example samples and adjusted image features of the negative example samples; adjusting network parameters of the falling object detection model to be trained based on the adjusted image features of the positive example samples and the adjusted image features of the negative example samples, thereby obtaining the trained falling object detection model; The image processing method according to claim 8.

10. An image processing device, a first acquisition module configured to acquire a placement area and a drivable area within the pasting target image; a first generation module configured to place an object image including a pasting object in the placement area to generate a composite image; a second generation module configured to generate label information indicating whether the object to be attached is a falling object, based on a positional relationship in the composite image between the travelable area and the placement area.

11. A computing device comprising a memory and a processor, 10. A computer device, wherein the memory is configured to store a computer program executable by a processor, the processor being configured to execute the program to perform the steps of the image processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium having stored thereon a computer program for causing a processor to execute the steps of the image processing method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program or instructions for causing a processor to carry out the steps of the image processing method according to any one of claims 1 to 9.

Images