Image processing method, device, system, electronic device and computer-readable storage medium
By extracting and setting weight values to correlate images taken by cameras inside and outside the vehicle, the problems of lack of fun and participation in the images in the prior art are solved, and images that are preferred by users are generated.
Patent Information
- Application Number
- CN202110089885.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-01-22
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-01-22
AI Technical Summary
In the prior art, cameras are used for autonomous driving assistance but have not been maximized, and the images taken by outdoor cameras lack interest and sense of participation.
By extracting key point information in the images taken by the camera inside and outside the car, setting the weight value, and correlating the two, generating an image with overlaid weight value to increase the fun and sense of participation of the image.
It realizes the association of images inside and outside the car, generates images that are preferred by users, and improves the fun and sense of participation of the images.
Smart Images

Figure CN114820402B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image processing method, device, system, electronic equipment and computer-readable storage medium. Background Art
[0002] In recent years, external recognition systems that obtain external information around a vehicle through external sensors such as cameras and identify external information through external recognition devices have gradually become popular. For example, the following technology is disclosed in Japanese invention patent application No. 2018-558842. The external recognition system enables the vehicle to perform automatic driving or assisted driving by identifying the position of the identification object around the vehicle.
[0003] In addition, the following technology is disclosed in Japanese Patent Publication No. 2016-71577: The driver's face is photographed by a camera installed in the vehicle cabin, and the number of changes in his expression within a specified period is used to determine whether the driver is in a careless state, thereby reminding the driver of driving operations by estimating the person's emotions. Summary of the invention
[0004] However, in the above-mentioned prior art, the camera is only used to capture external information outside the vehicle or to capture the facial expressions of people inside the vehicle for automatic driving assistance. The use value of the camera itself has not been maximized, thus requiring further development and utilization of cameras and other shooting devices. In addition, there is also a technology in the prior art that uses an external camera to capture the scenery outside the vehicle to obtain an image outside the vehicle, but the image only reflects the scenery outside the vehicle and lacks interest and participation.
[0005] The present invention is completed in consideration of such technical problems, and its purpose is to provide an image processing method and its device, system, etc., which can associate scene images outside the vehicle, such as those captured by a camera outside the vehicle, with images inside the vehicle, such as those captured by a camera inside the vehicle, to obtain the image desired by the user, thereby making the image itself interesting and participatory.
[0006] A first aspect of the present invention is to provide an image processing method, which includes acquiring a first original image in a first scene, wherein the first original image includes a first object, extracting first key point information of the first object, and setting a weight value for the first key point information, and using the weight value as the first weight value of the first original image; acquiring a second original image in a second scene outside the first scene, wherein the second original image includes a second object, extracting second key point information of the second object, and setting a weight value for the second key point information, and using the weight value as the second weight value of the second original image; and increasing the second weight value of the second original image based on the first weight value of the first original image, so that the second original image has a superimposed weight value.
[0007] According to the first aspect of the present invention, by increasing the second weight value of the second original image based on the first weight value of the first original image, so that the second original image has a superimposed weight value, the first original image can be associated with the second original image to obtain the image the user wants, thereby making the image itself interesting and engaging.
[0008] A second aspect of the present invention is that the second original image is an image acquired simultaneously with the first original image or acquired based on the first key point information in the first original image.
[0009] According to the second aspect of the present invention, since the second original image is acquired simultaneously with the first original image or based on the first key point information in the first original image, an image that suits the user's preferences can be acquired.
[0010] The third aspect of the present invention is that the first original image is an emotional image, the first key point information includes one or more of a facial image, a human posture and voice information, the expression information of the facial image is extracted, the expression information is classified, and a first weight value is set according to the expression category, the posture information of the human posture is extracted, the posture is classified, and a first weight value is set according to the posture category, the keywords in the voice information are extracted, the keywords are classified, and a first weight value is set according to the keyword category.
[0011] According to the third aspect of the present invention, since the first weight value of the first original image is set by one or more of the face image, the posture of the person and the voice information, the interest level of the person in the first original image can be accurately obtained.
[0012] A fourth aspect of the present invention is to obtain an average of first weight values of at least two of expression information, human posture, and keywords as the first weight value of the first key point information.
[0013] The fifth aspect of the present invention is that, in the first key point information, when there is consistency between the expression information of the facial image, the posture of the person and at least two of the keywords, the first weight value of the first key point information with consistency is increased.
[0014] According to the fourth and fifth aspects of the present invention, by obtaining the average of the first weight values of at least two of the facial expression information, the person's posture and the keywords as the first weight value of the first key point information, or by increasing the first weight value of the first key point information that is consistent, the interest level of the person in the first original image can be more accurately obtained.
[0015] According to a sixth aspect of the present invention, the second original image is a scene image, which includes a humanistic image, a landscape image and a scene image, and features of the scene image are extracted as the second key point information.
[0016] According to the sixth aspect of the present invention, since the second original image is a scene image including a humanistic image and a landscape image, by combining the degree of interest of the person in the first original image with the scene image, a scene image based on the user's preferences can be obtained, and then an image based on the user's preferences can be obtained.
[0017] A seventh aspect of the present invention is that the first scene is a scene inside a vehicle, and the second scene is a scene outside a vehicle.
[0018] According to the seventh aspect of the present invention, since the first scene is the scene inside the vehicle and the second scene is the scene outside the vehicle, it is possible to associate the state of the people inside the vehicle with the scene outside the vehicle, and the degree of preference of the passengers for the scene outside the vehicle can be reflected, thereby obtaining an image including the passengers based on the passengers' preferences.
[0019] The eighth aspect of the present invention is to implement the image processing method described in any one of the first aspect to the seventh aspect of the present invention, to synthesize the first original image and the second original image into a composite image, wherein the composite image is an image that synthesizes part or all of the first original image with part or all of the second original image; and to set the composite image weight value of the composite image based on the first weight value of the first original image and the superimposed weight value of the second original image.
[0020] According to the eighth aspect of the present invention, the first original image and the second original image are combined into a composite image to obtain an image of interest to the user. Furthermore, a composite image based on user preference can be generated by setting a composite image weight value of the composite image.
[0021] The ninth aspect of the present invention is to implement the image synthesis method described in the eighth aspect of the present invention; generate a first label corresponding to the first original image based on the first key point information; generate a second label corresponding to the second original image based on the second key point information; and generate a synthetic image label corresponding to the synthetic image based on the first label and the second label.
[0022] According to the ninth aspect of the present invention, since the first tag, the second tag, and the synthesized image tag are generated, it is convenient for the user to obtain the first original image, the second original image, and the synthesized image.
[0023] The tenth aspect of the present invention is to provide an image organization method, implementing the image label generation method described in the ninth aspect of the present invention; classifying the synthetic images based on the synthetic image labels; and generating a sequence of the synthetic images based on the synthetic image weight values of the classified synthetic images.
[0024] According to the tenth aspect of the present invention, since the composite images are classified and a sequence of composite images is generated based on the composite image weight values of the classified composite images, composite images preferred by the user can be arranged in a priority position for easy acquisition by the user.
[0025] The eleventh aspect of the present invention is to provide an image output method, which implements the image arrangement method described in the tenth aspect of the present invention; and selects and outputs a composite image from the sequence of composite images.
[0026] According to the eleventh aspect of the present invention, a synthesized image is selected from the sequence of synthesized images and output, so that an image desired by the user can be obtained.
[0027] The twelfth aspect of the present invention is to provide an image correction method, which implements the image output method described in the eleventh aspect of the present invention; based on the feedback of the audience evaluation on the output composite image, the composite image weight value of the composite image is corrected, and then the sequence of the composite image is adjusted.
[0028] According to the twelfth aspect of the present invention, based on the feedback of the audience evaluation on the output composite image, the composite image weight value of the composite image is corrected, and then the sequence of the composite images is adjusted. Therefore, the sequence of the composite images can be optimized and the user experience can be improved.
[0029] The thirteenth aspect of the present invention is to provide an image processing device, including: a first image acquisition unit, which acquires a first original image in a first scene, wherein the first original image includes a first object; a first key point information extraction unit, which extracts first key point information of the first object; a first image weight setting unit, which sets a weight value for the first key point information, and uses the weight value as the first weight value of the first original image; and also includes: a second image acquisition unit, which acquires a second original image in a second scene outside the first scene, wherein the second original image includes a second object; a second key point information extraction unit, which extracts second key point information of the second object; a second image weight setting unit, which sets a weight value for the second key point information, and uses the weight value as the second weight value of the second original image; the image processing device increases the second weight value of the second original image based on the first weight value of the first original image, so that the second original image has a superimposed weight value.
[0030] The fourteenth aspect of the present invention is that the second original image is an image acquired simultaneously with the first original image or acquired based on the first key point information in the first original image.
[0031] The fifteenth aspect of the present invention is that the first original image is an emotional image, and the first key point information includes one or more of a facial image, a human posture, and voice information. When the first key point information extraction unit extracts the expression information of the facial image, the expression information is classified, and the first image weight setting unit sets a first weight value according to the expression category. When the first key point information extraction unit extracts the posture information of the human posture, the posture is classified, and the first image weight setting unit sets a first weight value according to the posture category. When the first key point information extraction unit extracts the keywords in the voice information, the keywords are classified, and the first image weight setting unit sets a first weight value according to the keyword category.
[0032] The sixteenth aspect of the present invention is that the first image weight setting unit obtains the average of the first weight values of at least two of the expression information, the person's posture and the keyword as the first weight value of the first key point information.
[0033] The seventeenth aspect of the present invention is that, in the first key point information, when the expression information of the facial image, the posture of the person and at least two of the keywords are consistent, the first image weight setting unit increases the first weight value of the first key point information that is consistent.
[0034] According to an eighteenth aspect of the present invention, the second original image is a scene image, which includes a humanistic image, a landscape image and a scene image, and the second key point information extraction unit extracts features of the scene image as the second key point information.
[0035] The nineteenth aspect of the present invention is to provide an image synthesis device, which synthesizes the first original image and the second original image into a composite image based on the processing performed by the image processing device described in any one of the thirteenth to eighteenth aspects of the present invention, wherein the composite image is an image in which part or all of the first original image is synthesized with part or all of the second original image; and based on the first weight value of the first original image and the superimposed weight value of the second original image, sets the composite image weight value of the composite image.
[0036] The twentieth aspect of the present invention is to provide an image label generating device, which generates a first label corresponding to the first original image according to the first key point information based on the processing performed by the image synthesis device described in the nineteenth aspect of the present invention; generates a second label corresponding to the second original image according to the second key point information; and generates a composite image label corresponding to the composite image according to the first label and the second label.
[0037] The twenty-first aspect of the present invention is to provide an image organizing device, which classifies the synthetic images according to the synthetic image labels based on the processing performed by the image synthesis device described in the twentieth aspect of the present invention; and generates a sequence of the synthetic images according to the synthetic image weight values of the classified synthetic images.
[0038] The twenty-second aspect of the present invention is to provide an image output device, which selects and outputs a synthetic image from the sequence of synthetic images based on the processing performed by the image sorting device described in the twenty-first aspect of the present invention.
[0039] The twenty-third aspect of the present invention is to provide an image correction device, which is based on the processing performed by the image output device described in the twenty-second aspect of the present invention, and corrects the composite image weight value of the composite image according to the feedback of the audience evaluation on the output composite image, thereby adjusting the sequence of the composite images.
[0040] According to the thirteenth to twenty-third aspects of the present invention, by providing an image processing device, an image synthesis device, an image label generation device, an image sorting device, an image output device and an image correction device, a synthetic image based on user preferences can be generated.
[0041] The twenty-fourth aspect of the present invention is to provide an image processing system, including an image processing module, an image synthesis module, an image label generation module, an image sorting module, an image output module and an image correction module. In the image processing module, a first original image is obtained in a first scene, the first original image contains a first object, the first key point information of the first object is extracted, and a weight value is set for the first key point information, and the weight value is used as the first weight value of the first original image; a second original image is obtained in a second scene outside the first scene, the second original image contains a second object, the second key point information of the second object is extracted, and a weight value is set for the second key point information, and the weight value is used as the second weight value of the second original image; based on the weight value of the first original image, the second weight value of the second original image is increased, so that the second original image has a superimposed weight value; in the image synthesis module, the first original image and the second original image are synthesized into a synthesized image, wherein the synthesized image is An image is generated by combining part or all of the first original image with part or all of the second original image, and a composite image weight value of the composite image is set based on the first weight value of the first original image and the superimposed weight value of the second original image; in the image label generation module, a first label corresponding to the first original image is generated based on the first key point information, a second label corresponding to the second original image is generated based on the second key point information, and a composite image label corresponding to the composite image is generated based on the first label and the second label; in the image sorting module, the composite images are classified based on the composite image labels, and a sequence of composite images is generated based on the composite image weight values of the classified composite images; in the image output module, a composite image is selected and output from the sequence of composite images; in the image correction module, the composite image weight value of the composite image is corrected based on the feedback of the audience evaluation on the output composite image, thereby adjusting the sequence of composite images.
[0042] According to the twenty-fourth aspect of the present invention, by setting the image processing system, it is possible to generate a composite image based on user preference.
[0043] The twenty-fifth aspect of the present invention is to provide an electronic device, comprising: a memory for storing executable instructions; a processor for implementing the image processing method described in any one of the first to seventh aspects of the present invention, or the image synthesis method described in the eighth aspect of the present invention, or the image tag generation method described in the ninth aspect of the present invention, or the image organization method described in the tenth aspect of the present invention, or the image output method described in the eleventh aspect of the present invention, or the image correction method described in the twelfth aspect of the present invention when running the executable instructions stored in the memory.
[0044] The twenty-sixth aspect of the present invention is that the electronic device also includes a display, which is divided into a plurality of display areas, respectively used to display the first original image, the second original image and the synthesized image.
[0045] According to the twenty-fifth and twenty-sixth aspects of the present invention, by setting up an electronic device and displaying the electronic device to display the first original image, the second original image and the synthesized image, it is possible to conveniently generate a synthesized image based on user preferences.
[0046] The twenty-seventh aspect of the present invention is a computer-readable storage medium storing an executable program, characterized in that when the executable program is executed by a processor, it implements the image processing method described in any one of the first to seventh aspects of the present invention, or implements the image synthesis method described in the eighth aspect of the present invention, or implements the image label generation method described in the ninth aspect of the present invention, or implements the image organization method described in the tenth aspect of the present invention, or implements the image output method described in the eleventh aspect of the present invention, or implements the image correction method described in the twelfth aspect of the present invention.
[0047] According to the twenty-seventh aspect of the present invention, by providing a computer-readable storage medium storing an executable program, a composite image based on user preference can be generated.
[0048] The above-mentioned objects, features and advantages will be easily understood through the following description of the embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 The present invention is a block diagram of the structure of an image association processing device including an image processing device, an image synthesis device, an image label generation device, an image arrangement device, an image output device and an image correction device.
[0050] Figure 2 It is a block diagram showing the structure of an image processing device.
[0051] Figure 3The present invention is a flowchart of an image association processing method including an image processing step, an image synthesis step, an image label generation step, an image arrangement step, an image output step and an image correction step.
[0052] Figure 4 yes Figure 3 Flowchart of the image processing steps in .
[0053] Figure 5 yes Figure 3 Flowchart of the image synthesis steps in .
[0054] Figure 6 yes Figure 3 Flowchart of the steps for generating image labels in .
[0055] Figure 7 yes Figure 3 Flowchart of the image organization steps in .
[0056] Figure 8 yes Figure 3 Flowchart of the image output steps in .
[0057] Fig. 9 yes Figure 3 Flowchart of the image correction steps in .
[0058] Fig.10 It is a structural block diagram of an image association processing system including an image processing module, an image synthesis module, an image label generation module, an image sorting module, an image output module and an image correction module.
[0059] Fig.11 This is an example of a composite image.
[0060] Fig.12 This is another example showing a composite image. DETAILED DESCRIPTION
[0061] Preferred embodiments are listed below, and the image processing method and its apparatus, system, electronic device, and computer-readable storage medium involved in the present invention are described in detail with reference to the accompanying drawings.
[0062] Figure 1 The structure block diagram of the image association processing device 1 includes an image processing device, an image synthesis device, an image label generation device, an image sorting device, an image output device and an image correction device. The arrows in the structure block diagram indicate the output flow of the processed image, and the composition and function of each device can be clearly understood from the following description.
[0063] [Image processing device 10]
[0064] like Figure 2As shown, the image processing device 10 of the present application includes: a first image acquisition unit 11, which acquires a first original image in a first scene, wherein the first original image contains a first object; a first key point information extraction unit 12, which extracts first key point information of the first object; and a first image weight setting unit 13, which sets a weight value for the first key point information and uses the weight value as the first weight value of the first original image.
[0065] like Figure 2 As shown, the image processing device 10 also includes: a second image acquisition unit 14, which acquires a second original image in a second scene outside the first scene, wherein the second original image contains a second object; a second key point information extraction unit 15, which extracts second key point information of the second object; and a second image weight setting unit 16, which sets a weight value for the second key point information and uses the weight value as the second weight value of the second original image.
[0066] The image processing device 10 increases the second weight value of the second original image based on the first weight value of the first original image, so that the second original image has a superimposed weight value.
[0067] The second original image is an image acquired simultaneously with the first original image or acquired based on the first key point information in the first original image.
[0068] The first original image is an emotional image, and the first key point information includes one or more of a facial image, a person's posture, and voice information. When the first key point information extraction unit 12 extracts the expression information of the facial image, the expression information is classified, and the first image weight setting unit 13 sets a first weight value according to the expression category. When the first key point information extraction unit 12 extracts the posture information of the person's posture, the posture is classified, and the first image weight setting unit 13 sets a first weight value according to the posture category. When the first key point information extraction unit 12 extracts the keywords in the voice information, the keywords are classified, and the first image weight setting unit 13 sets a first weight value according to the keyword category.
[0069] The first image weight setting unit 13 obtains an average of first weight values of at least two of expression information, human posture and keywords as the first weight value of the first key point information.
[0070] Alternatively, in the first key point information, when there is consistency between the expression information of the facial image, the posture of the person and at least two of the keywords, the first image weight setting unit 13 increases the first weight value of the first key point information that is consistent.
[0071] The second original image is a scene image, which includes a humanistic image, a landscape image and a scene image.
[0072] The second key point information extraction unit 15 extracts features of the scene image as the second key point information.
[0073] [Image synthesis device 20]
[0074] The image synthesis device 20 of the present application performs the following processing in addition to the processing performed by the above-mentioned image processing device 10 .
[0075] The image synthesis device 20 synthesizes the first original image and the second original image into a composite image based on the processing performed by the above-mentioned image processing device 10, wherein the composite image is an image in which part or all of the first original image is synthesized with part or all of the second original image; and sets the composite image weight value of the composite image according to the first weight value of the first original image and the superimposed weight value of the second original image.
[0076] [Image label generating device 30]
[0077] The image tag generating device 30 of the present application performs the following processing in addition to the processing performed by the above-mentioned image synthesizing device 20 .
[0078] Based on the processing performed by the above-mentioned image synthesis device 20, the image label generating device 30 generates a first label corresponding to the first original image according to the first key point information; generates a second label corresponding to the second original image according to the second key point information; and generates a composite image label corresponding to the composite image according to the first label and the second label.
[0079] Specifically, for example, after acquiring the key point information in the emotional image and the key point information in the scene image, the image label generation device 30 determines the emotional label corresponding to the key point information of the emotion and the scene label corresponding to the key point information of the scene according to the preset classification model. In order to reduce the burden of the GPU (Graphics Processing Unit), a classification algorithm based on the CPU (Central Processing Unit) can be used in this embodiment, such as a softmax classification algorithm, an XGBoost classifier, a random forest, and other classification algorithms.
[0080] It should be noted that since emotion labels and scenario labels have different attributes, using different classification models for classification and recognition can improve their accuracy. In addition, the fewer the number of classifications, the lower the possibility of overfitting of the classifier, thereby improving the classification accuracy.
[0081] Taking scene labels as an example, any one of the above classification algorithms is used to build a classifier, which takes scenes as input and scene labels as output. Among them, scene labels can include sea, mountains, crowds, etc. The classifier is trained using training videos (images) to determine model parameters. Among them, it is possible to collect out-of-car videos by simulating a variety of user usage scenarios and obtain training videos based on out-of-car videos. In addition, scene labels can have time labels, content labels, and geographic location labels corresponding to the image content, such as January 1, 2020, people, scenery, animals, Beijing, Shanghai, etc.
[0082] For emotional labels, another classifier is constructed separately. The input video of the classifier is the emotional feature, and the output video is the emotional label. Among them, the emotional labels include happy, excited, depressed, sad, bored, dazed, etc., which represent the user's current emotional state. The training method of this classifier is the same as the training method of the first classifier of the scenario label. In addition, the emotional label can also have a time label, content label, and geographic location label corresponding to the image content, for example, January 1, 2020, happy, excited, depressed, sad, bored, dazed, Beijing, Shanghai, etc.
[0083] [Image Arrangement Device 40]
[0084] The image organizing device 40 of the present application performs the following processing in addition to the processing performed by the above-mentioned image tag generating device 30 .
[0085] The image arrangement device 40 classifies the composite image according to the composite image label based on the processing performed by the image label generation device 30, and generates a sequence of the composite images according to the composite image weight values of the classified composite images. Here, the composite image label can be a combination of an emotion label and a scene label, and the classification of the composite image can be a category belonging to an emotion label or a scene label, or can be both an emotion label and a scene label.
[0086] [Image Output Device 50]
[0087] The image output device 50 of the present application performs the following processing in addition to the processing performed by the image organizing device 40 .
[0088] The image output device 50 selects and outputs a synthesized image from the sequence of synthesized images based on the processing performed by the image organizing device 40 described above.
[0089] The image processing device, image synthesis device, image label generating device, image sorting device, and image output device of the present invention and the image association processing device composed of these devices can be performed in the cloud.
[0090] [Image Correction Device 60]
[0091] The image correction device 60 of the present application performs the following processing in addition to the processing performed by the above-mentioned image output device 50 .
[0092] The image correction device 60 corrects the synthetic image weight value of the synthetic image according to the feedback of the audience evaluation on the output synthetic image based on the above-mentioned image output device 50, and further adjusts the sequence of the synthetic images.
[0093] The image correction device of the present invention may be provided on an APP on a user terminal described later.
[0094] This application also discloses an image association processing method. Figure 3 The present invention is a flowchart of an image association processing method including an image processing step, an image synthesis step, an image label generation step, an image arrangement step, an image output step and an image correction step.
[0095] like Figure 3 As shown, the image association processing method includes an image processing step S10, an image synthesis step S20, an image tag generation step S30, an image arrangement step S40, an image output step S50 and an image correction step S60. The steps are described below.
[0096] like Figure 4 As described above, the image processing step S10 is performed by the above-mentioned image processing device 10 .
[0097] The image processing steps include:
[0098] Step S101: acquiring a first original image in a first scene through a first image acquisition unit 11;
[0099] Step S102: extracting first key point information of a first object in the acquired first original image by the first key point information extraction unit 12;
[0100] Step S103: setting a weight value for the first key point information extracted by the first image weight setting unit 13, and using the weight value as a first weight value of the first original image;
[0101] Step S104: acquiring a second original image in a second scene outside the first scene by the second image acquisition unit 14;
[0102] Step S105: extracting second key point information of the second object in the acquired second original image by the second key point information extraction unit 15;
[0103] Step S106: setting a weight value for the second key point information by the second image weight setting unit 16, and using the weight value as a second weight value of the second original image; and
[0104] Step S107: increasing the second weight value of the second original image based on the first weight value of the first original image, so that the second original image has a superimposed weight value.
[0105] Figure 5 yes Figure 3 The image synthesis step S20 is executed by the image synthesis device 20 based on the output of the above-mentioned image processing step S10.
[0106] The image synthesis step includes:
[0107] Step S201: synthesizing the first original image and the second original image acquired in step S101 and step S104 into a composite image, wherein the composite image is an image synthesized by partially or completely synthesizing the first original image with partially or completely synthesized the second original image;
[0108] Step S202: setting a composite image weight value of the composite image based on the first weight value of the first original image and the superimposed weight value of the second original image.
[0109] Figure 6 yes Figure 3 The image label generation step S30 is a flowchart of the image label generation step S30 in FIG. The image label generation step is based on the output of the above-mentioned image synthesis step S20 and is executed by the image label generation device 30 .
[0110] The image tag generation step S30 includes:
[0111] Step S301: generating a first label corresponding to the first original image based on the first key point information;
[0112] Step S302: generating a second label corresponding to the second original image based on the second key point information;
[0113] Step S303: generating a composite image label corresponding to the composite image based on the first label and the second label.
[0114] The method for generating the label has been described in detail in the description of the corresponding device above, and will not be repeated here.
[0115] Here, the label of the synthetic image may be a combination of an emotion label and a scene label, and the classification of the synthetic image may be a category belonging to an emotion label or a scene label, or may be a category belonging to both an emotion label and a scene label.
[0116] Figure 7 yes Figure 3 This step is performed by the image organizing device 40 based on the output of the above-mentioned image label generating step S30.
[0117] The image arrangement step includes:
[0118] Step S401: classifying the synthetic image based on the synthetic image label;
[0119] Step S402: generating a sequence of the synthetic images based on the synthetic image weight values of the classified synthetic images.
[0120] Figure 8 yes Figure 3 The image output step S50 is a flowchart of the image output step S50. The image output step S50 is performed by the image output device 50 based on the output of the above-mentioned image arrangement step S40. The image output step includes:
[0121] Step S501: selecting an image from the sequence of composite images;
[0122] Step S502: output the selected image.
[0123] Fig. 9 yes Figure 3 Flowchart of the image correction step S60 in FIG. The image correction step S60 is executed by the image correction device 60 based on the output of the image output step S50.
[0124] The image correction step S60 includes:
[0125] Step S601: based on feedback on the outputted composite image, modifying the composite image weight value of the composite image;
[0126] Step S602: adjusting the synthesized image sequence.
[0127] The image processing in the above steps S101 to 501 may be performed on the cloud.
[0128] The above step S601 can be performed on an APP on a user terminal described later.
[0129] In step S101, a first original image is acquired in a first scene, which may be inside a vehicle, a ship, an airplane, a room, etc., and the first original image is an image related to a person in the vehicle, the ship, the airplane, the room, etc.
[0130] In step S104, a second original image is acquired in a second scene outside the first scene. Corresponding to the first scene, the second scene may be outside a vehicle, outside a ship, outside an airplane, outside a room, etc., and the second original image is a scene image outside a vehicle, outside a ship, outside an airplane, outside a room, etc., including a humanistic image, a landscape image, and a scene image. The acquisition of the second original image is performed simultaneously with the acquisition of the first original image in step S101, or the second original image is acquired based on the first key point information in the first original image.
[0131] In step S102, first key point information of the first object in the acquired first original image is extracted, and the first key point information includes one or more of a face image, a person's posture, and voice information. When the first key point information is a face image, the expression information of the face image is extracted; when the first key point information is a person's posture, the posture information of the person's posture is extracted; when the first key point information is voice information, keywords in the voice information are extracted.
[0132] In step S103, a weight value is set for the first key point information extracted, and the weight value is used as the first weight value of the first original image. When the expression information of the face image is extracted in step S102, the expression information is classified and the first weight value is set according to the expression category; when the posture information of the posture of the person is extracted in step S102, the posture is classified and the first weight value is set according to the posture category; when the keywords in the voice information are extracted in step S102, the keywords are classified and the first weight value is set according to the keyword category. And the average value of the first weight value of at least two of the expression information, the posture of the person and the keyword is obtained as the first weight value of the first key point information. Or, in the first key point information, when there is consistency between the expression information of the face image, the posture of the person and at least two of the keywords, the first weight value of the first key point information with consistency is increased.
[0133] In step S105, second key point information of the second object in the acquired second original image is extracted. The second original image is a scene image, which includes a humanistic image, a landscape image and a scene image, and features of the scene image are extracted as the second key point information.
[0134] In step S106, a weight value is set for the second key point information extracted in step S105 as a second weight value of the second original image.
[0135] In step S107, the second weight value of the second original image is increased based on the first weight value of the first original image obtained in step S103, that is, the second weight value of the second original image set in step S106 is increased, so that the second original image has an overlay weight value of the overlay weight.
[0136] By setting a second weight value for the second original image and increasing the second weight value of the second original image based on the first weight value of the first original image, so that the second original image has a superimposed weight value of superimposed weight, the user's favorite image can be ranked in a preferred position, so that the user can get the desired image.
[0137] The above device and the above method are described below with examples.
[0138] First, the first scene and the second scene are scenes in different areas but related to each other. For example, the first scene can be inside a vehicle, inside a ship, inside an airplane, inside a room, etc., and corresponding to the first scene, the second scene can be outside a vehicle, outside a ship, outside an airplane, outside a room, etc., and the first original image is an image related to people inside a vehicle, inside a ship, inside an airplane, inside a room, etc., and the second original image is a scene image outside a vehicle, outside a ship, outside an airplane, outside a room, etc., including humanistic images, scenery images, and scene images, etc., wherein humanistic images are images related to people's cultural activities, such as cultural temple fairs, etc., scenery images are images related to scenery, such as the sea, mountains, etc., and scene images are scenes related to social activities, such as crowds of people.
[0139] The image processing device 10 is described below mainly by taking an example in which the first scene is a scene inside a vehicle and the second scene is a scene outside the vehicle.
[0140] The vehicle may be any type of vehicle, for example, a car, a van, a truck, a bus, etc. The vehicle may be an ordinary vehicle, an autonomous driving vehicle, an intelligent networked vehicle, etc. A first image acquisition unit 11 is provided in the vehicle. The first image acquisition unit 11 includes a first image input device and a voice input device.
[0141] The first image input device is one or more cameras installed in the vehicle compartment to capture the faces of the passengers. The camera may also be an infrared camera. When one camera is installed, the camera is configured in a position where the faces of all passengers in the vehicle compartment can be captured, for example, near a rearview mirror. When multiple cameras are installed, the camera is installed at each seat.
[0142] In addition, the first image input device also captures the posture of the occupant in the vehicle (including movement of the face, inclination of the body, shift of sight, etc.).
[0143] The voice input device is arranged in the vehicle, and the voice input device in the vehicle includes voice components such as a microphone. Multiple voice components can be arranged and arranged in a position where all passengers in the vehicle can clearly record voice, such as on the dashboard or roof of the vehicle, or on each seat.
[0144] In the above structure, the in-car camera and the voice component are separately arranged, but the invention is not limited thereto, and the in-car camera and the voice component can also be arranged as one body, which can simplify the in-car video acquisition device and save space.
[0145] In addition, a second image acquisition unit 14 is provided outside the vehicle. The second image acquisition unit 14 includes a second image input device. The second image input device is one or more cameras, for example, multiple cameras provided at the four corners outside the vehicle and on the roof. The cameras provided at the two corners on the front side of the vehicle shoot the scene in front of the vehicle in the direction of travel, the cameras provided at the two corners on the rear side of the vehicle shoot the scene behind the direction of travel of the vehicle, and the camera provided on the roof is a stereo camera, which can shoot 360° of the entire vehicle plane when looking down, and can also shoot the sky.
[0146] In step S101, the image processing device 10 acquires an in-cabin image as a first original image in the vehicle cabin through the first image acquisition unit 11. The in-cabin image includes facial image information and human posture of an in-cabin occupant as a first object captured by a camera, and also includes voice information recorded by a voice component.
[0147] While executing step S101 , step S104 is also executed. In step S104 , the image processing device 10 uses the second image acquisition unit 14 to enable the vehicle exterior camera to capture the scene outside the vehicle compartment and acquire vehicle exterior image information as a second original image.
[0148] For example, when a vehicle is driving on the seashore, the switch for simultaneous shooting inside and outside the vehicle is turned on. At this time, the camera inside the vehicle is turned on to capture the facial expressions, postures and line of sight of the people inside the vehicle to form a character image, the microphone is turned on to input the content of the conversation of the people inside the vehicle to form a voice, and at the same time, the camera outside the vehicle is turned on to capture the scene of the sea outside the vehicle to form a landscape image. When the driver and passengers turn sideways or turn their eyes to one side to look at the beautiful scenery outside the window and show joyful expressions, while exclaiming "The sea is so beautiful under the sunset", the camera inside the vehicle captures the posture of the driver and passengers turning sideways or turning their eyes to one side, the joyful expressions they show, and the voice of "The sea is so beautiful under the sunset" through the microphone. At the same time, the scenery of the sea and the sunset outside the window is captured through the camera outside the vehicle. In addition, the switch for simultaneous shooting inside and outside the vehicle can be turned on by the operation of the passenger, or by setting trigger conditions, such as prompt sound, prompt gesture, prompt posture, vehicle location, etc. When the switch for simultaneous shooting inside and outside the vehicle is turned on by the trigger condition, the vehicle system can first give a shooting confirmation instruction and the passenger can give feedback to agree to the shooting, and then turn on the switch for simultaneous shooting inside and outside the vehicle to shoot. Among them, using the location of the vehicle as a trigger condition means that the user's favorite location is recorded in the system in advance and the vehicle is located by GPS and has driven to the location as a trigger condition.
[0149] Next, in step S102, the first key point information extraction unit 12 extracts one or more of the facial image of the occupant in the vehicle interior image acquired in step S101, i.e., the expression of joy, the posture of the person, i.e., the body and the line of sight directed to one side, and the voice information expressing evaluation in the voice information, and in step S105, the second key point information of the second object in the acquired second original image, i.e., the beautiful scenery outside the car window, i.e., the sea and the sunset, is extracted by the second key point information extraction unit 15.
[0150] Among them, the extraction process of key point information can be carried out through filtering method, packaging method, integration method and other methods.
[0151] The filtering method refers to filtering the extracted key point information to delete redundant feature videos.
[0152] The packaging method is used to filter the extracted key point information.
[0153] The integration method refers to integrating multiple key point information extraction methods together to construct a more efficient and accurate key point information extraction method for extracting key point information.
[0154] The extraction of the emotional features expressed by the facial image in the first key point information can be performed in the following manner.
[0155] An expression recognition method based on the fusion of traditional biometric features of facial key point areas and global deep features of the face is adopted. Traditional local binary pattern features, local direction number features, etc. are used with deep neural network features to train facial expression classifiers respectively, and these classifiers are used to vote for face recognition.
[0156] The image features are deeply extracted by pre-trained CNN (Convolutional Neural Networks) as emotional features. A large amount of expression data is collected to train the deep neural network to extract low-dimensional global deep features, and the high-dimensional traditional biometric features of the local area around the key points are combined to perform multi-feature fusion classification and recognition. The CNN model can be set with multiple convolution parts and pooling parts, without a fully connected part, and its input video is a face image. The video output of the last convolution part or pooling part is used as the output video of the model, wherein the output video is a facial image feature. Non-public facial expression data is collected through the network, and the expression data is divided into multiple expressions such as happiness, surprise, sadness, anger, disgust, fear and neutrality. The facial key point detection algorithm in the dilb library is used to detect the facial key points of the facial expression data, and multiple facial key points are obtained for each face. The facial area is obtained using the multiple facial key points, and the facial area is calibrated and normalized to a fixed size. The key point position is used to obtain traditional expression features, and the traditional expression features include facial eyebrows, eyes, nose and mouth features. The facial expression feature extraction uses local binary pattern LBP features, local direction number LDN features and curvelet features. According to the obtained facial area, a deep neural network is trained to obtain deep learning features. The deep learning features refer to the vector values of the convolutional layer or the fully connected layer obtained by inputting the expression image into the trained neural network. In the recognition technology, the phasor value is used as the deep feature vector of the image. A classification result can be obtained by inputting 3 traditional features and 1 deep feature of the expression to obtain the emotional characteristics of the person in the car.
[0157] In addition, the extraction of keywords in the voice information in the first key point information can be performed in the following manner.
[0158] Method 1: Collect voice files through the voice component; convert the voice video into a spectrogram according to the audio feature extraction algorithm; generate a semantic feature vector of the voice video according to a preset autoencoder convolutional neural network and the spectrogram, and use the semantic feature vector as the emotional feature.
[0159] The audio feature extraction algorithm may be an MFCC (Mel Frequency Cepstrum Coefficient) algorithm or an FFT (Fast Fourier Transformation) algorithm, which converts the speech video into a spectrogram through the audio feature extraction algorithm, and uses the spectrogram as the input video and output video of the autoencoder convolutional neural network to extract semantic feature vectors from the network. Similar to the above-mentioned autoencoder recurrent neural network, the autoencoder convolutional neural network is also an autoencoder, in which the autoencoder convolutional neural network uses a convolutional part to construct an autoencoder, and the output video of the autoencoder is trained to be consistent with the input video to obtain valuable information in the middle hidden part.
[0160] Method 2: In addition, the speech can also be converted into text for storage, that is, by preprocessing the acquired speech, obtaining speech features corresponding to the speech, storing the speech features in a database model library, and performing matching processing to obtain a speech signal, processing the speech signal based on a preset language model, determining a speech sequence corresponding to the speech signal, decoding the speech sequence based on a preset sound construction model, and obtaining text information corresponding to the speech sequence.
[0161] In addition, the extraction of the person's posture in the first key point information can be performed in the following manner.
[0162] After obtaining an image of the occupant in the vehicle through a camera in the vehicle, the image is transmitted to a human posture recognition model for recognition, and the human posture recognition model outputs a posture action recognition result, wherein the human posture recognition model is a recognition model constructed based on a neural network, and is used to recognize human posture features in an input image. Before the human posture recognition model recognizes the input image, it also includes: training the human posture recognition model; the specific training process is: obtaining training samples and transmitting the training samples to the human posture recognition model, and adjusting the neural network until the training of the human posture recognition model is completed.
[0163] Specifically, after constructing the above-mentioned human posture recognition model, the network architecture needs to be trained. This step mainly outputs a large number of samples to the neural network to fine-tune the entire deep neural network so that it can quickly identify the action characteristics of the human posture during vehicle driving. In addition, the pre-established human posture recognition model can be trained and updated. If the system is allowed to be connected to the Internet, the system will update the server data, which is the data after the system performs deep learning on a large number of samples; updating the data is conducive to the system to more accurately identify actions and recognize more actions; wherein, the acquired data is the data after a large number of action learning of human posture, and updating the data is conducive to the system to more accurately identify human actions and postures, and recognize more actions. For example, the human posture features of the body, face and sight towards one side in the posture action recognition result are compared with the action features of the body, face and sight in the human posture recognition model. When the similarity reaches a preset value, the human posture features in the posture action recognition result are determined to be the desired features and used as the first key point information. In addition, for example, the duration or movement frequency of the human posture features of the body, face and line of sight toward one side in the posture and action recognition results can also be compared with a preset value in the human posture recognition model. When the preset value is reached, the human posture features in the posture and action recognition results are determined to be the desired features and used as the first key point information.
[0164] The scene image in the second key point information may be extracted in the following manner.
[0165] The vehicle exterior scene image is decomposed into three parts: image features, audio features, and text features. The image features, audio features, and text features of the vehicle exterior scene image are input into the trained prediction model. The gated context algorithm is used to calculate the correlation between each feature, and then the weight of each feature in the output vehicle exterior scene image is adjusted based on the calculation result of the correlation. The weight of the feature with strong correlation is increased, and the weight of the feature with weak correlation is reduced. If the vehicle exterior scene image belongs to a silent video, the audio feature is not extracted. By using the directional gradient histogram algorithm, the linear prediction analysis algorithm, and the word frequency-inverse document frequency algorithm, the image, audio, and text in the vehicle exterior scene image are respectively extracted to obtain the feature extraction result. In this way, the feature analysis is more accurate and the generalization of the vehicle exterior scene image is also higher. For example, when the vehicle exterior scene content is: there is a fishing ban sign, the sea patting the coast under the sunset, and there are many seagulls flying and singing on the sea. The key point information can be extracted as: "there are seagulls, sunset, the sound of patting the coast, the sea in the fishing ban period", etc.
[0166] After the above extraction of key point information is completed, in step S103, the first image weight setting unit 13 sets the first weight value for the extracted first key point information, i.e., the expression of the face image, the direction of the body and sight in the posture of the person, and the voice information indicating the evaluation in the voice information. If there is consistency among the expression of the face image, the direction of the body and sight in the posture of the person, and the voice information indicating the evaluation in the voice information, that is, all the actions of the occupants in the vehicle indicate that a scene that they like or dislike appears outside the vehicle, in step S103, the first weight value of the first key point information with consistency, i.e., the face image, the posture of the person, and the voice information is increased. If there is no consistency between any two of the expression of the face image, the direction of the body and sight in the posture of the person, and the voice information, in step S103, the first image weight setting unit 13 obtains the average of the first weight values of at least two of the expression information, the posture of the person, and the keyword as the first weight value of the first key point information. The first weight value as the average value is less than the first weight value increased due to the consistency.
[0167] When setting the first weight value of a face image, the first weight value can be set according to the above-mentioned expression data of the facial expression of the face, for example, the first weight value -2, -1, 0, 1, 2, ... can be set for the face image according to disgust, disdain, indifference, smile, surprise, ... When setting the first weight value of a person's posture, the first weight value -2, -1, 0, 1, 2, ... can be set for the person's posture, for example, there has been no movement toward one side, the body looks toward one side for a while and then returns to a normal posture and no longer looks toward one side, the body looks toward one side for a while and then returns to a normal posture and then looks toward one side once again, the body looks toward one side for a specified time (for example, one minute), the body looks toward one side for more than a specified time or the number of times the body looks toward one side exceeds a specified number (for example, 6 times) within a specified time (for example, one minute), ... When setting the first weight value of voice information, the first weight value -2, -1, 0, 1, 2, ... can be set for the voice information according to the degree of semantic evaluation in the voice information, for example, very poor, not good, average, good, very good, ...
[0168] For example, when a person's facial expression is disdainful, the first weight value of the face image is set to -1; when a person's body is facing one side for more than a specified time or the number of times the body is facing one side exceeds a specified number of times within a specified time, the first weight value of the person's posture is set to 2; when the evaluation level in the voice information is good, the first weight value of the voice information is set to 1. If the above three occur at the same time, it means that there is no consistency between the expression of the face image, the person's posture and the voice information. At this time, the average value of the weight values of the two can be taken. For example, the average value of the first weight value of the person's posture and the voice information is (1+2) / 2=1.5, or the average value of the three is (-1+1+2) / 3=0.67. At this time, the weight value of the first key point information is 1.5 or 0.67.
[0169] When a person's facial expression is surprise, the first weight value of the face image is set to 2. When a person's body is facing one side for more than a specified time or the number of times the body is facing one side exceeds a specified number of times within a specified time, the first weight value of the person's posture is set to 2. When the evaluation level in the voice information is very good, the first weight value of the voice information is set to 2. If the above three occur at the same time, it means that there is consistency between the expression of the face image, the person's posture and the voice information. Then increase the weight value of the first key point information. For example, multiply the average value of the three by a coefficient such as 3, that is, (2+2+2) / 3*3=6. At this time, the increased weight value of the first key point information is 6.
[0170] The above example is explained using one passenger as an example. When there are multiple passengers in the car, the weight value of the first key point information of each passenger can be calculated separately, and then the average value of the first key point information of multiple passengers is taken.
[0171] While executing step S103 , step S106 is also executed.
[0172] In step S106, the second image weight setting unit 16 sets a second weight value for the second key point information, and uses the second weight value as the second weight value of the second original image.
[0173] The second weight value can be set based on the content and quantity of the key point information in the second key point information. The content of the key point information refers to scenes that are generally recognized as beautiful, such as the sea, beach, and starry sky, or scenes that are generally recognized as disgusting, such as sewage, garbage, and dilapidated scenes, or scenes that are indifferent to the public, such as streets and trees without any characteristics. When the content of the key point information is scenes that are generally recognized as beautiful, such as the sea, beach, and starry sky, a positive value such as 2 is set for the sea, beach, and starry sky respectively; when the content of the key point information is scenes that are generally recognized as beautiful, such as sewage, garbage, and dilapidated scenes, a negative value such as -2 is set for the sewage, garbage, and dilapidated scenes respectively; when the content of the key point information is scenes that are generally recognized as disgusting, such as streets and trees without any characteristics, a value such as 0 is set for the streets and trees respectively. The number of key point information means that the second original image contains multiple key point information, for example, the sea scene contains the sea, the beach, the sailboat, and the sunset. At this time, a weight value is set for each key point information, and the sum or average of the weight values of the key point information is the second weight value of the second original image. In the following description, the sum of the weight values of the key point information is used as an example of the second weight value of the second original image.
[0174] The second weight value is set according to the preferences of the general public, which can be obtained based on feedback from the media audience's evaluation of the later-mentioned composite image including the second original image. For the occupants, the occupants' preferences can also be stored in the vehicle-mounted system or cloud processing unit in advance, thereby setting the second weight value based on the occupants' preferences set in advance.
[0175] When the second weight value is set according to the preferences of the occupant, an occupant recognition unit may be provided in the vehicle compartment, and a camera may capture the face of the occupant and output the acquired image information to the occupant recognition unit.
[0176] The occupant recognition unit includes a computing device with a processor, a storage device, a communication interface, etc. The processor executes a general face recognition algorithm to recognize the occupant. The storage device associates the image information of the occupant's face with the occupant's personal information (personal identification information) and stores them in a database. The occupant recognition unit recognizes the occupant based on the facial image information obtained by the camera and the information in the database. The occupant's personal information includes the occupant's hobbies, etc.
[0177] Next, in step S107, the second weight value of the second original image is increased based on the first weight value of the first original image, so that the second original image has a superimposed weight value. When the first key point information, i.e., the expression of the face image, the posture of the person, and the behavior expressed by the voice information representing the evaluation in the voice information, is consistent and the key words "sunset" and "sea" in the voice information of the occupant are consistent with the image information in the second original image, the above-mentioned consistency due to the expression of the face image, the posture of the person, and the voice information representing the evaluation in the voice information is set to a coefficient of 3, so that the weight value of the second original image, whose sum of the second weight values of the key point information "sunset" and "sea" in the second original image is 4, is increased to, for example, the second weight value 4 is multiplied by the coefficient 3, so that the superimposed weight value of the second original image is 12.
[0178] In addition to increasing the second weight value in a positive direction, increasing the second weight value also includes increasing the second weight value in a reverse direction. The priority of the sequence of the second original image is arranged in descending order according to the second weight value, that is, the larger the positive second weight value is, the higher the ranking in the sequence is, and the easier it is to be searched and selected by the user, while the larger the reverse second weight value is (the smaller the value is), the lower the ranking in the sequence is, and the less likely it is to be searched by the user. The reverse second weight value is obtained when a negative number appears in the first weight value, such as an expression of disgust, a posture that has not moved to one side, and voice information with a very poor evaluation degree.
[0179] The method of increasing the second weight value may be to add or multiply the second weight value by the first weight value, or to multiply the second weight value by a coefficient corresponding to the first weight value, etc. The coefficient corresponding to the first weight value is set to 3 when the expression of the face image, the posture of the person, and the voice information representing the evaluation in the voice information are substantially consistent, set to 2 when two of them are consistent, and set to 1 when all three are inconsistent.
[0180] Next, in step S201, the first original image acquired by the first image acquisition unit 11 in step S101 and step S104 and the second original image acquired by the second image acquisition unit 14 are synthesized into a synthesized image by an image synthesis device. The synthesized image is an image synthesized by a part or all of the first original image and a part or all of the second original image. For example, the synthesized image may include the expression of joy in the face image in the image inside the vehicle, all behaviors shown by the body and sight line in the posture of the person and the voice information indicating the evaluation in the voice information, and the scene image including "sunset", "sea" and "seagull" in the image outside the vehicle, and the mark "sea in the fishing ban period" is deleted from the image.
[0181] In addition, the composite image may be a composite image in which the first original image and the second original image are displayed independently of each other (refer to Fig. 9 ), or a composite image obtained by fusing the first original image with the second original image (see Fig.10 ).
[0182] Then, in step S202, the first weight values corresponding to the facial image, human posture and voice information in the first original image are added to the second weight values corresponding to "sunset", "sea" and "seagull" in the second original image to form a composite image weight value of the composite image.
[0183] For example, in the above embodiment, the weight value of the first original image in which there is consistency among the facial expression, human posture and voice information is 6, while the superimposed weight value of the second original image after superposition is 12, then the composite image weight value of the composite image of the two is 6+12=18.
[0184] Furthermore, the following processing is performed on the first original image and the second original image by the image label generating device.
[0185] In step S301, a first label "joy" corresponding to the first original image is generated based on the first key point information, namely the expression of joy in the facial image, the body and sight line in the human posture facing one side, and the voice information indicating evaluation in the voice information.
[0186] While executing step S301, in step S302, based on the second key point information, namely, "sunset", "sea" and "seagull", a second label "sea" corresponding to the second original image is generated.
[0187] Next, in step S303, based on the first label "joy" and the second label "sea", a composite image label "sea in mind" or simply "joy, sea" corresponding to the composite image is generated.
[0188] Next, in step S401 , the image organizing device classifies the composite image into a landscape image based on the composite image label “the sea in my mind”, and further classifies it into the sea.
[0189] Next, in step S402, a sequence of synthetic images is generated based on the synthetic image weight values of the classified synthetic images. The sequence of synthetic images can be sorted by time. For example, when the synthetic image is the sea, the sequence of synthetic images can be sorted in the order of the synthetic image weight values of the sea in different periods of the day (early morning, morning, noon, afternoon, evening, etc.), or in different periods of the year (January, March, May, July, September, etc.); the sequence of synthetic images can also be sorted by location. Taking the sea as an example, the sequence of synthetic images can be sorted in the order of the synthetic image weight values of the sea in different locations (Dalian, Hainan, Kamakura, Japan, etc.). The scenes of the sea in different locations can be scenes that occur at the same time or at different times; the sequence of synthetic images can also be sorted by scene. Taking the sea as an example, as long as it is an image related to the sea, it can be sorted according to the scene. The image of the sea can be a real image or a virtual image (for example, from a film or television work, etc.), and the image of the sea is sorted in the order of the synthetic image weight value.
[0190] In addition, the length of the sequence of composite images may be set by the user, for example, an arrangement length of 10 composite images.
[0191] Next, in step S501, the image output device selects the synthetic image (the image with the largest weight) that is ranked first in the synthetic image sequence from the sequence of multiple synthetic images, and outputs the synthetic image in step S502. For example, when the user inputs "the sea in my mind", the synthetic image in the above embodiment can be obtained by the user first. After the user obtains the desired synthetic image, the synthetic image is manually edited or automatically edited through the automatic editing function of the application, and the edited synthetic image is uploaded to a communication medium such as TikTok.
[0192] Next, in step S601, the image correction device corrects the synthetic image weight value of the synthetic image based on the click rate or the degree of praise and the feedback of the audience evaluation of the uploaded video in the communication medium, and in step S602, the sequence of the synthetic images is adjusted. For example, the weight value of the synthetic image with a high click rate and a high degree of praise and a high feedback audience evaluation is increased, thereby making the synthetic image with a high feedback audience evaluation be ranked in a higher priority position in the image queue sequence. By repeating this process, the synthesis quality of the synthetic image can be continuously optimized, so that a synthetic image that fits the user's preferences can be obtained.
[0193] The above example is an explanation using the second original image as a scene image. The following example is also an explanation using a human image and a scene image. The processing of the human image and the scene image as the second original image is substantially the same as the processing of the scene image described above.
[0194] When the second original image is a humanistic image such as a cultural temple fair, in step S105, the second key point information extraction unit 15 extracts the second key point information of the second object in the cultural temple fair outside the car window, i.e., the second key point information of the cultural performances such as the crowd, snacks, and acrobatics, i.e., the crowd, food, and acrobatics. And in step S106, a weight value is set for the second key point information, and the weight value is used as the second weight value of the second original image. If the second key point information, i.e., the crowd, food, and acrobatics are scenes that the public likes, a positive value such as 2 is set respectively, and then the second weight value of the second original image is 2+2+2=6.
[0195] At this time, if the facial expression of the occupant in the vehicle is surprise, the first weight value of the face image is set to 2. When the person's body is facing one side for more than a specified time or the number of times the body is facing one side exceeds a specified number of times within a specified time, the first weight value of the person's posture is set to 2. However, when the occupant's voice information does not involve keywords such as crowds, food, acrobatics, etc., that is, when the second key point information in the second original image is not mentioned in the occupant's voice information, the first weight value of the voice information is set to 0, which means that there is no consistency between the expression of the face image, the person's posture and the voice information. At this time, the average of the weight values of the two can be taken. For example, the average of the first weight values of the face image and the person's posture is (2+2) / 2=2.
[0196] Since two of the three, the expression of the face image, the posture of the person, and the voice information indicating the evaluation, are consistent, the coefficient is 2.
[0197] Then in step S107 , the superposition weight value of the second original image is 6*2=12.
[0198] In step S202 , the composite image weight value of the composite image is 2+12=14.
[0199] When the second original image is a scene image such as a crowd, in step S105, the second key point information extraction unit 15 extracts the second key point information of the second object in the acquired second original image, i.e., the crowd outside the car window, the colliding vehicle, the traffic police, etc., i.e., the car accident, vehicle damage, violation, etc. In step S106, a weight value is set for the second key point information, and the weight value is used as the second weight value of the second original image.
[0200] If the second key point information, i.e., car accident, vehicle damage, and traffic violation, are scenes that are not popular with the public, a negative value, such as -2, is set respectively, and the second weight value of the second original image is -2-2-2=-6.
[0201] At this time, if the facial expression of the occupant in the vehicle is disgust, the first weight value of the face image is set to -2. When the person's body is facing one side for more than a specified time or the number of times the body is facing one side exceeds a specified number of times within a specified time, the first weight value of the person's posture is set to 2. When the occupant's voice information involves keywords such as car accidents, vehicle damage, and violations, that is, when the second key point information in the second original image is mentioned in the occupant's voice information, the first weight value of the voice information is set to 2. This means that there is no consistency between the expression of the face image, the person's posture, and the voice information. At this time, the average of the three weights can be taken. For example, the average of the first weight values of the face image, the person's posture, and the voice information is (-2+2+2) / 3=0.67.
[0202] Since two of the three, the expression of the face image, the posture of the person, and the voice information indicating the evaluation, are consistent, the coefficient is 2.
[0203] Then in step S107, the superposition weight value of the second original image is -6*2=-12.
[0204] In step S202, the synthetic image weight value of the synthetic image is 0.67-12=-11.33.
[0205] Of course, from the perspective of attracting people's attention, situations such as the above-mentioned car accidents can also increase their weight values.
[0206] Figure 8 It is a flow chart of an image processing module, an image synthesis module, an image label generation module, an image arrangement module, an image output module and an image correction module.
[0207] The present invention also relates to an image processing module in which the same processing as that performed by the above-mentioned image processing device and image processing method is performed.
[0208] That is, in the image processing module, a first original image is acquired in a first scene, the first original image includes a first object, first key point information of the first object is extracted, and a weight value is set for the first key point information, and the weight value is used as the first weight value of the first original image; a second original image is acquired in a second scene outside the first scene, the second original image includes a second object, second key point information of the second object is extracted, and a weight value is set for the second key point information, and the weight value is used as the second weight value of the second original image; based on the first weight value of the first original image, the second weight value of the second original image is increased, so that the second original image has a superimposed weight value.
[0209] The present invention also relates to an image synthesis module in which the same processing as that performed by the above-mentioned image synthesis device and image synthesis method is performed.
[0210] That is, in the image synthesis module, the first original image and the second original image are synthesized into a composite image, wherein the composite image is an image synthesized by part or all of the first original image and part or all of the second original image, and based on the first weight value of the first original image and the superimposed weight value of the second original image, a composite image weight value of the composite image is set.
[0211] The present invention also relates to an image label generation module in which the same processing as that performed by the above-mentioned image label generation device and image label generation method is performed.
[0212] That is, in the image label generation module, a first label corresponding to the first original image is generated based on the first key point information, a second label corresponding to the second original image is generated based on the second key point information, and a composite image label corresponding to the composite image is generated based on the first label and the second label.
[0213] The present invention also relates to an image arranging module in which the same processing as that performed by the above-mentioned image arranging device and image arranging method is performed.
[0214] That is, in the image arrangement module, the composite images are classified based on the composite image labels, and the sequence of the composite images is generated based on the composite image weight values of the classified composite images.
[0215] The present invention also relates to an image output module in which the same processing as that performed by the above-mentioned image output device and image output method is performed.
[0216] That is, in the image output module, a composite image is selected and output from the sequence of composite images.
[0217] The image processing module, image synthesis module, image label generation module, image arrangement module, and image output module of the present invention can be configured in a cloud processing unit, and the image processing can be performed in the cloud.
[0218] The present invention also relates to an image correction module in which the same processing as that performed by the above-mentioned image correction device and image correction method is performed.
[0219] That is, in the image correction module, based on the feedback for the outputted synthetic image, the synthetic image weight value of the synthetic image is corrected, thereby adjusting the sequence of the synthetic images.
[0220] The above-mentioned image correction module of the present invention can be set on the APP on the user terminal mentioned later.
[0221] In addition, the present invention also provides an electronic device, the electronic device comprising: a memory for storing executable instructions; a processor for implementing one or more of the above-mentioned image processing method, image synthesis method, image label generation method, image arrangement method, image output method, and image correction method when running the executable instructions stored in the memory. The electronic device also includes a display, the display is divided into multiple display areas, respectively used to display the first original image, the second original image, and the synthesized image.
[0222] The display may be a user terminal, which may include a smart phone, a tablet computer, a laptop computer, a PDA, a mobile internet device (MID), or other intelligent terminal with data processing functions (eg, multimedia data processing and playback functions).
[0223] The user searches for the composite image by entering the label of the composite image in the APP (Application) installed on the user terminal. The user can edit the composite image downloaded to the user terminal and then upload the edited image to a communication medium, such as TikTok. The editing of the composite image can be performed by one-click automatic generation or by manual production.
[0224] The APP will track the images uploaded by users to the communication media to understand people's preferences. When it is found that the feedback level of the audience evaluation of the uploaded image is high, such as the number of likes increases dramatically every day, the APP itself will take the content and form of the image with high feedback as a learning target to optimize its one-click generation system. The one-click generation system is a system that can automatically generate a synthetic image without manual editing by the user. When generating a synthetic image, the one-click generation system generates a video according to the content and form of the image with high feedback, thereby generating a synthetic image with high user preference. The content of the image refers to an image with a high number of likes, such as an image containing the sea, red sun, sailboat and excited passengers, while the image containing the sea, rocks and expressionless passengers with a bland reaction, i.e., a few likes, is not used as the content generated by one click; the form refers to an image form in which the images of the sea, red sun, sailboat and excited passengers are integrated through internal editing, while the image in which the images of the sea, red sun, sailboat and excited passengers are displayed separately with a bland reaction, i.e., a few likes, is not used as the form generated by one click. In addition, when the user chooses to manually generate, the image content and form with high feedback are pushed to the user for selection. In this way, the popularity of video production can be continuously improved through the feedback of the communication media, and the user's production time is saved, which improves the production efficiency.
[0225] Refer to the following Fig. 9 , Fig.10 The above-mentioned APP is described below.
[0226] When the user wants to open an APP installed on the user terminal and clicks it, the user terminal can respond to the user's start operation on the APP to start the APP in the user terminal and enter the homepage of the APP.
[0227] The homepage of the APP can be divided into the following three areas.
[0228] The bottom area: This area is used to set controls such as keyword or label input box, one-click generation, and manual generation. The manual generation includes, for example, time control, location control, scenario control, etc.
[0229] Among them, the keyword input box is used to input keywords to search for related composite images, the one-key generation control is used to automatically generate the searched and selected image into the image form you like with one key, and the manual generation control is for users to manually make images by themselves. During this manual operation, you can use time controls, location controls, scene controls, etc. The time control is an image that is associated in time, the location control is an image that is associated in location, and the scene control is an image that is associated in scene. The default display can be set to an image that is associated in time. Middle area: Two left and right display areas are set in the middle area, one of the two display areas is used to display the emotional image inside the vehicle, and the other is used to display the scene image outside the vehicle. For example, the left area displays the emotional image inside the vehicle, and the right area displays the scene image outside the vehicle.
[0230] Top area: The top area displays a composite image synthesized by the emotion image inside the vehicle and the scene image outside the vehicle. Before the synthesis of the composite image is completed, the control of the composite image is displayed. After the synthesis of the composite image is completed, the composite image after synthesis is displayed (refer to Fig. 9 , Fig.10 ).
[0231] For example, when a user searches for the sea, an image of the excited passengers in the car is displayed in the left display area in the middle area, while an image of the sunset over the sea outside the car taken at the same time as the image of the excited passengers in the car is displayed by default in the right display area. If the user likes these two images that are consistent in time, the control for the composite image in the uppermost area is clicked, and a composite image including the image of the excited passengers in the car and the image of the sunset over the sea outside the car is generated as the user clicks. If the user does not like the image of the sunset over the sea displayed in the right display area, the user can choose the image he likes. When the time control is clicked, the right display area shows the appearance of the sea here at different times of the day or at different times of the year. When the location control is clicked, the right display area shows the sea in different locations, such as the sea in Dalian, Hainan or Kamakura in Japan. The scenes of the sea in different locations may be scenes occurring at the same time or at different times. When the scene control is clicked, the scene of the sea is not limited to the association in time or place, but may also be a virtual scene from a film or television work, such as a sea from a distant planet.
[0232] After the user selects the image in the right display area that he likes by selecting the above time control, location control or scene control, he clicks the control of the composite image in the uppermost display area to composite the image. The composite image can be a composite image that displays the image of the excited passengers in the car and the image of the sea separately, for example Fig. 9The displayed image may also be a composite image that combines the image of the excited passengers in the car with the image of the sea, for example Fig.10 The image displayed.
[0233] In addition, the present invention also provides a computer-readable storage medium storing an executable program, which, when executed by a processor, implements one or more of the above-mentioned image processing methods, image synthesis methods, image label generation methods, image organization methods, image output methods, and image correction methods.
[0234] [Modifications]
[0235] In the above embodiment, the first original image and the second original image generated at the same time are synthesized to generate a synthesized image, but the present invention is not limited thereto. The first original image and the second original image generated at different times may be synthesized to generate a synthesized image with a time span.
[0236] For example, by recognizing sentences about past time periods such as "last year" and "last week" from the voice of a vehicle occupant, the image outside the vehicle of "last year" and "last week" and the current image inside the vehicle are associated with each other to create a composite image in which events in the image do not occur at the same time.
[0237] Similarly, other geographical locations different from the one in front of you can also be identified from the voice of the passengers in the vehicle. For example, when the current mountain is Huangshan, the passenger mentions "Mount Tai". At this time, "Mount Tai" can be associated with the current image inside the vehicle as an image outside the vehicle to create a composite image of events in an image that are not geographically in the same geographical area.
[0238] That is to say, as long as there is a certain association between the image inside the vehicle and the image outside the vehicle obtained by the vehicle, they can be combined into a composite image. In addition, the image outside the vehicle is not limited to the scene image existing in reality, but can also be a scene image appearing in film and television works, etc. For example, when the passengers in the vehicle want to make an interesting composite image, they intentionally add some words such as "volcano" and "Jurassic Park" in the voice talk. At this time, the image inside the vehicle can be associated with the images such as "volcano" and "Jurassic Park" obtained based on the key point information such as "volcano" and "Jurassic Park" recognized from the voice of the passengers in the vehicle to generate a composite image containing the passengers in the vehicle themselves and "volcano" or "Jurassic Park". Obviously, by generating a composite image in this way, the fun can be increased.
[0239] In addition, the present invention is not limited to the above-mentioned embodiment, and various structures can be adopted without departing from the gist of the present invention.
Claims
1. An image processing method, characterized in that: Acquire a first original image in a first scene, wherein the first original image includes a first object, extract first key point information of the first object, set a weight value for the first key point information, and use the weight value as a first weight value of the first original image; Acquire a second original image in a second scene outside the first scene, the second original image includes a second object, extract second key point information of the second object, set a weight value for the second key point information, and use the weight value as a second weight value of the second original image; The second weight value of the second original image is increased based on the weight value of the first original image, so that the second original image has a superimposed weight value.
2. The image processing method according to claim 1, characterized in that: The second original image is an image acquired simultaneously with the first original image or acquired based on the first key point information in the first original image.
3. The image processing method according to claim 2, characterized in that: The first original image is an emotional image, and the first key point information includes one or more of a face image, a person's posture, and voice information. extracting expression information of the face image, classifying the expression information, and setting a first weight value according to the expression category, extracting posture information of the posture of the person, classifying the posture, and setting a first weight value according to the posture category, Keywords in the voice information are extracted, the keywords are classified, and a first weight value is set according to the keyword category.
4. The image processing method according to claim 3, characterized in that: An average of first weight values of at least two of the expression information, the person's posture, and the keyword is obtained as a first weight value of the first key point information.
5. The image processing method according to claim 4, characterized in that: In the first key point information, when there is consistency between the expression information of the facial image, the posture of the person and at least two of the keywords, the first weight value of the first key point information with consistency is increased.
6. The image processing method according to claim 1, characterized in that: The second original image is a scene image, which includes a humanistic image, a landscape image and a scene image. The features of the scene image are extracted as the second key point information.
7. The image processing method according to any one of claims 1 to 6, characterized in that: The first scene is a scene inside the vehicle, and the second scene is a scene outside the vehicle.
8. An image synthesis method, characterized in that: Implementing the image processing method according to any one of claims 1 to 7, synthesizing the first original image and the second original image into a synthesized image, wherein the synthesized image is an image synthesized by partially or completely synthesizing the first original image with partially or completely synthesized the second original image; A composite image weight value of the composite image is set based on the first weight value of the first original image and the superimposed weight value of the second original image.
9. A method for generating an image label, characterized in that: Implementing the image synthesis method of claim 8; Generate a first label corresponding to the first original image based on the first key point information; Generate a second label corresponding to the second original image based on the second key point information; A composite image label corresponding to the composite image is generated based on the first label and the second label.
10. An image arrangement method, characterized in that: Implementing the image tag generation method according to claim 9; classifying the composite image based on the composite image label; A sequence of the composite images is generated based on the composite image weight values of the classified composite images.
11. An image output method, characterized in that: Implementing the image arrangement method according to claim 10; A composite image is selected from the sequence of composite images and outputted.
12. An image correction method, characterized in that: Implementing the image output method according to claim 11; Based on the feedback of the audience evaluation on the outputted composite images, the composite image weight values of the composite images are modified, thereby adjusting the sequence of the composite images.
13. An image processing device, It is characterized in that The method comprises: a first image acquisition unit, which acquires a first original image in a first scene, wherein the first original image includes a first object; a first key point information extraction unit, which extracts first key point information of the first object; and a first image weight setting unit, which sets a weight value for the first key point information and uses the weight value as a first weight value of the first original image; The system further includes: a second image acquisition unit, which acquires a second original image in a second scene outside the first scene, wherein the second original image includes a second object; a second key point information extraction unit, which extracts second key point information of the second object; and a second image weight setting unit, which sets a weight value for the second key point information and uses the weight value as a second weight value of the second original image; The image processing device increases the second weight value of the second original image based on the first weight value of the first original image, so that the second original image has a superimposed weight value.
14. The image processing device according to claim 13, characterized in that: The second original image is an image acquired simultaneously with the first original image or acquired based on the first key point information in the first original image.
15. The image processing device according to claim 14, characterized in that: The first original image is an emotional image, and the first key point information includes one or more of a face image, a person's posture, and voice information. When the first key point information extraction unit extracts the expression information of the face image, the expression information is classified, and the first image weight setting unit sets a first weight value according to the expression category. When the first key point information extraction unit extracts the posture information of the person's posture, the posture is classified, and the first image weight setting unit sets a first weight value according to the posture category. When the first key point information extraction unit extracts keywords from the voice information, the keywords are classified, and the first image weight setting unit sets a first weight value according to the keyword category.
16. The image processing device according to claim 15, characterized in that: The first image weight setting unit obtains an average of first weight values of at least two of expression information, human posture and keywords as the first weight value of the first key point information.
17. The image processing device according to claim 16, characterized in that: In the first key point information, when there is consistency between the expression information of the facial image, the posture of the person and at least two of the keywords, the first image weight setting unit increases the first weight value of the first key point information that is consistent.
18. The image processing device according to any one of claims 13 to 17, characterized in that: The second original image is a scene image, which includes a humanistic image, a landscape image and a scene image. The second key point information extraction unit extracts features of the scene image as the second key point information.
19. An image synthesis device, characterized in that: Based on the processing performed by the image processing device according to any one of claims 13 to 18, synthesizing the first original image and the second original image into a synthesized image, wherein the synthesized image is an image synthesized by partially or entirely combining the first original image with partially or entirely combining the second original image; and The composite image weight value of the composite image is set according to the first weight value of the first original image and the superimposed weight value of the second original image.
20. An image tag generating device, characterized in that: Based on the processing performed by the image synthesis device according to claim 19, Generate a first label corresponding to the first original image according to the first key point information; generating a second label corresponding to the second original image according to the second key point information; A composite image label corresponding to the composite image is generated according to the first label and the second label.
21. An image arrangement device, characterized in that: Based on the processing performed by the image tag generating device according to claim 20, classifying the composite image according to the composite image label; A sequence of the synthetic images is generated according to the synthetic image weight values of the classified synthetic images.
22. An image output device, characterized in that: Based on the processing performed by the image arrangement device according to claim 21, a composite image is selected from the sequence of composite images and output.
23. An image correction device, characterized in that: Based on the processing performed by the image output device according to claim 22, According to the feedback of the audience's evaluation on the outputted composite image, the composite image weight value of the composite image is modified, thereby adjusting the sequence of the composite images.
24. An image processing system, characterized in that: It includes image processing module, image synthesis module, image label generation module, image sorting module, image output module and image correction module. In the image processing module, a first original image is acquired in a first scene, the first original image includes a first object, first key point information of the first object is extracted, and a weight value is set for the first key point information, and the weight value is used as a first weight value of the first original image; Acquire a second original image in a second scene outside the first scene, the second original image includes a second object, extract second key point information of the second object, set a weight value for the second key point information, and use the weight value as a second weight value of the second original image; increasing a second weight value of the second original image based on the weight value of the first original image, so that the second original image has a superimposed weight value; In the image synthesis module, the first original image and the second original image are synthesized into a synthesized image, wherein the synthesized image is an image synthesized by partially or entirely combining the first original image with partially or entirely combining the second original image, and a synthesized image weight value of the synthesized image is set based on the first weight value of the first original image and the superimposed weight value of the second original image; In the image label generation module, a first label corresponding to the first original image is generated based on the first key point information, a second label corresponding to the second original image is generated based on the second key point information, and a composite image label corresponding to the composite image is generated based on the first label and the second label; In the image arrangement module, the composite images are classified based on the composite image labels, and a sequence of the composite images is generated based on composite image weight values of the classified composite images; In the image output module, a composite image is selected from the sequence of composite images and outputted; In the image correction module, based on the feedback of the audience evaluation on the outputted synthetic image, the synthetic image weight value of the synthetic image is corrected, thereby adjusting the sequence of the synthetic images.
25. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, for implementing the image processing method described in any one of claims 1 to 7, or the image synthesis method described in claim 8, or the image label generation method described in claim 9, or the image arrangement method described in claim 10, or the image output method described in claim 11, or the image correction method described in claim 12 when running the executable instructions stored in the memory.
26. The electronic device according to claim 25, characterized in that: The electronic device further includes a display, which is divided into a plurality of display areas, respectively used for displaying the first original image, the second original image and the synthesized image.
27. A computer-readable storage medium storing an executable program, characterized in that: When the executable program is executed by a processor, it implements the image processing method described in any one of claims 1 to 7, or implements the image synthesis method described in claim 8, or implements the image tag generation method described in claim 9, or implements the image organization method described in claim 10, or implements the image output method described in claim 11, or implements the image correction method described in claim 12.
