Methods, apparatus, equipment and media for optimizing electronic photo albums based on family image classification

Abstract

This invention relates to the field of video editing technology, and provides a method, apparatus, device, and medium for optimizing categorized electronic photo albums of family images. It solves the problem that currently automatically generated electronic photo albums cannot meet the diverse needs of users, leading to low user engagement. The method includes: acquiring a target video and user operation information for each electronic photo album corresponding to various automatically generated image types; optimizing a preset image classification rule based on the operation information to obtain a target classification rule; classifying each frame of the target video according to the target classification rule to obtain target image groups corresponding to each image category; and generating target electronic photo albums corresponding to each target image group based on each target image group, each electronic photo album, and the images already existing in each electronic photo album. The electronic photo albums obtained through this method are more in line with user preferences and usage habits.

Description

Technical Field

[0001] This invention relates to the field of video editing technology, and in particular to a method, apparatus, device, and medium for optimizing electronic photo albums of family images. Background Technology

[0002] With the widespread adoption of smart devices, the number of images and videos generated by users in their home life is rapidly increasing, and the types of videos and images taken by family members are becoming increasingly diverse. To facilitate users' browsing, organization, and review of family life moments, electronic photo album applications are gradually introducing automated image classification and album generation functions. By analyzing video frame images, images are aggregated into different categories of electronic photo albums.

[0003] In existing technologies, common image classification methods typically rely on preset classification rules. These rules can classify images to a certain extent in the initial stage. Existing patent CN113792822A (An Efficient Dynamic Image Classification Method, 2021.12.14) discloses a method that converts all raw natural images into features of different resolutions and uses a cross-fusion module to promote collaboration between features of different resolutions for image classification. However, because family photos have personalized and emotional characteristics, users' preferences during actual browsing and management often differ from the system's default classification rules. Furthermore, during the browsing of electronic photo albums, users may perform various operations such as moving images to other albums or clicking, zooming in, and zooming out. These operations contain information about subjective image preferences, but existing image classification methods often fail to utilize this information to optimize subsequent classification results, leading to electronic photo albums that do not meet user needs.

[0004] In summary, existing image classification and electronic photo album generation technologies generally lack adaptive classification capabilities and cannot dynamically optimize based on user behavior. As a result, the generated photo albums fail to reflect users' true preferences. Therefore, it is necessary to provide a new method for classifying family images and optimizing electronic photo albums so that the final generated electronic photo albums better match users' preferences and usage habits. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method, apparatus, device and medium for optimizing electronic photo albums for classifying family images, in order to solve the problem that automatically generated electronic photo albums in the prior art cannot meet the diverse needs of users, resulting in low user stickiness.

[0006] In a first aspect, embodiments of the present invention provide a method for optimizing electronic photo albums categorized from family images, the method comprising: Acquire the target video and the user's operation information for each electronic album corresponding to the automatically generated images; Based on the operation information, the preset classification rules for the image are optimized to obtain the target classification rules; According to the target classification rules, each frame of the target video is classified to obtain target image groups corresponding to each image category. Among them, each group of images corresponding to each electronic album category is denoted as the first image group, and each group of images that does not belong to any existing electronic album is denoted as the second image group. The first image group, the second image group, and each group of images corresponding to each electronic album are denoted as each target image group. Based on each target image group, each electronic album, and the images already existing in each electronic album, generate each target electronic album corresponding to each target image group.

[0007] Preferably, the acquisition of the target video and the user's operation information on each electronic album corresponding to the automatically generated images includes: Obtain the basic video and image filtering rules to be processed; According to the image filtering rules, each frame of the basic video is filtered to obtain each target image; The target video is generated based on the shooting sequence of each target image.

[0008] Preferably, the step of filtering each frame of the base video according to the image filtering rules to obtain each of the target images includes: Obtain the first overlap threshold between adjacent frames in the basic video, the second overlap threshold between two frames separated by at least one frame, and the threshold for the number of consecutive images with the required similarity. Based on the first overlap threshold and the second overlap threshold, the basic video is segmented to obtain each basic video segment; The total number of images in each of the basic video segments is compared with the continuous image threshold, and each target video segment is output. The target video segment is the basic video segment whose total number of images is greater than or equal to the continuous image number threshold. Each frame of the target video segment is acquired to obtain the target image.

[0009] Preferably, the step of optimizing the preset classification rules of the image based on the operation information to obtain the target classification rules includes: Obtain the vector difference and preset classification rule between the feature vectors of each image in each electronic album corresponding to the operation information before and after the user operation; The feature vectors corresponding to the preset classification rules are adjusted according to the differences between the vectors to obtain the target classification rules corresponding to each of the electronic photo albums.

[0010] Preferably, the step of classifying each frame of the target video according to the target classification rule to obtain a target image group corresponding to each image category includes: Obtain target image information of the target object in each image; Based on the size information of the target in the target image information and the size information of the reference object in the background image information, and / or the pattern information on the target object in the target image information, determine the age information of the target object; The target image group is obtained by filtering each frame of images based on the age information.

[0011] Preferably, generating each target electronic album according to the target scoring rules and image scoring thresholds includes: Each of the target image groups and each of the electronic photo albums is scored using the target scoring rules to obtain a score value for each image. The score value of each image is compared with the image score threshold to obtain the target image groups and the target images in each electronic album that meet the requirements; Each target electronic album is generated based on the target image corresponding to each target image group and each electronic album.

[0012] Preferably, generating each target electronic album according to the target scoring rules and image scoring thresholds includes: Obtain the number of target images; If the number of target images is greater than a first threshold and less than a second threshold, a transitional electronic album is generated based on the target image group and each target image in the corresponding electronic album; If the number of existing images exceeds the second threshold, then images are deleted at preset intervals according to the order of the target images to generate a transitional electronic album. Based on the similarity between the images in the transitional electronic album, the transitional electronic album is optimized to obtain the target electronic album.

[0013] Secondly, embodiments of the present invention provide a family image classification electronic photo album optimization device, the device comprising: Data acquisition module: Acquires the target video and user operation information for each electronic album corresponding to the automatically generated images; Rule optimization module: used to optimize the preset classification rules of the image based on the operation information to obtain the target classification rules; Image classification module: used to classify each frame of the target video according to the target classification rules, and obtain target image groups corresponding to each image category; Image scoring module: used to obtain the target scoring rules and image scoring thresholds for scoring each image of various types; Album generation module: used to generate electronic albums for each target based on the target scoring rules and image scoring thresholds.

[0014] Thirdly, embodiments of the present invention provide an electronic device, including: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of the first aspect described above.

[0015] Fourthly, embodiments of the present invention provide a storage medium storing computer program instructions, which, when executed by a processor, implement the method of the first aspect described above.

[0016] In summary, the beneficial effects of the present invention are as follows: The present invention provides a method, apparatus, device, and medium for optimizing electronic photo albums for classifying family images. By combining user operation information on electronic photo albums and images, the preset classification rules and scoring rules are dynamically optimized, thereby achieving adaptive adjustment to user preferences during image classification and album generation. The scoring rules are adjusted based on user operation behavior, making the album generation results more in line with user preferences, improving the personalization and accuracy of classification results. This achieves end-to-end adaptive adjustment of family images in the classification, filtering, and album generation process, which can significantly improve image management efficiency and the quality of electronic photo album generation, and the final generated electronic photo album is more in line with user preferences. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, and these are all within the protection scope of the present invention.

[0018] Figure 1 This is a flowchart illustrating the method for optimizing the classification of family images in an electronic photo album according to Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the process of acquiring the target video in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the image classification rule optimization process in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the process of grouping images by category in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the process for generating the target electronic photo album in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of the structure of the electronic photo album optimization device for classifying family images in Embodiment 2 of the present invention; Figure 7 This is a schematic diagram of the electronic device in Embodiment 3 of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, the element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Where there is no conflict, embodiments of the present invention and the various features thereof can be combined with each other, all of which are within the scope of protection of the present invention.

[0020] Example 1 With urbanization, the use of AI devices for infant and toddler care is gradually gaining acceptance among young parents. To better serve customers, AI-powered smart care devices use preset image classification rules to categorize captured images or recorded videos, creating corresponding electronic albums for users to view. These albums can include, but are not limited to, albums corresponding to actions such as laughing, crying, fussing, making faces, being playful, and being surprised. However, electronic albums generated directly using preset classification rules often suffer from image classification errors or fail to reflect individual user differences, resulting in poor album quality. This gradually diminishes user interest in the product and hinders the promotion of AI-powered smart care.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating the method for optimizing the classification of family images in an electronic photo album according to Embodiment 1 of the present invention. The method includes: S1: Obtain the target video to be processed and the user's operation information on each electronic album corresponding to the automatically generated images. The operation information includes first operation information on the album itself and second operation information on each image in the album. Specifically, during 24-hour infant monitoring using an AI caregiver (an AI caregiver is a smart camera equipped with a program that can recognize various activity states of infants and toddlers), the video data generated during the monitoring process is recorded in a loop according to a preset duration. In order to preserve the infants' wonderful movements, images are often captured according to preset capture rules or the images in each frame of the recorded video data are analyzed to select images that meet the requirements. Then, the captured images and / or the selected images are classified according to action categories to generate corresponding electronic albums for users to view. For example, the capture rule or selection rule is that when the imaging area of ​​the infant or toddler in the image occupies a proportion of the entire image that is greater than a preset value, it is captured or selected as the target image. Then, each target image is classified according to action category, including but not limited to recognizing body movements or micro-expressions. It should be noted that the preset capture rule or selection rule includes, but is not limited to: detecting facial information, centering of infant or toddler images, and overlap of multiple consecutive images, etc., which are not specifically limited here. The first operation information includes, but is not limited to: adjusting the name of the automatically generated electronic album, adjusting the name of each image, deleting images in the electronic album, and moving images between different electronic albums (including, but not limited to, moving images from electronic album A to electronic album B, adding images not selected in the basic video to the corresponding electronic album, and creating a new electronic album using the moved images). The second operation information includes, but is not limited to, cropping images, stitching multiple images together, adjusting the image (brightness adjustment, saturation adjustment, hue adjustment, etc.), and adding decorative elements to images (such as: animal shapes, anime shapes, text, decorations, and beautiful pictures, etc.).

[0022] In one embodiment, please refer to Figure 2 S1 includes: S11: Obtain image filtering rules and the underlying video to be processed; S12: Filter each frame of the basic video using the image filtering rules to obtain each of the target images; S13: Generate the target video according to the shooting sequence of each target image.

[0023] Specifically, image filtering rules are used to filter each frame of the latest stored base video to obtain target images that meet the requirements. Image filtering rules include, but are not limited to, the overlap between adjacent images, the number of consecutive images with the required overlap, image file size, infant image size, infant image position, and infant image completeness. For example, the target image file size is set to be greater than or equal to 512k, the overlap between two adjacent images is greater than 80%, the number of images between two frames with an overlap greater than 50% is no less than 5, the infant image area occupies more than 1 / 2 of the entire image, and the infant image must contain all parts of the infant's body, or at least the area above the infant's legs. Using these image filtering rules, each frame of the base video is filtered to obtain target images composed of frames that meet the requirements.

[0024] In one embodiment, S12 includes: S121: Obtain the first overlap threshold between adjacent images, the second overlap threshold between two frames of images with at least one frame interval, and the threshold for the number of consecutive images with the required similarity. S122: The basic video is segmented according to the first overlap threshold and the second overlap threshold to obtain each basic video segment; S123: Compare the total number of images in each of the basic video segments with the continuous image threshold, and output each target video segment, wherein the target video segment is the basic video segment whose total number of images is greater than or equal to the continuous image number threshold; Wherein, each of the target images is a frame image of each of the target video segments.

[0025] Specifically, by continuously comparing the overlap between each frame of images, the overlap comparison includes comparing two adjacent frames, as well as the overlap between the first frame and the current frame of consecutive frames that meet the overlap requirement. For example, if the video frame rate is 20 frames per second and each basic video is 30 seconds, when the overlap between the 9th and 10th frames is greater than the first overlap threshold (meets the overlap requirement), and the overlap between the 1st and 10th frames is less than the second overlap threshold (does not meet the overlap requirement), the first and second overlap values ​​of the 2nd to 9th frames all meet the requirements. In this case, the 1st frame is the aforementioned first frame, and the aforementioned current frame is the aforementioned 2nd to 10th frames. Because the overlap between the 1st and 10th frames is less than the second overlap threshold (does not meet the overlap requirement), the 10th frame will be used as the first frame of the next stage. This method divides the basic video into multiple video segments, and then uses a continuous image threshold to filter each basic video segment, obtaining video segments with stable images as target video segments. This completes the image filtering of each frame of the basic video and obtains each target image. This method can delete images with poor quality, such as ghosting, caused by rapid changes in the image in the basic video, reducing the amount of data processing required for subsequent image classification.

[0026] S2: Optimize the preset classification rules of the images according to the operation information corresponding to each of the electronic photo albums to obtain the target classification rules; Specifically, a large number of images of various infant and toddler movements are acquired. The movement categories corresponding to each image are labeled to form a sample set for model training. The model is trained using images of various infant and toddler movements from the sample set to obtain preset classification rules corresponding to various infant and toddler movements. When each frame of the target video is fed into the movement classification model, each frame is output according to the preset classification rules, thereby creating a corresponding electronic album based on each movement category. Users can operate on the images in the electronic album according to their own individual differences. The preset classification rules are optimized based on the user's operation information, making the image classification rules more in line with the individual differences of different users. For example, in the electronic album of infant and toddler smiling faces, all images with hand information in the facial area are filtered out to create a new electronic album; smiling face images with excessively large facial dimensions are deleted, etc. The preset classification rules are optimized based on the image information before and after the operation information. When classifying smiling face images using the optimized target classification rules, smiling face images corresponding to the above features will be excluded.

[0027] In one embodiment, please refer to Figure 3 S2 includes: S21: Obtain the vector difference between the feature vectors of each image in each electronic album corresponding to the operation information before and after the user operation, and the preset classification rule; Specifically, multidimensional feature vectors are used to represent the feature information of images, such as 256-dimensional feature vectors and 512-dimensional feature vectors. Multidimensional feature vectors of images before and after user operations in the electronic photo album are obtained, and the vector difference of the multidimensional feature vectors is calculated. In addition, a preset classification rule for image classification is obtained. It should be noted that the feature information corresponding to the preset classification rule is corresponding to the feature information of various actions. That is, the category to which the image belongs is determined by the vector value of the multidimensional vector.

[0028] S22: Adjust each feature vector corresponding to the preset classification rule according to the vector difference to obtain the target classification rule corresponding to each electronic album.

[0029] Specifically, the values ​​of the feature vectors corresponding to the preset classification rules are adjusted based on the differences between each vector. This includes, but is not limited to, adjusting the vector values ​​of the feature vectors corresponding to the preset classification rules by using the average value of each feature vector corresponding to each vector dimension of each frame image in the adjusted photo album. For example, if the user adjusts the brightness of image A in the smiley face photo album, the average value of the feature vectors representing brightness of all remaining images in the photo album (including image A after brightness adjustment) is calculated. Then, the difference between the average value of the feature vectors representing brightness before adjustment and the average value of the feature vectors representing brightness after adjustment is calculated. This difference is used to adjust the feature value of the feature vector corresponding to brightness in the preset classification rules for smiley faces, resulting in the target classification rule for the smiley face classification rule corresponding to the new feature vector representing brightness. By continuously optimizing the classification rules for each action category in this way, photo albums that meet the needs of different users can be established.

[0030] In one embodiment, S2 includes: Step 1: Obtain the album name information corresponding to the first operation information and the feature vectors representing the image information of each image in the album; Specifically, the album name information includes the name of the newly created electronic album, the original name and the current name of the existing album after the name has been adjusted, and the feature vector includes the feature vector before image adjustment and the feature vector after adjustment.

[0031] The second step involves comparing each album name with the name of each action category. If an action category exists that corresponds to the album name, the feature vectors of each image in that album are used to optimize the preset classification rule for that action category, resulting in the target classification rule for that action. If no existing action category matches the album name, the feature vectors of each frame in that album are used to establish a new preset classification rule for the action category that corresponds to the album name. This method can continuously optimize the action classification results. Furthermore, since infants and young children grow rapidly, this optimization method can adapt to individual differences in classification.

[0032] S3: Classify each frame of the target video using the target classification rules to obtain a target image group that corresponds one-to-one with each image category; Specifically, the target video's frames are classified using target classification rules to obtain image groups corresponding to each electronic album category, denoted as the first image group; and image groups not belonging to any existing electronic albums, denoted as the second image group. The first image group, the second image group, and the corresponding image groups in each electronic album are denoted as target image groups. It should be noted that the first image group and the images in the electronic album corresponding to the action category of the first image group are the same target image group. The target classification rules generate image groups outside of existing electronic albums, thereby achieving the subdivision of electronic albums to enable users to customize their electronic albums and increase user stickiness.

[0033] In one embodiment, S3 further includes: Step 1: Obtain the target image information of the target object in each image; Step 2: Determine the age information of the target object based on the size information of the target in the target image information and the size information of the reference object in the background image information, and / or the pattern information on the target object in the target image information; Step 3: Filter each frame of images based on the age information to obtain the target image group. Specifically, after classifying each frame of images by action category, the image size of the target object is compared with the image size of other reference objects in the environment, including: sofas, cribs, vehicles, tables, televisions, etc.; and / or, the age of the target object is determined based on the pattern information on the target object and the pattern information in the background image. If the pattern is a cartoon pattern, it is identified as an infant, thus filtering out infant images.

[0034] In one embodiment, please refer to Figure 4 S3 includes: S31: Obtain the first imaging information of the infant's face and the second imaging information of the infant's mouth in each frame image; S32: Based on the image proportion of each of the first imaging information and the target classification rule corresponding to the smiley face image, select a set of basic smiley face images that meet the requirements; S33: Based on the image proportion and image roundness of the second imaging information, each frame of the smiley face base image group is filtered to obtain the target image group that meets the requirements of the smiley face category; The smiley face category includes at least one of the following: a big laugh, a smile, and a fake smile, and the image roundness is the ratio of the radius of the largest circumcircle to the radius of the largest incircle of the image shape of the infant's mouth.

[0035] Specifically, in the process of creating an electronic album of smiling faces, firstly, the images of each frame of the target video are filtered according to the target classification rules corresponding to the smiling face images to obtain the basic image group of smiling faces corresponding to the smiling face actions. Then, the first imaging information of the infant's face and the second imaging information of the infant's mouth in each frame of the basic image group of smiling faces are obtained. Among them, the first imaging information includes, but is not limited to, the proportion of the face image size to the entire image, the distance between points of the geometric shape corresponding to the face image, side face, front face, etc. The second imaging information includes, but is not limited to, the proportion of the mouth image size to the face, the geometric shape of the mouth, and the roundness of the mouth, etc. The roundness of the mouth can be expressed by the formula Q = r / R, where r is the maximum inscribed radius of the mouth geometric shape, R is the maximum circumscribed radius of the mouth geometric shape, and Q is the roundness. When Q approaches 1, the smiling face is a big smile. When Q approaches 0, the roundness is... At 0.5, the smiley face is a slightly smiling face. A corresponding value range can be set to distinguish between a smile and a big laugh. At the same time, the value range can be automatically adjusted according to the user's response to a smile and / or a big laugh, such as putting a smiling image into a laughing photo album; by combining the size of the mouth image to the proportion of the face with the roundness, the images of each frame of the smiley face image group are filtered to obtain the target image group of the smiley face action.

[0036] S4: Generate each target electronic album corresponding to each target image group based on each target image group, each electronic album and the images already in each electronic album.

[0037] In one embodiment, please refer to Figure 5 S4 includes: S41: Obtain the target scoring rules and image scoring thresholds for each image of each type; In one embodiment, S41 includes: S411: Obtain the first duration of the image in the first state, the second duration of the image in the second state, and the duration threshold corresponding to the image in the first state, wherein the first state is the display state corresponding to the image not being enlarged or reduced, and the second state is the display state corresponding to the image being enlarged or reduced. S412: Based on the first duration and the second duration corresponding to the magnified area of ​​each image, adjust the preset scoring rules of the image to obtain the target scoring rules; Specifically, the first duration corresponding to the preset scoring rule is adjusted to be less than or equal to the duration threshold.

[0038] Specifically, the user's image viewing operation is divided into a viewing stage and a viewing end stage. Users can directly view the images in the album or zoom in on specific parts of the images before viewing. The state of directly viewing the image is recorded as the first state, and the state of zooming in is recorded as the second state. The longer the user views the image, the higher the image score. A first score weight is assigned to the first state, and a second score weight is assigned to the second state. The first score weight is less than the second score weight. The preset scoring rules are adjusted according to the features of the images viewed by the user. For example, if a user zooms in on the eye area of ​​an infant in image A of the smiling face album, the feature information of that eye is extracted and a higher score is given. If a user does not zoom in on image B and the first viewing time is very short, the feature information of various parts of image B is extracted and a lower score is given. It should be noted that when the first viewing time exceeds a threshold, it is considered that the user has left and has not continued to view the image. That is, the effective viewing time of directly viewing the image is less than or equal to the time threshold. This scoring method is used to filter target images that users like, increase users' interest in the album, and help promote the AI ​​care device.

[0039] S42: Using the target scoring rules, score each image in each target image group and each image in each electronic album to obtain the score value of each image; S43: Compare the score value of each image with the image score threshold to obtain each target image group and each target image in each electronic album that meets the requirements; S44: Generate each target electronic album based on each target image corresponding to each target image group and each electronic album.

[0040] In one embodiment, S44 includes: S441: Obtain the number of target images; Specifically, the number of target images includes all images in the electronic album corresponding to the target image group and the action category of the target image group.

[0041] S442: If the number of target images is greater than a first quantity threshold and less than a second quantity threshold, generate a transitional electronic album based on the target image group and each target image of the corresponding electronic album; S443: If the number of existing images is greater than the second quantity threshold, then delete the images according to the arrangement order of each target image at a preset interval, and generate a transitional electronic photo album; S443: Optimize the transitional electronic album based on the similarity between the images in the transitional electronic album to obtain the target electronic album.

[0042] Specifically, a first and a second threshold are set for the number of images in the electronic photo album. When the number of images in the electronic photo album is between the first and second thresholds, images that meet the similarity requirements are deleted. It should be noted that when deleting images that meet the similarity requirements from the album, the number of deleted images is not less than the first threshold. Therefore, multiple similarity thresholds are set for the image similarity. When more images need to be deleted, the similarity threshold is lower; when fewer images need to be deleted, the similarity threshold is higher. It is preferable to set a dynamic similarity threshold, that is, the similarity threshold requirement increases as the number of images to be deleted decreases. When the number of images exceeds the second threshold, a combination of direct deletion and deletion based on similarity is used for image deletion. Direct deletion involves deleting images at preset intervals. For example, if the album already contains 70 images and the second threshold is 50, at least 20 images need to be deleted. The preset interval is 1 image deleted every 5 images, leaving 56 images remaining. Then, at least 6 images are deleted based on image similarity. This method ensures that the number of images in the electronic album is within a reasonable range, so users do not need to spend too much time viewing the album, avoiding loss of user interest due to prolonged viewing time and improving user engagement.

[0043] It should be noted that when the number of images is less than the first threshold, the target electronic album is generated directly without similarity filtering, thus avoiding the problem of too few images in the electronic album affecting the user's viewing experience.

[0044] The family image classification electronic album optimization method of this embodiment obtains the user's operation information for electronic albums of various action categories, and uses this operation information to optimize the pre-set classification rules, thereby obtaining target classification rules that are closer to the user's aesthetic concepts; the optimized target classification rules are used to group the frames of newly stored target videos with classification; new target image groups for various actions are obtained. This method can not only increase the user's interest in each frame of the existing electronic album, but also refine the electronic albums corresponding to more action categories, thereby gaining user recognition of the electronic album and increasing user stickiness to the product.

[0045] Example 2 This invention also provides a device for optimizing electronic photo albums for classifying family images; please refer to [link / reference]. Figure 6 ,include: This invention also provides a device for optimizing electronic photo albums for classifying family images; please refer to [link / reference]. Figure 6 ,include: Data acquisition module: used to acquire the target video to be processed and the operation information of the user on each electronic album corresponding to the automatically generated images. The operation information includes first operation information on the album itself and second operation information on each image in the album. Data processing module: used to optimize the preset classification rules of the images according to the operation information corresponding to each of the electronic photo albums, and obtain the target classification rules; Image grouping module: used to classify each frame of the target video using the target classification rules to obtain target image groups that correspond one-to-one with each image category; Album generation module: used to generate each target electronic album corresponding to each target image group based on each target image group, each electronic album and the images already in each electronic album.

[0046] The family image classification electronic album optimization device of this embodiment obtains the user's operation information for electronic albums of various action categories, and uses this operation information to optimize the pre-set classification rules, thereby obtaining target classification rules that are closer to the user's aesthetic concepts; the optimized target classification rules are used to group the frames of newly stored target videos with classification; new target image groups of various actions are obtained. In this way, the user's interest in each frame of the existing electronic album can be increased, and more electronic albums corresponding to action categories can be refined, thereby gaining the user's recognition of the electronic album and increasing the user's stickiness to the product.

[0047] In one embodiment, the data acquisition module includes: Video acquisition unit: Acquires image filtering rules and the base video to be processed; Image filtering unit: Filters each frame of the base video using the image filtering rules to obtain each target image; Video generation unit: Generates the target video according to the shooting sequence of each target image.

[0048] In one embodiment, the image filtering unit includes: Image filtering parameter unit: obtains the first overlap threshold between adjacent images, the second overlap threshold between two frames of images with at least one frame interval, and the threshold for the number of consecutive images with the required similarity. Video segmentation unit: The basic video is segmented according to the first overlap threshold and the second overlap threshold to obtain each basic video segment; Target video unit: Compare the total number of images in each of the basic video segments with the continuous image threshold, and output each target video segment, wherein the target video segment is the basic video segment whose total number of images is greater than or equal to the continuous image number threshold; Wherein, each of the target images is a frame image of each of the target video segments.

[0049] In one embodiment, the data processing module includes: Operation parameter acquisition unit: acquires the vector difference between the feature vectors of each image in each electronic album corresponding to the operation information before and after the user operation, and the preset classification rule; Classification rule optimization unit: Adjusts each feature vector corresponding to the preset classification rule according to the difference between each vector to obtain the target classification rule corresponding to each electronic album.

[0050] In one embodiment, the image grouping module includes: Infant Imaging Information Unit: Acquires first imaging information of the infant's face and second imaging information of the infant's mouth in each frame of images; Smile Screening Unit: Based on the image proportion of each of the first imaging information and the target classification rule corresponding to the smile image, filter out the basic image group of smiles that meets the requirements; Target image filtering unit: Filters each frame of the smiley face base image group according to the image proportion and image roundness of the second imaging information to obtain the target image group that meets the requirements of the smiley face category; The smiley face category includes at least one of the following: a big laugh, a smile, and a fake smile, and the image roundness is the ratio of the radius of the largest circumcircle to the radius of the largest incircle of the image shape of the infant's mouth.

[0051] In one embodiment, the album generation module includes: Scoring parameter acquisition unit: acquires the target scoring rules and image scoring thresholds for scoring each image of various types; Image scoring unit: The scoring unit uses the target scoring rules to score each image in each target image group and each image in each electronic album to obtain the scoring value of each image. Target image generation unit: compares the score value of each image with the image score threshold to obtain each target image group and each target image in each electronic album that meets the requirements; Electronic photo album generation unit: Generates each target electronic photo album based on each target image corresponding to each target image group and each electronic photo album.

[0052] In one embodiment, the scoring parameter acquisition unit includes: Viewing duration unit: acquire the first duration of the image in the first state, the second duration of the image in the second state, and the duration threshold corresponding to the image in the first state, wherein the first state is the display state corresponding to the image not being enlarged or reduced, and the second state is the display state corresponding to the image being enlarged or reduced; Scoring rule optimization unit: Based on the first duration and the second duration corresponding to the magnified area of ​​each image, adjust the preset scoring rule of the image to obtain the target scoring rule; Specifically, the first duration corresponding to the preset scoring rule is adjusted to be less than or equal to the duration threshold.

[0053] The family image classification electronic album optimization device of this embodiment obtains the user's operation information for electronic albums of various action categories, and uses this operation information to optimize the pre-set classification rules, thereby obtaining target classification rules that are closer to the user's aesthetic concepts; the optimized target classification rules are used to group the frames of newly stored target videos with classification; new target image groups of various actions are obtained. In this way, the user's interest in each frame of the existing electronic album can be increased, and more electronic albums corresponding to action categories can be refined, thereby gaining the user's recognition of the electronic album and increasing the user's stickiness to the product.

[0054] Example 3 Specifically, the processor may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention. The electronic device includes at least one of the following: a camera, a mobile device with a camera, or a wearable device with a camera.

[0055] The memory may include a large-capacity storage device for data or instructions. For example, and not limitingly, the memory may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory may include removable or non-removable (or fixed) media. Where appropriate, the memory may be internal or external to a data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0056] The processor reads and executes computer program instructions stored in memory to implement any of the family image classification electronic album optimization methods in Embodiment 1 above.

[0057] In one example, the electronic device may also include a communication interface and a bus. The processor, memory, and communication interface are connected via the bus and communicate with each other.

[0058] The communication interface is mainly used to enable communication between various modules, devices, units and / or equipment in the embodiments of the present invention.

[0059] A bus, including hardware, software, or both, couples components of an electronic device together. For example, and not limitingly, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, a bus may include one or more buses. While specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.

[0060] In summary, the embodiments of the present invention provide a method, apparatus, device, and medium for optimizing the classification of family images into electronic photo albums. This involves acquiring user operation information for electronic photo albums of various action categories, using this operation information to optimize pre-set classification rules, thereby obtaining target classification rules that are closer to the user's aesthetic preferences; grouping the frames of newly stored target videos with classification using the optimized target classification rules; and obtaining new target image groups for various actions. This method can both increase the user's interest in each frame of the existing electronic photo album and refine the electronic photo albums corresponding to more action categories, thereby gaining user approval of the electronic photo album and increasing user stickiness to the product.

[0061] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0062] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing electronic photo albums by classifying family images, characterized in that, The method includes: Acquire the target video and the user's operation information for each electronic album corresponding to the automatically generated images; Based on the operation information, the preset classification rules for the image are optimized to obtain the target classification rules; According to the target classification rules, each frame of the target video is classified to obtain target image groups corresponding to each image category. Among them, each group of images corresponding to each electronic album category is denoted as the first image group, and each group of images that does not belong to any existing electronic album is denoted as the second image group. The first image group, the second image group, and each group of images corresponding to each electronic album are denoted as each target image group. Based on each target image group, each electronic album, and the images already existing in each electronic album, generate each target electronic album corresponding to each target image group.

2. The method for optimizing electronic photo albums for classifying family images according to claim 1, characterized in that, The process involves acquiring the target video and the user's operation information for each electronic album corresponding to the automatically generated images: Obtain the basic video and image filtering rules to be processed; According to the image filtering rules, each frame of the basic video is filtered to obtain each target image; The target video is generated based on the shooting sequence of each target image.

3. The method for optimizing electronic photo albums for classifying family images according to claim 2, characterized in that, The step of filtering each frame of the base video according to the image filtering rules to obtain each of the target images includes: Obtain the first overlap threshold between adjacent frames in the basic video, the second overlap threshold between two frames separated by at least one frame, and the threshold for the number of consecutive images with the required similarity. Based on the first overlap threshold and the second overlap threshold, the basic video is segmented to obtain each basic video segment; The total number of images in each of the basic video segments is compared with the continuous image threshold, and each target video segment is output. The target video segment is the basic video segment whose total number of images is greater than or equal to the continuous image number threshold. Each frame of the target video segment is acquired to obtain the target image.

4. The method for optimizing electronic photo albums for classifying family images according to claim 1, characterized in that, The step of optimizing the preset classification rules of the image based on the operation information to obtain the target classification rules includes: Obtain the vector difference and preset classification rule between the feature vectors of each image in each electronic album corresponding to the operation information before and after the user operation; The feature vectors corresponding to the preset classification rules are adjusted according to the differences between the vectors to obtain the target classification rules corresponding to each of the electronic photo albums.

5. The method for optimizing electronic photo albums for classifying family images according to claim 1, characterized in that, The step of classifying each frame of the target video according to the target classification rule to obtain a target image group corresponding to each image category includes: Obtain target image information of the target object in each image; Based on the size information of the target in the target image information and the size information of the reference object in the background image information, and / or the pattern information on the target object in the target image information, determine the age information of the target object; The target image group is obtained by filtering each frame of images based on the age information.

6. The method for optimizing electronic photo albums for classifying family images according to claim 1, characterized in that, The step of generating each target electronic album corresponding to each target image group based on each target image group, each electronic album, and the images already existing in each electronic album includes: Obtain the target scoring rules and image scoring thresholds for each image of various types; Each of the target image groups and each of the electronic photo albums is scored using the target scoring rules to obtain a score value for each image. The score value of each image is compared with the image score threshold to obtain the target image groups and the target images in each electronic album that meet the requirements; Each target electronic album is generated based on the target image corresponding to each target image group and each electronic album.

7. The method for optimizing electronic photo albums for classifying family images according to claim 6, characterized in that, The step of generating each target electronic album based on each target image corresponding to each target image group and each electronic album includes: Obtain the number of target images; If the number of target images is greater than a first threshold and less than a second threshold, a transitional electronic album is generated based on the target image group and each target image in the corresponding electronic album; If the number of existing images exceeds the second threshold, then images are deleted at preset intervals according to the order of the target images to generate a transitional electronic album. Based on the similarity between the images in the transitional electronic album, the transitional electronic album is optimized to obtain the target electronic album.

8. A device for optimizing electronic photo albums for classifying family images, characterized in that, The device includes: Data acquisition module: used to acquire the target video and the user's operation information on each electronic album corresponding to the automatically generated images; Rule optimization module: used to optimize the preset classification rules of the image based on the operation information to obtain the target classification rules; Image classification module: used to classify each frame of the target video according to the target classification rules to obtain target image groups corresponding to each image category. The image groups corresponding to each electronic album category are denoted as the first image group, and the image groups that do not belong to any existing electronic albums are denoted as the second image group. The first image group, the second image group, and the image groups corresponding to each electronic album are denoted as each target image group. Album generation module: Based on each target image group, each electronic album and the images already in each electronic album, generate each target electronic album corresponding to each target image group.

9. An electronic device, characterized in that, include: At least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, The method as described in any one of claims 1-7 is implemented when the computer program instructions are executed by the processor.

Images